Cannot upload file to Galaxy #271

mmeier93 · 2019-01-10T11:45:14Z

I am trying to use BioBlend to upload a file (locally stored on my computer). I have tried doing so in 2 ways:
(1) Using the original Bioblend API
(2) Using the OO BioBlend API

In both cases, I obtain the same error (at least it seems). Here is my code:

Using original BioBlend API (bioblend.galaxy):

import os
from bioblend.galaxy import GalaxyInstance
from bioblend.galaxy.histories import HistoryClient
from bioblend.galaxy.tools import ToolClient

os.chdir("/home/username/Documents/projects/galaxy")

# Set up GalaxyInstance ####
gi = GalaxyInstance("https://usegalaxy.org", "my-instance-number")

# Make a new history ####
historyClient = HistoryClient(gi)
hi = historyClient.create_history("tmp2-100119")

# Upload datasets ####
toolClient = ToolClient(gi)
counts = toolClient.upload_file("example.txt", hi["id"])
counts2 = toolClient.upload_file("counted.txt",hi["id"]) # not working

Using OO BioBlend API (bioblend.galaxy.objects):

import os
from bioblend.galaxy.objects import GalaxyInstance

os.chdir("/home/username/Documents/projects/galaxy")

# Set up GalaxyInstance ####
gi = GalaxyInstance("https://usegalaxy.org", "my-instance-number")

# Make a new history ####
hi = gi.histories.create("tmp-100119")

# Upload datasets ####
counts = hi.upload_file("example.txt")
counts2 = hi.upload_file("counted.txt", file_type="tabular") # not working

In both code version, I have the same problem: I can upload a simple file called space-delimited example.txt which looks like this below. On the Galaxy GUI, I can see this file and it says “format: txt” and “uploaded txt file”.

1 2 3
A B C
D E F

However, I cannot upload my counted.txt file. This file is a tab delimited file, which has 51,827 lines. It represents a matrix of counts I want to perform DE analysis on. It looks like so (first few lines) (not sure why the file shows up in such a weird manner on the website but it does look normal on my computer)

GeneID              S1  S2  S3  S4  S5  S6  S7  S8  S9
gene1  0     0     0     0       3       1       7    2    0
gene2  0     0     8     0       0       0       0    0    0
gene3  14    34    75    53      75      36      35   50   64
gene4  0     0     0     1       0       0       0    0    10

The error I get in my terminal is as follows (the upload_file commands takes a min or so before the error appears). There are slightly different for both codes but they both in a ConnectionError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’,)) error.

Traceback (most recent call last):

 File "<ipython-input-14-345175af4c4e>", line 1, in <module>
counts2 = hi.upload_file("counted.txt", file_type="tabular") # not working

File "/home/username/anaconda3/envs/myenv-3.6/lib/python3.6/site-packages/bioblend/galaxy/objects/wrappers.py", line 1000, in upload_file
out_dict = self.gi.gi.tools.upload_file(path, self.id, **kwargs)

File "/home/username/anaconda3/envs/myenv-3.6/lib/python3.6/site-packages/bioblend/galaxy/tools/__init__.py", line 162, in upload_file
return self._tool_post(payload, files_attached=True)

 File "/home/username/anaconda3/envs/myenv-3.6/lib/python3.6/site-packages/bioblend/galaxy/tools/__init__.py", line 232, in _tool_post
return self._post(payload, files_attached=files_attached)

 File "/home/username/anaconda3/envs/myenv-3.6/lib/python3.6/site-packages/bioblend/galaxy/client.py", line 152, in _post
files_attached=files_attached)

 File "/home/username/anaconda3/envs/myenv-3.6/lib/python3.6/site-packages/bioblend/galaxyclient.py", line 137, in make_post_request
timeout=self.timeout)

 File "/home/username/anaconda3/envs/myenv-3.6/lib/python3.6/site-packages/requests/api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)

 File "/home/username/anaconda3/envs/myenv-3.6/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)

 File "/home/username/anaconda3/envs/myenv-3.6/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)

 File "/home/username/anaconda3/envs/myenv-3.6/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)

 File "/home/username/anaconda3/envs/myenv-3.6/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)

ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

I have managed to upload this counted.txt file to the Galaxy GUI successfully. It says “format: tabular” and “uploaded tabular file”. I have even manage to run some analyses on it. This file is 2.1 MB in size.

I am very lost as to what is causing this error. Any help would be deeply appreciated.

The text was updated successfully, but these errors were encountered:

nsoranzo · 2019-01-10T12:47:42Z

Any chance you have access to the corresponding Galaxy logs?

mmeier93 · 2019-01-10T13:35:23Z

@nsoranzo How can I get access to the logs? Not sure I understand what you mean (I am quite new to Galaxy)

nsoranzo · 2019-01-10T13:48:01Z

Sorry, I didn't notice you are using the public Galaxy server usegalaxy.org . You may try to start a local Galaxy server on your workstation and run the BioBlend upload against that to see if it works there. If not, you can then see the corresponding Galaxy logs.

mmeier93 · 2019-01-10T13:52:23Z

So are you saying my code looks ok? I haven't made a mistake. Not sure I am in position to install a local Galaxy server atm.. Any chance I can still access the logs you are mentioning?

nsoranzo · 2019-01-10T14:02:21Z

So are you saying my code looks ok? I haven't made a mistake.

It does look OK, unless I am missing something. You can look at the test code at https://github.com/galaxyproject/bioblend/blob/master/bioblend/_tests/TestGalaxyTools.py#L58

Not sure I am in position to install a local Galaxy server atm. Any chance I can still access the logs you are mentioning?

I surely doubt that. Starting a test Galaxy server is quite easy, see https://galaxyproject.org/admin/get-galaxy/

mmeier93 · 2019-01-14T16:12:42Z

I might set up a server eventually, but in the meantime, I have done more testing and found out that I can upload my file (called counted.txt). This file is 2.2MB and has 51827 lines. I did a test and noticed that if I just take the first say 175 lines (thereby drastically reducing the size of the file), I can upload my file.

So is there a limit in file size? I also happened to have seen that Galaxy is reporting some slow upload speeds on their European server and are also reporting problems on the .org server on Twitter today. Is it possible this is not related to BioBlend at all?

nsoranzo · 2019-01-14T21:00:49Z

Without access to the Galaxy server logs it is basically impossible to tell, unfortunately.

mmeier93 · 2019-01-16T16:53:55Z

I think I have found the issue without needing to install the a Galaxy instance. I was using Python 3.6 (within an Anaconda environment). I switched to Python 2.7 (also within an Anaconda environment) and my uploadings now all work. So not sure if this is a bug or something?

I have a question regarding Bioblend in general? Are you still planning on having the Bioblend object-ortiented interface replacing the old modules (i.e. bioblend.galaxy.objects replacing all bioblend.galaxy.GalaxyInstance/HistoryClient/ToolClient), as was suggested in the 2014 paper on Bioblend? I find the OO interface harder to use (due to not many people having posted on it) so i am sticking to the original modules for now.

nsoranzo · 2019-01-16T17:07:58Z

I think I have found the issue without needing to install the a Galaxy instance. I was using Python 3.6 (within an Anaconda environment). I switched to Python 2.7 (also within an Anaconda environment) and my uploadings now all work. So not sure if this is a bug or something?

I'm happy you workarounded the issue, but this is surely surprising! It would be interesting to compare the list of packages installed in the 2 conda environments, if you still have the first around. conda env export -n myenv-3.6 > myenv-3.6.yml should create a myenv-3.6.yml file with such a list for the Python 3.6 conda env. Similarly for the 2.7 one.

I have a question regarding Bioblend in general? Are you still planning on having the Bioblend object-oriented interface replacing the old modules (i.e. bioblend.galaxy.objects replacing all bioblend.galaxy.GalaxyInstance/HistoryClient/ToolClient), as was suggested in the 2014 paper on Bioblend? I find the OO interface harder to use (due to not many people having posted on it) so i am sticking to the original modules for now.

BioBlend.objects is still catching up in term of functionalities with respect to the function-oriented BioBlend, and usually new stuff is first developed for the latter and eventually added to the OO one later. So no worries!

mmeier93 · 2019-01-17T11:57:42Z

(Apologies I accidentally clicked on "Close issue" by mistake!)

myenv-3.6.txt
myenv-2.7.txt

Here are the 2 list of packages (I used the command you suggested but I saved them to .txt files instead of .yml files as I cannot upload yml files to GitHub. At first glance I'm not sure what to look for I have to admit. I should however add a bit of context:

As I mentioned before, I first started using BioBlend in a Python 3.6 Anaconda environment. I initially installed it with: conda install -c bioconda bioblend. I then re-read the Bioblend documentation and saw that it was suggested to download the stable release of Bioblend using pip. So I removed my initial install and then subsequently used pip to re-install BioBlend.
On the other hand, for the Python 2.7 environment (which was created later), I immediately installed Bioblend using pip.

Bottom line is both environments now have the same pip-installed Bioblend version and my last uploading tests were performed and compared using the same pip-installed BioBlend. I hope that makes sense.

Going back to the OO interface, thanks for your answer. I guess I just wanted to check that BioBlend as a whole is still in active development. Is that right? I will probably stick to the original API modules for now.

nsoranzo · 2019-01-17T12:41:58Z

@mmeier93 Thanks a lot for following up on this! Something that may cause your issue is that in the 3.6 Conda env there is a quite old requests-toolbelt 0.5.0 (latest is 0.8.0) . Can you try to update it (either through conda or uninstalling it and re-installing it via pip) and see if that solves the original problem?

I guess I just wanted to check that BioBlend as a whole is still in active development. Is that right?

Not in super-active development (43 commits last year), but surely maintained.

mmeier93 · 2019-01-18T11:16:33Z

That solved it! I uninstalled requests-toolbelt (which I guess was also installed with Bioconda when I originally installed Bioblend) and reinstalled with pip and it now all works. Thanks so much!

Ok great!

nsoranzo · 2019-01-18T11:39:24Z

Thanks again for the feedback! I'll raise the minimum requests-toolbelt version in setup.py to prevent this to happen in the future.

pcm32 · 2019-04-12T15:48:11Z

Hi! I think I'm hitting this from a conda installation of bioblend:

Failed due to ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
Traceback (most recent call last):
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()  
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()  
  File "/path-to/conda/envs/_bioblend@0.12.0_py3/lib/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

it doesn't happen on other setups where I don't use conda. The environment, where I only installed bioblend, looks like this:

# packages in environment at /nfs/production3/ma/home/nextflow_scxa/conda/envs/_bioblend@0.12.0_py3:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                py36_1003    conda-forge
bioblend                  0.12.0                     py_0    bioconda
boto                      2.49.0                     py_0    conda-forge
ca-certificates           2019.3.9             hecc5488_0    conda-forge
certifi                   2019.3.9                 py36_0    conda-forge
cffi                      1.12.2           py36hf0e25f4_1    conda-forge
chardet                   3.0.4                 py36_1003    conda-forge
cryptography              2.6.1            py36h72c5cf5_0    conda-forge
idna                      2.8                   py36_1000    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 8.2.0                hdf63c60_1  
libstdcxx-ng              8.2.0                hdf63c60_1  
ncurses                   6.1               hf484d3e_1002    conda-forge
openssl                   1.1.1b               h14c3975_1    conda-forge
pip                       19.0.3                   py36_0    conda-forge
pycparser                 2.19                     py36_1    conda-forge
pyopenssl                 19.0.0                   py36_0    conda-forge
pysocks                   1.6.8                 py36_1002    conda-forge
python                    3.6.7             h381d211_1004    conda-forge
pyyaml                    5.1              py36h14c3975_0    conda-forge
readline                  7.0               hf8c457e_1001    conda-forge
requests                  2.21.0                py36_1000    conda-forge
requests-toolbelt         0.5.0                    py36_0    bioconda
setuptools                41.0.0                   py36_0    conda-forge
six                       1.12.0                py36_1000    conda-forge
sqlite                    3.26.0            h67949de_1001    conda-forge
tk                        8.6.9             h84994c4_1001    conda-forge
urllib3                   1.24.1                py36_1000    conda-forge
wheel                     0.33.1                   py36_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.1.7             h14c3975_1001    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge

so I guess that the bioconda entry for bioblend is still using an older version of requests-toolbelt.

pcm32 · 2019-04-12T15:54:12Z

Doing on the same environment

conda install requests-toolbelt

gets a newer version of toolbelt, and with that it works, great!

nsoranzo · 2019-04-12T15:59:41Z

@pcm32 Should be fixed by bioconda/bioconda-recipes#14452

pcm32 · 2019-04-12T16:03:42Z

pfff... you beat me...

nsoranzo · 2019-04-12T16:30:22Z

Sorry 😉

Can you check the new bioblend-0.12.0-py_1 ?

mmeier93 closed this Jan 17, 2019

mmeier93 reopened this Jan 17, 2019

nsoranzo closed this in 0cbc8f1 Jan 18, 2019

galaxyproject / bioblend Public

Cannot upload file to Galaxy #271

Cannot upload file to Galaxy #271

mmeier93 commented Jan 10, 2019 •

edited

Loading

nsoranzo commented Jan 10, 2019

mmeier93 commented Jan 10, 2019

nsoranzo commented Jan 10, 2019 •

edited

Loading

mmeier93 commented Jan 10, 2019

nsoranzo commented Jan 10, 2019

mmeier93 commented Jan 14, 2019

nsoranzo commented Jan 14, 2019

mmeier93 commented Jan 16, 2019

nsoranzo commented Jan 16, 2019

mmeier93 commented Jan 17, 2019 •

edited

Loading

nsoranzo commented Jan 17, 2019

mmeier93 commented Jan 18, 2019

nsoranzo commented Jan 18, 2019

pcm32 commented Apr 12, 2019

pcm32 commented Apr 12, 2019

nsoranzo commented Apr 12, 2019

pcm32 commented Apr 12, 2019

nsoranzo commented Apr 12, 2019

galaxyproject / bioblend Public

Cannot upload file to Galaxy #271

Cannot upload file to Galaxy #271

Comments

mmeier93 commented Jan 10, 2019 • edited Loading

nsoranzo commented Jan 10, 2019

mmeier93 commented Jan 10, 2019

nsoranzo commented Jan 10, 2019 • edited Loading

mmeier93 commented Jan 10, 2019

nsoranzo commented Jan 10, 2019

mmeier93 commented Jan 14, 2019

nsoranzo commented Jan 14, 2019

mmeier93 commented Jan 16, 2019

nsoranzo commented Jan 16, 2019

mmeier93 commented Jan 17, 2019 • edited Loading

nsoranzo commented Jan 17, 2019

mmeier93 commented Jan 18, 2019

nsoranzo commented Jan 18, 2019

pcm32 commented Apr 12, 2019

pcm32 commented Apr 12, 2019

nsoranzo commented Apr 12, 2019

pcm32 commented Apr 12, 2019

nsoranzo commented Apr 12, 2019

mmeier93 commented Jan 10, 2019 •

edited

Loading

nsoranzo commented Jan 10, 2019 •

edited

Loading

mmeier93 commented Jan 17, 2019 •

edited

Loading