Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3Objectstore fails uploading large files #4331

Closed
VJalili opened this issue Jul 21, 2017 · 2 comments
Closed

S3Objectstore fails uploading large files #4331

VJalili opened this issue Jul 21, 2017 · 2 comments

Comments

@VJalili
Copy link
Member

VJalili commented Jul 21, 2017

s3_multipart_upload fails to upload a file of size 15MB.
The object_store_conf.xml which I'm using looks as the following:

<?xml version="1.0"?>
<object_store type="s3" id="corral">
    <files_dir path="/galaxy/test/files"/>
    <auth access_key="..." secret_key="..." />
    <bucket name="vahid-objectstore-tests" use_reduced_redundancy="False" />
    <cache path="database/object_store_cache" size="1000" />
    <extra_dir type="job_work" path="database/job_working_directory_s3"/>
    <extra_dir type="temp" path="database/tmp_s3"/>
</object_store>

and I get an error with the following trace:

  File "/Users/user/galaxy/lib/galaxy/objectstore/s3.py", line 604, in update_from_file
    self._push_to_os(rel_path, source_file)
  File "/Users/user/galaxy/lib/galaxy/objectstore/s3.py", line 384, in _push_to_os
    multipart_upload(self.s3server, self.bucket, key.name, source_file, mb_size)
  File "/Users/user/galaxy/lib/galaxy/objectstore/s3_multipart_upload.py", line 87, in multipart_upload
    enumerate(split_file(tarball, mb_size, cores)))):
  File "/Users/user/galaxy/lib/galaxy/objectstore/s3_multipart_upload.py", line 104, in wrap
    return func(self, timeout=timeout if timeout is not None else 1e100)
  File "/Users/user/galaxy/lib/galaxy/objectstore/s3_multipart_upload.py", line 104, in wrap
    return func(self, timeout=timeout if timeout is not None else 1e100)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 668, in next
    raise value
TypeError: transfer_part() takes exactly 6 arguments (1 given)
@dannon
Copy link
Member

dannon commented Jul 21, 2017

There's a PR open for fixing this, which looked good to me at first glance, I was just hoping to dig into it a little more and figure out what changed causing the errors

@VJalili
Copy link
Member Author

VJalili commented Jul 21, 2017

I guess #4315 is the PR. Thanks for the info, I'm closing this issue and hope the PR resolves this issue.

@VJalili VJalili closed this as completed Jul 21, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants