Missing Large Object support

[cyberduck]

2f14288 created the issue

Hi,

I have just deployed a minimal OpenStack Swift Object Store v1.4.1 (5 storage nodes, 1 proxy server) and have been trying out CyberDuck from my Mac OS X Lion laptop. It all works fine with smaller files but when I try to upload a large file (6.1GiB, i.e. above the 5GiB single object limit in Swift) CyberDuck tried to upload the whole file as a single object - at least it appears so from the log on the swift proxy server which logged this:

Jul  7 09:55:16 swift proxy-server 131.111.10.57 131.111.10.57 07/Jul/2011/08/55/16 PUT /v1/AUTH_uxsup/UxSup-Test-Container/SnowLeopard_10A432_UserDVD_2Z693-6990-068.dmg HTTP/1.0 413 - Cyberduck/4.0.2%20%28Mac%20OS%20X/10.7%29%20%28i386%29 uxsup%2CAUTH_FOOBARAUTHKEYREPLACED - - - - - 214.0182

(Note I replaced the authentication token with FOOBARAUTHKEYREPLACED for obvious reasons!)

And cyberduck threw up an error message saying:

HTTP Error: Upload failed
Unexpected result.  Request Entity Too Large.

You can see an image of the error message and the activity window showing the 6.1GiB file that it uploaded unsuccessfully in the attachment.

As far as I can see large files must be uploaded in a segmented fashion as described here:

(http://swift.openstack.org/overview_large_objects.html)(http://swift.openstack.org/overview_large_objects.html)

Not sure whether to class this as a bug or feature request but given it results in an error I will submit it as a "defect" for now... Also, the Platform is Mac OS X 10.7 (GM, not a beta) and not 10.6 like I entered in the Platform field but you do not list 10.7 amongst the choices and 10.6 was the closest match. In any case I don't see how that would matter!

It would be great if this could be fixed/implemented so large file uploads work.

Thanks a lot in advance!

Best regards,

Anton

---

Attachments

• SwiftLargeUploadFailure.jpg (59.0 KiB)

[cyberduck]

@dkocher commented

Thanks for the detailed bug report. Segmented uploads are not currently supported.

[cyberduck]

@dkocher commented

Milestone 4.2 deleted

[cyberduck]

@dkocher commented

#7069 closed as duplicate.

[cyberduck]

e2f2ab4 commented

In version 4.3.1 on Mac OS X , downloaded a segmented file (does not need to be larger than 5GB) from Openstack swift results in a 'Transfer incomplete' message even though the file is transferred successfully. I tested with a 1MB file which was uploaded in 500KB segments using the Dynamic Large Object (DLO) feature.

The Openstack swift proxy server handles sending clients all the segments of a file transparently when a user requests a manifest file (which represents a segmented upload).

I would imagine this is because the manifest file on the object store is 0 bytes is length (as reported correctly by Cyberduck in the file listing) but the actual file that is downloaded is > 0 bytes.

Would this issue be considered as part of #6056 or should it be split into a separate ticket ?

[cyberduck]

@dkocher commented

Replying to [comment:7 david cole]:

In version 4.3.1 on Mac OS X , downloaded a segmented file (does not need to be larger than 5GB) from Openstack swift results in a 'Transfer incomplete' message even though the file is transferred successfully. I tested with a 1MB file which was uploaded in 500KB segments using the Dynamic Large Object (DLO) feature.

The Openstack swift proxy server handles sending clients all the segments of a file transparently when a user requests a manifest file (which represents a segmented upload).

I would imagine this is because the manifest file on the object store is 0 bytes is length (as reported correctly by Cyberduck in the file listing) but the actual file that is downloaded is > 0 bytes.

Would this issue be considered as part of #6056 or should it be split into a separate ticket ?

Handling manifest files for Dynamic Large Objects would be a separate issue from segmented uploads then. I assume Static Large Objects will work as the documentation says

The response’s Content-Length for a GET or HEAD on the manifest file will be the sum of all the segments in the / listing.

[cyberduck]

@dkocher commented

Referring to #7211. Does jclouds handle this transparently?

[cyberduck]

e2f2ab4 commented

Will you accept a patch for large file support to https://github.com/dkocher/java-openstack-swift ? We've not written it yet, but I want check if it's okay in principle !

[cyberduck]

@dkocher commented

Replying to [comment:10 david cole]:

Will you accept a patch for large file support to https://github.com/dkocher/java-openstack-swift ? We've not written it yet, but I want check if it's okay in principle !

Sure! That would be fantastic.

[cyberduck]

@dkocher commented

Current API documentation.

[cyberduck]

e2f2ab4 commented

We think this should be implemented in two phases:

(1) Single TCP streams. On upload, split the large files into adjustable size pieces (segments) and each segment is uploaded serially. A JSON user defined manifest is then uploaded on the successful upload of all segments. This is the Static Large Object type of segmentation.

(2) Multiple TCP streams. On upload, split the large files into adjustable size pieces (segments) and each segment is uploaded in parallel. A JSON user defined manifest is then uploaded on the successful upload of all segments. This is the Static Large Object type of segmentation.

A PoC of phase 1 is what we are working towards in the short term.

[cyberduck]

@dkocher commented

Replying to [comment:13 david cole]:

We think this should be implemented in two phases:

+1

[cyberduck]

@dkocher commented

Replying to [comment:13 david cole]:

We think this should be implemented in two phases:

Progress so far?

[cyberduck]

e2f2ab4 commented

Replying to [comment:15 dkocher]:

Replying to [comment:13 david cole]:

We think this should be implemented in two phases:

Progress so far?

Sorry for going quiet. We have been through the python-swiftclient code to understand the special steps required on upload & delete - in particular what to do if you upload a standard or segmented object over an existing segmented upload. We should have something for you in a week or so.

[cyberduck]

@dkocher commented

Ongoing work in this fork.

[cyberduck]

4f2e4bd commented

Replying to [comment:17 dkocher]:

Ongoing work in this fork.

---

Progress update:

I have made quite a bit of progress here by creating a new branch (phase2) in the above repository in which I've separated out some useful methods for dealing with large objects. Separately I have a checkout of the Cyberduck Subversion repository in which I have added a SwiftThresholdUploadService, a SwiftSmallObjectUploadFeature and a SwiftLargeObjectUploadFeature. I have also modified the SwiftDeleteFeature to remove large object segments. The behaviour of each of these features is configurable through the Preferences.

I am currently going through some testing, which is throwing up some interesting problems. All the features seem to be working against a devstack vm on my laptop, but I have hit some problems related to bulk deletes when testing against another system. Will update here when I've worked out what the issues are.

[cyberduck]

@dkocher commented

Replying to [comment:18 joel-wright]:

Replying to [comment:17 dkocher]:

Ongoing work in this fork.

---

Progress update:

I have made quite a bit of progress here by creating a new branch (phase2) in the above repository in which I've separated out some useful methods for dealing with large objects. Separately I have a checkout of the Cyberduck Subversion repository in which I have added a SwiftThresholdUploadService, a SwiftSmallObjectUploadFeature and a SwiftLargeObjectUploadFeature. I have also modified the SwiftDeleteFeature to remove large object segments. The behaviour of each of these features is configurable through the Preferences.

I am currently going through some testing, which is throwing up some interesting problems. All the features seem to be working against a devstack vm on my laptop, but I have hit some problems related to bulk deletes when testing against another system. Will update here when I've worked out what the issues are.

We can disable bulk deletes if required, this is not a blocker.

[cyberduck]

@dkocher commented

Added implementation in 4ad0539 and test in 54956bd.

[cyberduck]

@dkocher commented

Enabled in 985e573.