Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3-compatible endpoint consistency causing 409 Upload-Offset errors #12

Open
chendo opened this Issue Mar 9, 2019 · 2 comments

Comments

Projects
None yet
2 participants
@chendo
Copy link

chendo commented Mar 9, 2019

We're using an S3 API 'compatible' storage which seems to occasionally take a second or two to actually persist the update. This seems to be causing our 409 Upload-Offset header doesn't match current offset errors as when subsequent chunks are uploaded, the change to the info hasn't updated yet.

I'm thinking of subclassing the S3 storage adapter to add a delay for subsequent chunks, or potentially adding some kind of automatic retry mechanism on the client side. Is there a better solution?

@janko

This comment has been minimized.

Copy link
Owner

janko commented Mar 9, 2019

Thanks for the report. The approach that tus-ruby-server uses is that it stores the state of the upload in a separate S3 object (in JSON format). When a PATCH request arrives and a new multipart part is uploaded, tus-ruby-server will write the new Upload-Offset to that S3 object. Then on next request it will retrieve that S3 object and compare the Upload-Offset.

The reason why I mention this is because tusd, the official golang tus server implementation, uses a different approach. Instead of storing this state in the S3 object, on each PATCH request it fetches the parts of the multipart upload and sums up their sizes, and uses that as the current offset, comparing it to the Upload-Offset provided by the client.

Since I don't see any 409 errors reported on their issue tracker, I'm wondering if that approach is generally more reliable. Some time ago I asked the maintainers which approach is better, and we came to a conclusion that both probably "suffer" from eventual consistency (even when you look at AWS S3 itself). However, because tusd is more battle tested and has more contributors, I'm guessing it would be better if tus-ruby-server just copies their approach. Though it would require some rewriting.

If you see these errors consistently, it would be helpful if you could try running tusd as the tus server, and tell me if these errors are gone. That might tell us whether multipart uploads are "more consistent" than regular uploads, and whether tus-ruby-server should use that as the source of truth.

@chendo

This comment has been minimized.

Copy link
Author

chendo commented Mar 10, 2019

Interesting, I will give tusd a go. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.