-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tsd-s3cmd user experience and issues for large-scale data sync #34
Comments
Yep, failed for me too (also using multipart-chunk), but honestly I thought it was ‘just’ due to crappy connection with MMIL, the only one I tried. I ended up using a very cumbersome workaround, spltting into smaller files and working via NIRD. Would indeed be great to get this fixed, given the huge syncing task.
… On 13 Mar 2020, at 09:14, Oleksandr Frei ***@***.***> wrote:
I think we experience multiple issues with tsd-s3cmd during large-scale data sync (100GB+), both for import and export.
Some of the issues are missing functionality (#1 <#1>), but there is more to it.
I have an open ticket [rt.uio.no #3797526] tsd-s3cmd fails uploading large file, about intermittent failures. My workaround was to use --multipart-chunk-size-mb=1000, and keep re-running the same command 10+ times until it succeeded.
@denvdm <https://github.com/denvdm> , what's your experience?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#34>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACE3BEFGLL24OCS4N3ARR5DRHHTO3ANCNFSM4LG5ILCA>.
|
Another relevant issue - s3-api-access-keys seem to be missing on new projects. I'm trying to clarify whether we need to request s3-api for each project separately, including white lists of IP addresses. |
Old ticket, I think this was fixed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I think we experience multiple issues with tsd-s3cmd during large-scale data sync (100GB+), both for import and export.
Some of the issues are missing functionality (#1), but there is more to it.
I have an open ticket
[rt.uio.no #3797526] tsd-s3cmd fails uploading large file
, about intermittent failures. My workaround was to use--multipart-chunk-size-mb=1000
, and keep re-running the same command 10+ times until it succeeded.@denvdm , what's your experience?
The text was updated successfully, but these errors were encountered: