New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
storj: performance improvement for large file uploads #7198
Conversation
storj.io/uplink v1.11.0 comes with an improved logic for uploading large files where file segments are uploaded concurrently instead of serially. This allows to fully utilize the network connection during the entire upload process. This change enable the new upload logic.
Result from integration tests against Storj:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm happy to merge this as-is - I'd just like to query the use of the private
package.
@@ -24,6 +24,7 @@ import ( | |||
|
|||
"storj.io/uplink" | |||
"storj.io/uplink/edge" | |||
"storj.io/uplink/private/testuplink" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are using something out of private
? Maybe it should be public?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not yet a public API. We will follow with a PR when it becomes a new API or the default behavior of storj.io/uplink. Until it is decided, we want to have this performance improvement in Rclone.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK. I'll merge as-is and you can send a follow up later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is follow up PR: #7305
Thank you :-) |
The improved upload logic is active by default in uplink v1.12.0, so the `testuplink.WithConcurrentSegmentUploadsDefaultConfig(ctx)` is not required anymore. See rclone#7198
The improved upload logic is active by default in uplink v1.12.0, so the `testuplink.WithConcurrentSegmentUploadsDefaultConfig(ctx)` is not required anymore. See #7198
The improved upload logic is active by default in uplink v1.12.0, so the `testuplink.WithConcurrentSegmentUploadsDefaultConfig(ctx)` is not required anymore. See rclone#7198
The improved upload logic is active by default in uplink v1.12.0, so the `testuplink.WithConcurrentSegmentUploadsDefaultConfig(ctx)` is not required anymore. See #7198
What is the purpose of this change?
storj.io/uplink v1.11.0 has an improved logic for uploading large files where file segments are uploaded concurrently instead of serially. This allows to fully utilize the network connection during the entire upload process.
This change enables the new upload logic.
Was the change discussed in an issue or in the forum before?
No.
Checklist