Merge TM feature branch to mainline #3619
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…f S3ClientConfiguration with builder methods of S3AsyncClient (#3247)
…tination based filter is removed (#3258)
… package. Added integration tests for s3 select using S3CrtAsyncClient (#3289)
1. Rename destinationDirectory to destination. 2. Move DownloadDirectoryRequest.prefix and delimiter to just rely on modifying the list requests. 3. Remove upload directory recursive option in favor of using maxDepth(1). 4. Rename UploadDirectoryRequest's prefix and delimiter to s3Prefix and s3Delimiter. 5. Rename ResumableFileDownload's to* and writeTo* methods to serializeTo*. Remove charsets from write/read methods, and just use UTF-8. 6. Do not base64 encode when writing ResumableFileDownload to disk.
* Refactoring of Transfer manager APIs * Merging the integ test failure Pr 2119 from stagging branch
* Add flexible checksum support and update perf tests * fix checkstyle * Update based on latest change
…File (#3357) * Implement pause and resume for uploadFile * Fix merge conflict * Adding more tests * Add more tests * Update Javadocs * address feedback
…nt (#3403) * Implement automatic multipart copy functionality in S3 CRT async client * Add more tests * fix cancellation logic * Refactor CopyRequestProvider, fix request conversion and add more tests * Fix checkstyle
* Remove use of Junit4, clean up and consolidate tests in tm module * Ignoring the test if unicode can't be used as directory name
…#3432) * Support serialization and deserialization of ResumableFileUpload * Address feedback * Empty json should be unmarshalled to empty map
* Errors should not be wrapped * update handleException() * Changelog entry * Resolve comments Update changelog description, refactor handleException(), add test * Add failed message to SdkException
* Replace inline snippets with external compilable snippets * Fix build and address feedback * Fix build
* Only use CRT flexible checksum for getObject and putObject * Fix build
* integrate with crt s3 flow control * Update benchmark code * Add backpressure config * Change window size * Update initial window size * Change intial window size * Use heap max memory for initial window size * Give some buffer * change window size * Make read buffer size configurable * Log result to a file * Various updates * Various updates * Add CRT benchmark * Various updates * Fix checkstyle errors and tests * Fix flaky test * Fix checkstyle errors * Add validation * Add tests
#3549) * For copy operation, always forward multipart copy exception from one request to other multipart copy requests
* Add benchmarks for copy, uploadDirectory and downloadDirectory
* Update sample code and fix snippet path * Fix link
* Integrate with CRT checksum fix
* Use ByteBufferStoringSubscriber * Add a comment * Create constant for bytes bufferred
… rename S3TransferManager.builder().s3AsyncClient to .s3Client (#3584)
…ot for upload and copy (#3585)
#3586) * Throw UnsupportedOperationException if a user tries to pause a upload with non CRT-based S3 client
* Update javadoc
* Integrate with latest CRT pause/resume fix * Bump CRT version
…ect request threw exception in copy (#3609)
* Make aws-crt an optional dependency in s3-transfer-manager module. * Update README
|
Integ and stability tests passed |
millems
approved these changes
Dec 14, 2022
|
SonarCloud Quality Gate failed.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Merge TM feature branch to mainline
All changes have been reviewed previously except 21243fb