Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update documentation for Transfer Manager #3592

Merged
merged 7 commits into from
Dec 8, 2022
Merged

Update documentation for Transfer Manager #3592

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9d71292
Update javadoc
zoewangg Dec 6, 2022
b8bebe9
Remove use of external snippet
zoewangg Dec 6, 2022
2409cf3
Update README
zoewangg Dec 6, 2022
77203cc
Add a note for FileUpload#pause
zoewangg Dec 6, 2022
bd14714
Address feedback
zoewangg Dec 7, 2022
4478a90
Fix Override order
zoewangg Dec 7, 2022
03bdbef
Address feedback
zoewangg Dec 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 0 additions & 17 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,9 +233,6 @@
<doclint>none</doclint>
<!-- Exclude service packages to accelerate build -->
<excludePackageNames>software.amazon.awssdk.services.*:*.codegen</excludePackageNames>
<additionalJOption>
${additionalJavadocOption}
</additionalJOption>
</configuration>
</execution>
</executions>
Expand Down Expand Up @@ -734,17 +731,6 @@
</properties>
</profile>

<profile>
<id>jdk-18-plus</id>
<activation>
<jdk>[18,)</jdk>
</activation>
<properties>
<!-- snippet-path only works with JDK 18+ -->
<additionalJavadocOption>--snippet-path=${basedir}/src/test/java:${basedir}/service-custom/s3-transfer-manager/src/test/java</additionalJavadocOption>
cenedhryn marked this conversation as resolved.
Show resolved Hide resolved
</properties>
</profile>

<profile>
<id>quick</id>
<properties>
Expand Down Expand Up @@ -1003,9 +989,6 @@
<nodeprecatedlist>true</nodeprecatedlist>
<additionalJOptions>
<additionalJOption>--allow-script-in-comments</additionalJOption>
<additionalJOption>
${additionalJavadocOption}
</additionalJOption>
</additionalJOptions>
<windowtitle>AWS SDK for Java - ${awsjavasdk.version}</windowtitle>
<encoding>UTF-8</encoding>
Expand Down
154 changes: 102 additions & 52 deletions services-custom/s3-transfer-manager/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
## Overview

This project provides a much improved experience for S3 customers needing to easily perform uploads and downloads of
objects to and from S3 by providing the S3 S3TransferManager, a high level
library built on the [AWS Common Runtime S3 Client](https://github.com/awslabs/aws-crt-java).
The S3 Transfer Manager is a high-level transfer utility built on top of the asynchronous S3 client.
It provides a simple API to allow you to transfer files and directories between your application
and Amazon S3. The S3 Transfer Manager also enables you to monitor a transfer's progress in real-time,
as well as pause the transfer for execution at a later time.

## Getting Started

### Add a dependency for the transfer manager
### Add a dependency for the S3 Transfer Manager

First, you need to include the dependency in your project.

Expand All @@ -21,88 +22,137 @@ First, you need to include the dependency in your project.
Note that you need to replace `${awsjavasdk.version}` with the latest
SDK version.

### Instantiate the transfer manager
### Instantiate the S3 Transfer Manager

You can instantiate the transfer manager easily using the default settings:

```java
S3TransferManager tm = S3TransferManager.create();
S3TransferManager transferManager = S3TransferManager.create();
```

If you wish to configure settings, we recommend using the builder instead:
If you wish to configure settings, or use an underlying CRT-based S3 client you have already constructed,
we recommend using the builder instead:


```java
S3TransferManager tm =
S3AsyncClient s3AsyncClient =
S3AsyncClient.crtBuilder()
.credentialsProvider(DefaultCredentialsProvider.create())
.region(Region.US_WEST_2)
.targetThroughputInGbps(20.0)
.minimumPartSizeInBytes(8 * MB)
.build();

S3TransferManager transferManager =
S3TransferManager.builder()
.s3AsyncClient(S3CrtAsyncClient.builder()
.credentialsProvider(credentialProvider)
.region(Region.US_WEST_2)
.targetThroughputInGbps(20.0)
.minimumPartSizeInBytes(8 * MB))
.s3Client(s3AsyncClient)
.build();
```

### Upload a file to S3
### Transfer a single object

To upload a file to S3, you just need to provide the source file path and the `PutObjectRequest` that should be used for the upload:
#### Upload a file to S3 and log the upload’s progress with a TransferListener
To upload a file to Amazon S3, you need to provide the source file path and a PutObjectRequest specifying the target bucket and key.
Optionally, you can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener
logs a basic progress bar; users can also implement their own listeners.

```java
FileUpload upload =
tm.uploadFile(u -> u.source(Paths.get("myFile.txt"))
.putObjectRequest(p -> p.bucket("bucket").key("key")));
upload.completionFuture().join();
S3TransferManager transferManager = S3TransferManager.create();

UploadFileRequest uploadFileRequest =
UploadFileRequest.builder()
.putObjectRequest(req -> req.bucket("bucket").key("key"))
// attaching a LoggingTransferListener that will log the progress
.addTransferListener(LoggingTransferListener.create())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're adding a listener here, but not demonstrating how "log the upload’s process with a TransferListener" works.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment there. LMK if there's a better way

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's fine. Ideally it'd be nice with more sections in the README on extra features like TransferListeners, but if we don't have bandwidth right now we can skip it. We will have documentation in the dev guide?!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to add a section for TransferListeners, but I feel like I was just copying pasting code from single-object upload/download sections, so I ended up just adding additional notes for TransferListener. LMK what you think

.source(Paths.get("myFile.txt"))
.build();

FileUpload upload = transferManager.uploadFile(uploadFileRequest);

// Wait for the transfer to complete
upload.completionFuture().join();
```

### Download an S3 object to a file
#### Download an S3 object to a local file and log the download’s progress with a TransferListener

To download an object, you just need to provide the destination file path and the `GetObjectRequest` that should be used for the download:
To download an object, you need to provide the destination file path and a GetObjectRequest specifying the source bucket and key.
Same as upload, you can monitor the progress of the transfer by attaching a TransferListener.

```java
FileDownload download =
tm.downloadFile(d -> d.getObjectRequest(g -> g.bucket("bucket").key("key"))
.destination(Paths.get("myFile.txt")));
download.completionFuture().join();
```
S3TransferManager transferManager = S3TransferManager.create();

### Upload any content to S3
DownloadFileRequest downloadFileRequest =
DownloadFileRequest.builder()
.getObjectRequest(req -> req.bucket("bucket").key("key"))
.destination(Paths.get("myFile.txt"))
// attaching a LoggingTransferListener that will log the progress
.addTransferListener(LoggingTransferListener.create())
.build();

You may upload any arbitrary content to S3 by providing an `AsyncRequestBody`:
FileDownload download = transferManager.downloadFile(downloadFileRequest);

```java
Upload upload =
tm.upload(u -> u.requestBody(AsyncRequestBody.fromString("Hello world"))
.putObjectRequest(p -> p.bucket("bucket").key("key")));
upload.completionFuture().join();
// Wait for the transfer to complete
download.completionFuture().join();
```

Refer to the static factory methods available in `AsyncRequestBody` for other content sources.
#### Copy an S3 object from one location to another
To copy an object, you need to provide a CopyObjectRequest with source and destination location.

### Download an S3 object to a custom destination
```
S3TransferManager transferManager = S3TransferManager.create();
CopyObjectRequest copyObjectRequest = CopyObjectRequest.builder()
.sourceBucket("source_bucket")
.sourceKey("source_key")
.destinationBucket("dest_bucket")
.destinationKey("dest_key")
.build();
CopyRequest copyRequest = CopyRequest.builder()
.copyObjectRequest(copyObjectRequest)
.build();

Copy copy = transferManager.copy(copyRequest);

// Wait for the transfer to complete
CompletedCopy completedCopy = copy.completionFuture().join();
```

### Transfer multiple objects in the same directory

You may download an object from S3 to a custom destination by providing an `AsyncResponseTransformer`:
#### Upload a local directory to an S3 bucket

*(This example buffers the entire object in memory and is not suitable for large objects.)*
To upload a local directory recursively to an S3 bucket, you need to provide the source directory and the target bucket.

```java
Download<ResponseBytes<GetObjectResponse>> download =
tm.download(d -> d.getObjectRequest(g -> g.bucket("bucket").key("key"))
.responseTransformer(AsyncResponseTransformer.toBytes()));
download.completionFuture().join();
```
S3TransferManager transferManager = S3TransferManager.create();
DirectoryUpload directoryUpload = transferManager.uploadDirectory(UploadDirectoryRequest.builder()
.sourceDirectory(Paths.get("source/directory"))
.bucket("bucket")
.build());

Refer to the static factory methods available in `AsyncResponseTransformer` for other destinations.
// Wait for the transfer to complete
CompletedDirectoryUpload completedDirectoryUpload = directoryUpload.completionFuture().join();

### Attach a TransferListener
// Print out any failed uploads
completedDirectoryUpload.failedTransfers().forEach(System.out::println);
```

#### Download S3 objects within the same bucket to a local directory

To monitor a transfer's progress, you can include a `TransferListener` with your transfer request:
To download all S3 objects within the same bucket, you need to provide the destination directory and the source bucket.

```java
FileUpload upload =
tm.uploadFile(u -> u.source(Paths.get("myFile.txt"))
.putObjectRequest(p -> p.bucket("bucket").key("key"))
.addTransferListener(LoggingTransferListener.create()));
upload.completionFuture().join();
S3TransferManager transferManager = S3TransferManager.create();
DirectoryDownload directoryDownload =
transferManager.downloadDirectory(
DownloadDirectoryRequest.builder()
.destination(Paths.get("destination/directory"))
.bucket("bucket")
// only download objects with prefix "photos"
.listObjectsV2RequestTransformer(l -> l.prefix("photos"))
.build());
// Wait for the transfer to complete
CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();

// Print out any failed downloads
completedDirectoryDownload.failedTransfers().forEach(System.out::println);
```

You can provide your own implementation of a `TransferListener` to implement progress-bar-type functionality.
19 changes: 19 additions & 0 deletions services-custom/s3-transfer-manager/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,25 @@
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>${maven-javadoc-plugin.version}</version>
<executions>
<execution>
<id>generate-javadocs</id>
cenedhryn marked this conversation as resolved.
Show resolved Hide resolved
<goals>
<goal>jar</goal>
</goals>
<configuration>
<detectJavaApiLink>false</detectJavaApiLink>
<source>8</source>
<notree>true</notree>
<includeDependencySources>false</includeDependencySources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

Expand Down
Loading