Amazon S3 Transfer Manager for Swift

⚠️ Developer Preview

This library is currently in developer preview and is NOT recommended for production environments.

It is meant for early access and feedback gathering at this time. We'd love to hear from you on use cases, feature prioritization, and API feedback.

See the AWS SDK and Tools maintenance policy descriptions for more information.

Overview

The Amazon S3 Transfer Manager for Swift (S3TM for short) is a high-level library built on top of the AWS Swift SDK S3 client. It provides an intuitive transfer API for reliable and performant data transfer between your Swift application and Amazon S3, as well as the ability to monitor the progress of the transfers in real-time.

There are 4 transfer operations supported by S3TM:

• Upload a single object
• Download a single object
• Upload a directory
• Download a bucket

The S3TM allows monitoring the progress of all 4 operations listed above.

Getting Started

Add the dependency to your Xcode project

1. Open your project in Xcode and click on your .xcodeproj file, located at the top of the file navigator on the left pane.
2. Click the project name that appears on the left pane of the .xcodeproj file window.
3. Click on Package Dependencies tab, and click + button.
4. In Search or Enter Package URL search bar, enter git@github.com:aws/aws-sdk-swift-s3-transfer-manager.git.
5. Wait for package to load, and once it's loaded, choose the target you want to add the S3TransferManager module to.

Add the dependency to your Swift package

1. Add below to your package definition:

dependencies: [
    .package(url: "https://github.com/aws/aws-sdk-swift-s3-transfer-manager.git", from: "<VERSION_STRING>")
],

2. Add S3TransferManager module dependency to the target that needs it. For example:

targets: [
    .target(
        name: "YourTargetThatUsesS3TM",
        dependencies: [
            .product(name: "S3TransferManager", package: "aws-sdk-swift-s3-transfer-manager")
        ]
    )
]

Initialize the S3 Transfer Manager

You can initialize a S3TM instance with all-default settings by simply doing this:

// Creates and uses default S3TM config & S3 client.
let s3tm = try await S3TransferManager()

Or you could pass the config object to the initializer to customize S3TM by doing this:

// Create the custom S3 client config that you want S3TM to use.
let customS3ClientConfig = try S3Client.S3ClientConfiguration(
    region: "us-west-2",
    . . . custom S3 client configurations . . .
)

// Create the custom S3TM config with the S3 client config initialized above.
let s3tmConfig = try await S3TransferManagerConfig(
    s3ClientConfig: customS3ClientConfig,
    targetPartSizeBytes: 10 * 1024 * 1024, // 10MB part size.
    multipartUploadThresholdBytes: 100 * 1024 * 1024, // 100MB threshold.
    multipartDownloadType: .part
)

// Finally, create the S3TM using the custom S3TM config.
let s3tm = S3TransferManager(config: s3tmConfig)

For more information on what each configuration does, please refer to the documentation comments on S3TransferManagerConfig.

Amazon S3 Transfer Manager for Swift usage examples

Upload an object

To upload a file to Amazon S3, you need to provide the input struct UploadObjectInput, which contains a subset of PutObjectInput struct properties and an array of transfer listeners. You must provide the destination bucket, the S3 object key to use, and the object body.

When object being uploaded is bigger than the threshold configured by multipartUploadThresholdBytes (16MB default), S3TM breaks them down into parts, each with the part size configured by targetPartSizeBytes (8MB default), and uploads them concurrently using S3’s multipart upload feature.

// Construct UploadObjectInput.
let uploadObjectInput = UploadObjectInput(
    body: ByteStream.stream(
        FileStream(fileHandle: try FileHandle(forReadingFrom: URL(string: "file-to-upload.txt")!))
    ),
    bucket: "destination-bucket",
    key: "some-key"
)

// Call .uploadObject and save the returned task.
let uploadObjectTask = try s3tm.uploadObject(input: uploadObjectInput)

// Optional for every transfer operation: await on the returned task and retrieve the operation output or an error.
// Even if you don't do this, the task executes in the background.
do {
    let uploadObjectOutput = try await uploadObjectTask.value
} catch {
    // Handle error.
}

Download an object

To download an object from Amazon S3, you need to provide the input struct DownloadObjectInput, which contains the download destination, a subset of GetObjectInput struct properties, and an array of transfer listeners. The download destination is an instance of Swift’s Foundation.OutputStream. You must provide the download destination, the source bucket, and the S3 object key of the object to download.

When object being downloaded is bigger than the size of a single part configured by targetPartSizeBytes (8MB default), S3TM downloads the object in parts concurrently using either part numbers or byte ranges as configured by multipartDownloadType (.part default).

// Construct DownloadObjectInput.
let downloadObjectInput = DownloadObjectInput(
    outputStream: OutputStream(toFileAtPath: "destination-file.txt", append: true)!,
    bucket: "source-bucket",
    key: "s3-object.txt"
)

// Call .downloadObject and save the returned task.
let downloadObjectTask = try s3tm.downloadObject(input: downloadObjectInput)
let downloadObjectOutput = try await downloadObjectTask.value

Upload a directory

To upload a local directory to a S3 bucket, you need to provide the input struct UploadDirectoryInput and provide the destination bucket, and the source directory’s URL.

The UploadDirectoryInput struct has several optional properties that configure the transfer behavior. For more details on what each input configuration does, refer to the documentation comments on the UploadDirectoryInput.

// Construct UploadDirectoryInput.
let uploadDirectoryInput = try UploadDirectoryInput(
    bucket: "destination-bucket",
    source: URL(string: "source/directory/to/upload")!
)

// Call .uploadDirectory and save the returned task.
let uploadDirectoryTask = try s3tm.uploadDirectory(input: uploadDirectoryInput)
let uploadDirectoryOutput = try await uploadDirectoryTask.value

Download a bucket

To download a S3 bucket to a local directory, you need to provide the input struct DownloadBucketInput and provide the source bucket, and the destination directory URL.

The DownloadBucketInput struct has several optional properties that configure the transfer behavior. For more details on what each input configuration does, refer to the documentation comments on the DownloadBucketInput.

// Construct DownloadBucketInput.
let downloadBucketInput = DownloadBucketInput(
    bucket: "source-bucket",
    destination: URL(string: "destination/directory/for/download")!
)

// Call .downloadBucket and save the returned task.
let downloadBucketTask = try s3tm.downloadBucket(input: downloadBucketInput)
let downloadBucketOutput = try await downloadBucketTask.value

Monitor transfer progress

You can optionally configure transfer listeners for any of the S3TM operations above. The Amazon S3 Transfer Manager for Swift provides 2 canned transfer progress listeners for you. They’re LoggingTransferListeners and StreamingTransferListeners. There's a specific listener type for each operation, e.g., UploadObjectLoggingTransferListener is a LoggingTransferListener for the single object upload operation.

The LoggingTransferListeners log transfer events to the console using swift-log. The StreamingTransferListeners publish transfer events to its AsyncThrowingStream instance property, which can be awaited on to consume and handle events as needed. You can configure any number of transfer listeners for the S3TM operations via their inputs (e.g., UploadObjectInput's transferListeners field). You can add your own custom transfer listeners as well, by implementing a struct or a class that conforms to the TransferListener protocol and configuring it in the input structs.

See below for the example usage of the two canned transfer listeners.

LoggingTransferListener

// Assume s3tm: S3TransferManager is initialized.

let uploadObjectInput = UploadObjectInput(
    body: ByteStream.stream(
        FileStream(fileHandle: try FileHandle(forReadingFrom: URL(string: "file-to-upload.txt")!))
    ),
    bucket: "destination-bucket",
    key: "some-key",
    transferListeners: [UploadObjectLoggingTransferListener()]
)

// Call .uploadObject and save the returned task.
let uploadObjectTask = try s3tm.uploadObject(input: uploadObjectInput)

// Task will output real-time upload transfer progress to the console as it executes.

StreamingTransferListener

For StreamingTransferListener, you must close the underlying AsyncThrowingStream after transfer completion by explicitly calling closeStream() on the StreamingTransferListener instance to prevent memory leaks and hanging stream consumers.

let s3tm = try await S3TransferManager()
        
// Create the UploadObjectStreamingTransferListener.
let uploadObjectStreamingTransferListener = UploadObjectStreamingTransferListener()

// Start up the background Task that consumes events from the corresponding stream.
Task {
    for try await uploadObjectTransferEvent in uploadObjectStreamingTransferListener.eventStream {
        switch uploadObjectTransferEvent {
        case .initiated(let input, let snapshot):
            print("UploadObject operation initiated. ID: \(input.id)")
        case .bytesTransferred(let input, let snapshot):
            print("Transferred more bytes. Running total: \(snapshot.transferredBytes)")
        case .complete(let input, let output, let snapshot):
            print("Successfully finished UploadObject. ID: \(input.id)")
            uploadObjectStreamingTransferListener.closeStream() // Close stream explicitly if it won't be used anymore.
        case .failed(let input, let snapshot):
            print("UploadObject failed. ID: \(input.id)")
            uploadObjectStreamingTransferListener.closeStream() // Close stream explicitly if it won't be used anymore.
        }
    }
}

let fileToUpload = URL(string: "file-to-upload.txt")!

// Invoke the transfer manager operation with the streaming transfer listener configured in the input.
let uploadObjectTask = try s3tm.uploadObject(input: UploadObjectInput(
    body: ByteStream.stream(FileStream(fileHandle: FileHandle(forReadingFrom: fileToUpload))),
        bucket: "destination-bucket",
    key: "some-key",
    transferListeners: [uploadObjectStreamingTransferListener]
))

// Task will output real-time upload transfer progress to the console as it executes.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.