Skip to content

S3 slow download compared to aws cli #1024

Answered by rcoh
CAReyes asked this question in Q&A
Discussion options

You must be logged in to vote

First, of course, is to double check that you're building in --release. At only 10-15MiB/s this is a possible culprit.

If you are using a new version of the CLI, the CLI is using the CRT, a C library to automatically split the download into multiple ranged GETs—the SDK does not do that automatically. You would need to write code to perform ranged GETs and use write_at to write the chunks to disk.

We have a code snippet in our S3 benchmark that shows how to use ranged GETs.

Note that build a library for customers to do this directly is high on our priority list for 2024.

We've benchmarked the Rust SDK achieving multiple Gib/S on large instance types so let me know if you're unable to match…

Replies: 1 comment 6 replies

Comment options

You must be logged in to vote
6 replies
@pavlospt
Comment options

@Velfi
Comment options

@pavlospt
Comment options

@Velfi
Comment options

@pavlospt
Comment options

Answer selected by Velfi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
4 participants