S3P - 5x to 50x faster than aws-cli

S3P provides a radically faster way to copy, list, sync and do other bulk operations over large AWS S3 buckets.

You can use it as a command-line tool for common operations, or you can use it as a library for nearly anything you can imagine.

Why is S3P so fast?

S3's API is structured around listing items in serial - request 1000 items, wait, then request the next 1000. This is how nearly all S3 tools work. S3P, however, can list items in parallel. It leverages S3's ability to request the first 1000 items equal-to or after a given key. Then, with the help of algorithmic bisection and some intelligent heuristics, S3P can scan the contents of a bucket with an arbitrary degree of parallism. In practice, S3P can list buckets up to 20x faster than conventional methods.

S3P is really just a fancy, really fast, S3 listing tool. Summarizing, copying and synching are all boosted by S3P's core ability to list objects radically faster.

We've sustained copy speeds up to 8gigabytes/second between two buckets in the same region using a single EC2 instance to run S3P.

S3P Blog Post

Read more about S3P on Medium.

Requirements

1. NodeJS

   The aws-cli is no longer required. Large files (>= 100 megabytes by default, see --large-copy-threshold) are now copied with server-side parallel multipart copy (s3.uploadPartCopy) - pure JavaScript, no data flows through your machine, and files larger than 5 gigabytes (s3.copyObject's hard limit) just work.

2. Key names must use a limited character set:

   <space>
   !"#$%&'()*+,-./
   0123456789:;<=>?@
   ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`
   abcdefghijklmnopqrstuvwxyz{|}~
   

   Why? Since Aws-S3 doesn't support listing Keys in descending order, S3P uses a character-range-based divide-and-conquer algorithm.

AWS Credentials

s3p uses the same credentials aws-cli uses, so see their documentation: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

You can also select a specific AWS profile with --profile, exactly like the aws-cli. The profile's credentials, region and endpoint_url are all respected, with standard AWS precedence (explicit options override environment variables, which override the profile's settings).

Cross-Environment Compare, Sync & Copy

⚠️ New feature - cross-environment COPYING is under-tested. Cross-account compare and sync --dryrun (listing and diffing two buckets with two credential sets) have been tested against real AWS accounts and work great. Actual cross-account copying - especially stream mode - has so far only been verified against local MinIO test environments, not real AWS. Before trusting it with important data: run sync --dryrun first, try a small prefix, and verify the results (sizes, metadata) on the target. Feedback welcome!

compare, sync and cp work across two completely separate environments - different AWS accounts, regions, or even S3-compatible services like MinIO. Each side gets its own client, configured the AWS-standard way: with profiles.

# ~/.aws/config
# [profile staging]     [profile production]
# region = us-west-2    region = us-east-1

# compare the same bucket name across two accounts
npx s3p compare --bucket my-bucket --to-bucket my-bucket \
  --from-profile staging --to-profile production

# pretend-sync: detect and report every discrepancy without copying anything
npx s3p sync --bucket my-bucket --to-bucket my-bucket \
  --from-profile staging --to-profile production --dryrun

# real sync across two accounts
npx s3p sync --bucket my-bucket --to-bucket my-bucket \
  --from-profile staging --to-profile production

Per-side options: --from-profile/--to-profile, --from-region/--to-region and --from-endpoint/--to-endpoint. Any side without per-side options uses your ambient environment (env vars or --profile), so typically you only need to add one flag for the "other" side. The programmatic API additionally accepts fromCredentials/toCredentials (any AWS SDK v3 credentials object or provider).

How copying works across environments: when both sides share credentials and endpoint, copies are server-side (s3.copyObject, or parallel multipart s3.uploadPartCopy for large files) - zero bytes flow through your machine, even cross-region, so s3p remains pure orchestration you can run from home broadband. When the sides have different credentials or endpoints, server-side copy is impossible (one request = one signer), so s3p automatically streams each file through your machine: getObject from the source, multipart upload to the target, preserving ContentType, CacheControl, custom metadata, etc. Force a specific strategy with --copy-mode server|stream.

Cross-account does NOT require streaming. If one principal can read the source and write the destination - typically the source account grants your destination-account user read access with a bucket policy - then use plain --profile with that principal and copies stay server-side, entirely in-cloud. Reserve two-profile stream mode for when the accounts truly cannot share a principal.

Stream mode notes: your machine's bandwidth is the bottleneck - for big transfers, run s3p on an EC2 instance in one of the two regions. Concurrency is bounded by local memory, not request count: each actively-streaming file buffers up to 64MB, and --stream-memory (default 512MB, allowing 8 concurrent files) or --stream-concurrency control the commitment. Streaming lots of small files? Raise them.

CLI

There is no need to install s3p directly. As long as you have NodeJS installed, you can run s3p directly using npx.

The built in help details all the commands, options, and provides many examples:

# list commands and get overall help
npx s3p help

Use the --help option for detailed help and examples for each command:

# get specific command help and example uses
npx s3p cp --help

Install NPM Package

You can also install s3p locally which will allow it to run faster.

# install s3p on your current machine
npm install s3p -g

# now it runs from the local install:
npx s3p help

Features

In addition to performance, S3P provides flexible options for custom list, copying and comparing:

• Only list files with a matching prefix, starting-after a given key, and/or stopping-at a given key. These options are very fast; the rest of the bucket not matching these criteria is ignored completely.
• Filter source files with arbitrary JavaScript. Further filter every file listed arbitrarily based on Key, Size, or Date. This is slower, since every file must be filtered in JavaScript but none-the-less, quite useful.
• When copying, syncing or comparing, re-key files by replacing prefixes, adding prefixes, or with an arbitrary JavaScript function.

Performance

Surprisingly, you don't even need to run S3P in the cloud to see much of its benefits. You can run it on your local machine and, since server-side S3 copying never goes directly through S3P, it doesn't use up any of your bandwidth. (The one exception: cross-environment stream mode, where two credential sets make server-side copy impossible - see "Cross-Environment Compare, Sync & Copy" above.)

S3-bucket-listing performance can hit almost 20,000 50,000 items per second (as-of S3Pv3.5).

S3-bucket-copying performance can exceed 8 gigabytes per second.

Yes, I've seen 9 gigabytes per second sustained! This was on a bucket with an average file size slightly larger than 100 megabytes. S3P was running on a single c5.2xlarge instance. By comparison, I've never seen aws-s3-cp get more than 150mB/s. That's over 53x faster.

The average file-size has a big impact on s3p's overall bytes-per-second:

location	command	aws-cli	s3p	speedup	average size
local	ls	2500 items/s	50000 items/s	20x	n/a
local	cp	30 mB/s	150 mB/s	5x	512 kB
ec2	cp	150 mB/s	8 gB/s	54x	100 mB

S3P was developed to operate on buckets with millions of items and 100s of terabytes. Currently, S3P is still only a single-core NodeJS application. There are opportunities for even more massively parallel S3 operations by forking workers or even distributing the work across instances with something like Elastic-Queue. If someone needs solutions that are 100-1000x faster than aws-cli, let us know. We'd love to work with you.
- shane@genui.com

Documentation

All the documentation is embedded in the CLI help pages. Use:

# get a list of commands
npx s3p help

# get a list of options for a command
# example:
npx s3p cp --help

API

All the capabilities of the CLI are also available as an API - every command is an exported async function:

const s3p = require("s3p");

Exported Functions

Each returns a Promise. Options are the lowerCamelCase equivalents of the CLI options (--to-bucket → toBucket, --from-profile → fromProfile, ...).

function	description
ls(options)	list matching keys; resolves to an array of key strings (or full items with raw: true)
listBuckets(options)	resolves to a map of bucket names to creation dates
summarize(options)	scan a bucket and resolve to size/count statistics, size histograms, and optional per-folder or custom groupBy totals
compare(options)	diff two buckets (any two environments); resolves to {counts, bytes} - same, needToCopy, needToReplace, needToDelete
cp(options)	copy everything from one bucket/folder to another; resolves to {finalStats}
sync(options)	copy only what's missing or different (see the sync CLI help for the exact rules); resolves to {finalStats}
delete(options)	delete matching keys (requires confirmDeleteItemsFromBucket); resolves to {finalStats}
each(options)	custom iteration: your map/mapList function is called for every matching item
map(options)	map-reduce over all matching items with map, reduce, and optional finally functions
version	s3p's version string (a property, not a function)

API-only options, in addition to everything the CLI accepts:

• fromCredentials / toCredentials: any AWS SDK v3 credentials object or provider, per side - an alternative to fromProfile/toProfile for cross-environment operations
• credentials: same, applied to both sides
• filter, toKey, map, mapList, reduce, finally: plain JavaScript functions (the CLI takes js:-prefixed strings)

Example - compare two accounts programmatically:

const { counts, bytes } = await s3p.compare({
  bucket: "my-bucket",
  toBucket: "my-bucket",
  fromProfile: "staging",
  toProfile: "production",
  quiet: true,
});

Learning API Options via the CLI

To learn the API call for a specific CLI command, run that command on the command-line with the --api-example option. This will output example JavaScript code for invoking that command programmatically.

NOTE: When you use --api-example on the command-line, your command won't actually run. S3P will only output the JavaScript equivalent of the CLI command to the console and then quit.

Example

Run:

> npx s3p ls --bucket foo --quiet --api-example

Output:

require("s3p").ls({
  bucket: "foo",
  quiet: true,
});
// > Promise

Test run:

> node

Paste:

require("s3p")
  .ls({
    bucket: "foo",
    quiet: true,
  })
  .then((out) => console.log(out));

Output:

[
  'item1',
  'item2',
  'item3',
  ... 8463 more items
}

Developed

S3P was originally developed in 2020 by me, Shane Delamore, while I was working at GenUI.com in conjunction with a project for Resolution Bioscience, Inc.. It was open source with ResBio's permission.

Since then I've been maintaining and improving S3P in my free time. Recently (in 2026) I've decided to fork it over to my own account and continue maintaining it from here. As part of that, the ISC license copyright has been updated to me, Shane Brinkman-Davis Delamore. Nothing else has changed, though I'm not migrating any past issues or pull requests.