aws s3 cp does not work with multiple files

[flaviut]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug

aws s3 cp does not support multiple files. The documentation says multiple files are supported, and v1 supports multiple files.

SDK version number

aws-cli/2.2.30 Python/3.8.8 Linux/5.13.12-arch1-1 exe/x86_64.arch prompt/off

Platform/OS/Hardware/Device

Linux/5.13.12-arch1-1

To Reproduce (observed behavior)

$ aws s3 cp s3://a/b s3://a/c ./
Unknown options: s3://a/b,s3://a/c,./

Expected behavior

Copy all the files listed to the last parameter.

Logs/output
Get full traceback and error logs by adding --debug to the command.

Additional context
Add any other context about the problem here.

[tim-finnigan]

Hi @flaviut, thanks for reaching out. You can use the --recursive parameter to copy multiple files. Here is the AWS CLI v2 s3 cp documentation for reference: https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html

Based on the example you provided, I think you want something like this:

aws s3 cp s3://<bucket>/a/ ./ --recursive

Or if you don’t want to copy every file, then you could use the parameters --exclude and --include like this:

aws s3 cp s3://<bucket>/a/ ./ --recursive --exclude "*" --include "b/*" --include "c/*"

You can find more aws s3 cp examples here in our User Guide: https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-objects-copy

I hope that helps! Please let us know if you have any other questions or feedback.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[flaviut]

Hey, thanks for taking a look!

Recursive isn't what I'm looking for here--let's say that I have a long list of files I want to download, and there's no pattern to them that can be expressed with globs. Or maybe I'm exploring, and I find it easier to just use the regular multi-parameter syntax.

Sure, I could use a bash loop or parallel, but I'd really rather use the same sort of syntax I'm used to with regular cp or with the aws-cli v1 aws s3 cp.

[tim-finnigan]

Hi @flaviut, I see what you’re saying. If you wanted to download specific filenames with the CLI then you could do something like this:

aws s3 cp s3://<bucket_name>/ ./ --recursive --exclude "*" --include "file-1.rtf" --include "file-2.rtf"

You might also want to consider looking into our SDKs for additional functionality in various programming languages: https://aws.amazon.com/getting-started/tools-sdks/

For example, in our Python SDK (boto3) this would allow you to loop through a list of specified files and download them:

import boto3
boto3.set_stream_logger('')

s3_client = boto3.client('s3')
s3_resource = boto3.resource('s3')

mybucket = s3_resource.Bucket('<bucket_name>')

list_of_files = ['file-1.txt', 'file-2.txt']

for obj in mybucket.objects.all():
    if obj.key in list_of_files:
        s3_client.download_file('<bucket_name>',
                                obj.key,
                                obj.key)

[flaviut]

It's great that there's a few different workarounds, and I really appreciate you going in depth to help me accomplish my tasks, but fundamentally this is a UX issue.

I expected things to work a certain way due to my experience with aws-v1 cp and just regular cp, and they work differently here, which is strange and frustrating.

[tim-finnigan]

I’m sorry to hear you are frustrated. Can you tell us how you were using aws s3 cp with v1 of the CLI? I’m not aware of changes with that command, at least not documented here in the breaking changes from migrating to v2: https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html

I know you preferred not to use a bash loop but I thought I'd share this one-liner solution anyway as it might help:
for file in $(cat filenames.txt); do {aws s3 cp s3://<bucket>/$file ./ } done