Unable to get JSON output from aws s3 ls command

[boosh]

Output from aws s3 ls is always returned as text regardless of the value of the --output parameter.

[danielgtaylor]

I can confirm that this does not work as expected. If you are looking for JSON output from a specific S3 command, you can however use the low level API:

$ aws s3api list-buckets --output json
...
$ aws s3api list-buckets --output table
...

[jamesls]

The --output parameter is not intended to work with the aws s3 commands. You'll need to use the s3api commands to get different output. Given that --output is a global argument, the only real option we have is to document this.

[bes1002t]

this is a mess. S3 is able to sync and to ls but not to output the result as JSON string. S3api is able to output in JSON strings but not able to use sync command or ls. Many different tools and non of them is able to provide this simple featureset.

[dagwieers]

@bes1002t This works for me

$ aws --output json s3api list-objects --bucket your-bucket --prefix 'some/path/'

As an alternative to:

$ aws --output json s3 ls s3://your-bucket/some/path/

However, I do not see a good reason why JSON output would not be supported by aws s3 ls.

@jamesls Would a PR to add this be accepted?

[dannystaple]

Perhaps if it's not supported, the command should output a message stating that the output format isn't supported, and at a stretch, suggest an alternative that is supported? Kind of in the way git has the "did you mean" suggestions.

[bes1002t]

@dagwiers list-objects prefix could be a solution. When I posted my rant I did not know about that command. :P

How about the performance of list-objects vs ls? I experienced some performance issues using list-objects because with large buckets it compares the prefix with EACH file in bucket, which needs a huge amount of time for many files.

[dagwieers]

@bes1002t That is why I asked if a PR would be accepted. Now it looks like this was by design and will not be improved/fixed.

[nikolai-derzhak-distillery]

Yes. On large buckets s3api takes forever. So what is solution really ? Besides using external DB to index s3 objects :)

[bes1002t]

@nikolai-derzhak-distillery it seems there is no real solution so far. I tried several ways and none of them was a good solution for large buckets. I don't understand why nobody cares about such an performance issue.

[nCubed]

Where in the docs does it state aws s3 sync has opted out of the global parameters?

Not on the sync docs. Not on the global docs. 🤢 🤮 🤬

Just spent the past hour+ trying to figure out why no json output; this is about the only real reference confirming the entire --output option is disabled with s3.

[randomanderson]

@jamesls Why was this closed as "Completed". The bug still exists and cost me a couple hours trying to figure out why

[Hellseher]

It's still present:

aws --version
aws-cli/2.9.7

There is not any other type can be picked, but it listed in aws s3 ls help as available, but marked as ignored, why?

DESCRIPTION
       List  S3  objects and common prefixes under a prefix or all S3 buckets.
       Note that the --output and --no-paginate arguments are ignored for this
       command.

and

aws s3 ls help | grep output
       Note that the --output and --no-paginate arguments are ignored for this
          [--output <value>]
       The formatting style for command output.
       Turn on/off color output.
       Disable cli pager for output.
       Rather than showing PRE dirname/ in the output, all the  content  in  a

[landsman]

not working for me either

[DonBower]

if --output is not supported on aws s3 ls command, then why is it documented?
aws s3 ls see: Synopsis [--output <value>]

[jamest-gaia]

There should be a warning printed when an ignored option is used.

Also, it would be nice if this could be documented under GLOBAL OPTIONS, which is where I found the --output option. I didn't notice the note at the top under DESCRIPTION until after reading this issue.

$ aws s3 ls help
GLOBAL OPTIONS
       --debug (boolean)

       Turn on debug logging.

       --endpoint-url (string)

       Override command's default URL with the given URL.

       --no-verify-ssl (boolean)

       By  default, the AWS CLI uses SSL when communicating with AWS services.
       For each SSL connection, the AWS CLI will verify SSL certificates. This
       option overrides the default behavior of verifying SSL certificates.

       --no-paginate (boolean)

       Disable automatic pagination.

       --output (string)

       The formatting style for command output.

       o json

       o text

       o table

       o yaml

       o yaml-stream

       --query (string)

       A JMESPath query to use in filtering the response data.

       --profile (string)

       Use a specific profile from your credential file.

       --region (string)

       The region to use. Overrides config/env settings.
...

P.S. please also printout about using aws <command> help when someone types aws <command> --help

$ aws s3 ls --help
usage: aws [-h] [--profile PROFILE] [--debug]

optional arguments:
  -h, --help         show this help message and exit
  --profile PROFILE
  --debug

I didn't realise that help was an option until reading this issue.

[taras-sereda]

the issue is still there. No warnings, just a silent ignore of the option.

aws s3 ls --output json

my aws cli version is: 2.12.6

[fenixnet-net]

13 years later and still not fixed. Unbelievable.

[landsman]

Guys, can you recheck this, please? It's weird behavior.
For example, in Google Cloud CLI I can do this just fine...

cc @kyleknap, @jamesls, @elysahall, @aws-sdk-python-automation, @kdaily

[fenixnet-net]

I'm still seeing this on current aws CLI tools. In fact, it was searching for issues with the --output global option that brought me here.

It's ludicrous to me that the global option for choosing format exists, and yet fails silently. I'm stuck using awk to parse s3 ls output, which is brittle at best.

AWS runs something like 1/4 of the content on the Internet. How is it that 13 years on this issue has never been sorted out?

[nkirsch]

Same.

[Vortexdude]

13 years later and still not fixed. Unbelievable.

Yes, its still there,
they force me to do using aws s3api

[mxkpp]

Subtle warnings are not adequate IMO. aws s3 ls should simply return a non-zero error code if any of its arguments are not used.

[AndreC10002]

I'd like to bring this conversation back. The fact that the documentation (aws s3 ls help) shows options that simply aren't implemented it serious. I'm surprised this wasn't properly addressed after 10 years.