s3-credentials list-bucket bucket-name command

[simonw]

Mainly useful for ease of testing that the temporary credentials created in #27 actually work.

[simonw]

Prototype:

@cli.command()
@click.argument("bucket")
@common_boto3_options
def list_bucket(bucket, access_key, secret_key, session_token, endpoint_url):
    "List content of bucket"
    s3 = make_client("s3", access_key, secret_key, session_token, endpoint_url)
    paginator = s3.get_paginator("list_objects_v2")
    keys = []
    for page in paginator.paginate(Bucket=bucket):
        for row in page["Contents"]:
            click.echo(json.dumps(row, indent=4, default=str))

[simonw]

Related:

• s3-credentials put-object and s3-credentials get-object commands #38

[simonw]

Looks like I accidentally checked in the implementation of this in f6fd163 - but it still needs documentation and tests.

s3-credentials/s3_credentials/cli.py

Lines 668 to 677 in 75ce3b2

	@cli.command()
	@click.argument("bucket")
	@common_boto3_options
	def list_bucket(bucket, **boto_options):
	"List content of bucket"
	s3 = make_client("s3", **boto_options)
	paginator = s3.get_paginator("list_objects_v2")
	for page in paginator.paginate(Bucket=bucket):
	for row in page["Contents"]:
	click.echo(json.dumps(row, indent=4, default=str))

[simonw]

Spotted a bug - throws an error if run against a bucket that contains no keys:

% s3-credentials list-bucket pysqlite3-wheels                   
Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/s3-credentials-J8M1ChYK/bin/s3-credentials", line 33, in <module>
    sys.exit(load_entry_point('s3-credentials', 'console_scripts', 's3-credentials')())
  File "/Users/simon/.local/share/virtualenvs/s3-credentials-J8M1ChYK/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/.local/share/virtualenvs/s3-credentials-J8M1ChYK/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/share/virtualenvs/s3-credentials-J8M1ChYK/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/simon/.local/share/virtualenvs/s3-credentials-J8M1ChYK/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/simon/.local/share/virtualenvs/s3-credentials-J8M1ChYK/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/simon/Dropbox/Development/s3-credentials/s3_credentials/cli.py", line 676, in list_bucket
    for row in page["Contents"]:
KeyError: 'Contents'

[simonw]

Also it currently outlines not-quite-newline-delimited-JSON (it's newline delimited but also pretty printed, which I don't think will work) - need to design better default and output options for it.

% s3-credentials list-bucket static.niche-museums.com | head -n 20
{
    "Key": "Griffith-Observatory.jpg",
    "LastModified": "2020-01-05 16:51:01+00:00",
    "ETag": "\"a4cff17d189e7eb0c4d3bf0257e56885\"",
    "Size": 3360040,
    "StorageClass": "STANDARD"
}
{
    "Key": "IMG_0353.jpeg",
    "LastModified": "2019-10-25 02:50:49+00:00",
    "ETag": "\"d45bab0b65c0e4b03b2ac0359c7267e3\"",
    "Size": 2581023,
    "StorageClass": "STANDARD"
}

[simonw]

I'm going to steal some of the output option ideas from sqlite-utils - in particular I will default to streaming regular JSON that looks like this:

% sqlite-utils rows ../datasette/fixtures.db searchable
[{"pk": 1, "text1": "barry cat", "text2": "terry dog", "name with . and spaces": "panther"},
 {"pk": 2, "text1": "terry dog", "text2": "sara weasel", "name with . and spaces": "puma"}]

Then offer --nl and --csv options too.

[simonw]

I just spotted list-buckets has the same not-quite-newline-delimited JSON output format, which is a bad default. I should fix that too.

[simonw]

This is looking good, needs documentation.

[simonw]

Documentation: https://github.com/simonw/s3-credentials/blob/07fa2073c784c69f2eb4245a664df161e933012f/README.md#list-bucket