Does --inventory require some specific metadata fields to be present?

[theodrim]

I'm trying to use --inventory since my bucket have around ~22800 objects and doing just regular --s3 --bucket name --inventory --output name.html only generate up to 1000 objects (can this be overridden by the way? Since aws s3api list-object-versions can show all of the objects), but upon issuing ./dir_sizer.py --s3 --bucket 90poe-athens-go-mod-proxy --inventory --output ~/s3-cost/90poe-athens-go-mod-proxy.html I'm greeted with:

Using S3 Inventory report "test-inventory" generated 2022-08-25 03:00:00...
Scanning, gathered 1000 totaling 175.90 MiB...Traceback (most recent call last):
  File "/Users/alexk/src/dir_sizer/./dir_sizer.py", line 264, in <module>
    main()
  File "/Users/alexk/src/dir_sizer/./dir_sizer.py", line 190, in main
    for filename, size in load_files(opts, abstraction):
  File "/Users/alexk/src/dir_sizer/./dir_sizer.py", line 256, in load_files
    for filename, size in abstraction.scan_folder(opts):
  File "/Users/alexk/src/dir_sizer/s3_abstraction.py", line 310, in scan_folder
    for i, cur in enumerate(s3_list_objects(msg, opts, s3)):
  File "/Users/alexk/src/dir_sizer/s3_abstraction.py", line 274, in s3_list_objects
    'Size': int(row['Size']),
ValueError: invalid literal for int() with base 10: ''

What am I doing wrong here?

Thanks for the awesome project otherwise!

[seligman]

Interesting, this should just work. Is it possible to include the inventory report itself so I can look into the issue?

[theodrim]

Sure I don't think it have some secrets, since it's just a caching go proxy - but let me get back to you after approval.

[theodrim]

3eca360f-e360-4415-b8dc-83b67847855f.csv.gz
Here is inventory which caused a problem. Forgot to mentioned - I've used dir_sizer from master commit a03cd87

[seligman]

Perfect, thanks for the sample file, the code was breaking with delete markers. 62ba3f2 should fix it.

Feel free to re-open if it doesn't fix it on your side, or open a new bug if you run into one.