add "limit" option to "ls" and "la" commands to return the specified number of objects instead of returning all objects.

[mozawa]

Currently "ls" command returns all objects within a bucket under a certain prefix using the delimiter '/' and there seems to be no option to limit the number of returned objects. If the contents of the main folder or a sub-folder is huge, it will continue to list all objects until it finishes. Its possible to take time to finish it and it could be overloaded the S3 server side.

This change is to add "max-ls-keys" option to "ls" command to return the specified number of objects instead of returning all objects.

[fviard]

@mozawa Thank you for your contribution, I think that the idea is good.
One point I see is that it could be better to rename the option to have something more generic, like "limit" so this option can also be re-used in different contexts.

[mozawa]

@fviard Thank you for the comment. You mean its better for the option to use "--limit" rather than "--max-ls-keys"? If you have any recommended option name, please let me know and I will change it.

[fviard]

Yes, I think that "--limit" instead of "--max-ls-keys" will more understandable and reusable.
I think that "limit" is good but it is the first thing that came to my mind, so it's not a problem if you think about a better "generic" name.

[mozawa]

@fviard Thanks. I made changes.

[mozawa]

@fviard I changed my mind a little bit. For my current changes, "ls" and "la" commands still will return all objects in the bucket even if the bucket is huge(e.g. millions of objects in the bucket) as default. This also could make the S3 server overloaded. So I'm going to change as follows.

• Have an internal default e.g. "100000"
• "ls " or "la" command will return objects up to an internal default. If the list is truncated, it indicates to let the user know.
• "ls" or "la" command with "--limit=10" return objects up to 10. If the list is truncated, it indicates to let the user know.
• If you want to get all objects, use "--limit=-1" or "--unlimit" option.

What do you think?

[fviard]

Hi @mozawa : I'm sorry for the big delay for my reply. (Probably my holidays are to blame).
I like the concept of --limit.
Here are my remarks:

• Regarding your previous message, I think that it is better to let "unlimited" be the default.
• That could be good if you could add the list of command that supports "--limit" in the limit's help string, as it is quite limited for the moment.
• The truncation message is good, but it could probably be reduced to a single line. No need to explain to the user that he has to remove the limit, but maybe if the first one you can add something like "because the settings limit was reached".
• For this message, I would advise you to use "warning()" instead of "output()" so the user is warned, without corrupting the "output" if the output was to be machine parsed.
• If think that your change has to be a little more complicated. Amazon S3 doesn't return more than 1000 keys in one go, and max-keys is only the maximum of one request, not the total of entries. So max-keys is supposed to be less or equal to max-keys.
  So, I think that what you have to do is add some math when limit is set, and add the number of entries returned at each requests and just set the "max-keys" for the last request when there is less than 1000 keys to return. Probably you can just set max-keys at each request and just decrease it by the number of entries returned. But take care that maybe amazon will raise an error if this number exceed 1000.

[mozawa]

@fviard Thank you for the advice. I made some changes based on your suggestion. Please review this again. Thanks!

[fviard]

Merged, thank you very much!

[mozawa]

@fviard Thank you. BTW, I'd like to know when will the next official release including this change be? The latest is 1.6.1 which was released on 2016-01-20 I guess.

[fviard]

@mozawa Time past fast ;-) There could be a new release soon, but I still would like to have 1 or 2 issues resolved before.