rgw: allow enforcing of maximum requested in listing when possible

[ivancich]

Some Swift clients assume that the number of items requested in a bucket listing will be returned if possible. This allows the underlying code to enfore that behavior, so Swift can get that behavior, but S3 does not have to.

Tracker: https://tracker.ceph.com/issues/44554

[matthewoliver]

Yeah the swift API and therefore clients I've seen expect up to the max container listing limit of a single req (10 000, but configurable), clients should see this value from the /info call.
If < max is returned, it's assumed we have it all, if == max, then there could be more, so client paginate.

Obivously there is also prefix, marker, end_marker, and reverse which changes potential output, but still that's up to the client to use and understand. (ie, prefix for psudo containers, or marker/end_marker to paginate by some bounds).

[ivancich]

@matthewoliver Thank you for weighing in.

I wish this API documentation were more clear: Swift API documentation

[matthewoliver]

Let me see what I can do re: Swift api on the Swift side :)

…

On Wed., 18 Mar. 2020, 1:53 am J. Eric Ivancich ***@***.*** wrote: @matthewoliver <https://github.com/matthewoliver> Thank you for weighing in. I wish this API documentation were more clear: Swift API documentation <https://docs.openstack.org/api-ref/object-store/?expanded=show-container-details-and-list-objects-detail#show-container-details-and-list-objects> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33864 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAK3SZ6JTNE4HMUVJZUV5LRH6FIDANCNFSM4LFGYH6A> .

[ivancich]

@mattbenjamin This now has both commits.

[smithfarm]

please change the base branch from "octopus" to "master"

base changed to master 28 days ago

[cbodley]

Yeah the swift API and therefore clients I've seen expect up to the max container listing limit of a single req (10 000, but configurable), clients should see this value from the /info call.

we advertise this value of RGW_LIST_BUCKETS_LIMIT_MAX=10000 in the swift /info api here:

void RGWInfo_ObjStore_SWIFT::list_swift_data(Formatter& formatter,
                                              const ConfigProxy& config,
                                              RGWRados& store)
{ 
  formatter.open_object_section("swift");
  formatter.dump_int("max_file_size", config->rgw_max_put_size);
  formatter.dump_int("container_listing_limit", RGW_LIST_BUCKETS_LIMIT_MAX);

i think we should return the value of rgw_max_listing_results here, and use that value for RGWListBucket_ObjStore_SWIFT::default_max as well. then we don't need to change the semantics of parse_value_and_bound(), which is used elsewhere

[ivancich]

we advertise this value of RGW_LIST_BUCKETS_LIMIT_MAX=10000 in the swift /info api here:

void RGWInfo_ObjStore_SWIFT::list_swift_data(Formatter& formatter,
                                              const ConfigProxy& config,
                                              RGWRados& store)
{ 
  formatter.open_object_section("swift");
  formatter.dump_int("max_file_size", config->rgw_max_put_size);
  formatter.dump_int("container_listing_limit", RGW_LIST_BUCKETS_LIMIT_MAX);

i think we should return the value of rgw_max_listing_results here, and use that value for RGWListBucket_ObjStore_SWIFT::default_max as well. then we don't need to change the semantics of parse_value_and_bound(), which is used elsewhere

The challenge is that rgw_max_listing_results is set to 1000, which works well for s3. However, the swift api specifies that 10,000 objects are returned when that many can be returned (see: https://docs.openstack.org/api-ref/object-store/?expanded=show-container-details-and-list-objects-detail). I think technically we'd be OK as long as we "advertise" that value, but that might be problematic with users.

But ... clients might assume that without configuration changes they'll get 10,000 back.

And ... how would we configure an RGW installation with both s3 and swift users, such that s3 users get 1,000 and swift users get 10,000 entries?

[ivancich]

QA Results:
http://pulpito.ceph.com/ivancich-2020-05-08_18:24:50-rgw-wip-eric-testing-distro-basic-smithi/
http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/

Any failures are from other sources and do not represent regressions.

[ivancich]

jenkins test make check

[ivancich]

@cbodley : did I address your concerns adequately in #33864 (comment) ? If not, can we talk in order to move this PR forward?

[mattbenjamin]

@ivancich @cbodley thanks for highlighting this, it's important to bring this upstream

[smithfarm]

jenkins test make check

[stale]

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

[stale]

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

[stale]

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

[stale]

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

[stale]

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

[github-actions]

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!