[openstack] swift: add support for max_keys when retrieving dir items

[mtekel]

Currently there is no remapping in getting directory keys. This means that people are forced to add 'limit' explicitly into the calls. This is a problem when you try to have code that's compatible with multiple cloud vendors, as this breaks some other APIs, like Google Cloud Storage.

Raised PR, but it needs a bit more work. I am opening this issue so that someone else can take this over, as I have no SWIFT currently to test against.

[geemus]

@Ladas @TerryHowe perhaps one of you could provide some insight? Thanks!

[Ladas]

@geemus hm seems like @mtekel is trying to unify interface of Google storage with OpenStack Swift, so attribute names are translated from google format to others. Is that right?

The placement is probably wrong, there would need need to be some Google -> OpenStack interface adapter layer for this, so we know what is translated to what, and you could even pick it e.g. while building OpenStack storage object. Something like Openstack::Storage.new(GoogleInterfaceAdapter) or AmazonInterfaceAdapter, etc.

Do we use somewhere single pane of glass for multiple providers like this? Should it be solved in Fog? That is a question I guess. :-)

[geemus]

Ah, I guess I should have read more closely. Where it is feasible/makes sense, unifying seems good. This sounds, at a high level, like it could be a reasonable case to do this. But I'm not sure about exactly how we should do it (and/or which keys should be the common ones).

[mtekel]

Hi,

I thought that using max_keys is the standard, abstract fog way to declare how many items you want to receive. It is upon each implementation to transfer this into whatever parameter it needs to. Like e.g.:

• AWS https://github.com/fog/fog-aws/blob/master/lib/fog/aws/models/storage/directories.rb#L18
• internet-archive https://github.com/fog/fog/blob/master/lib/fog/internet_archive/models/storage/directories.rb#L19
• Google cloud storage https://github.com/fog/fog-google/blob/master/lib/fog/google/models/storage/directories.rb#L19

For openstack, you just need to translate this attribute into limit. That way, you don't have to explicitly include limit=<# of items you want to limit to> in higher level calls, like here: https://github.com/cloudfoundry/cloud_controller_ng/pull/452/files#diff-aa2798782d5185bb6e352d8fcec4aceaL14

In that specific case limit=1 is included specifically to tell openstack to limit number of returned elements. However including that attribute into request also breaks GCS API...

[geemus]

I think it may have (up to this point anyway) been simply coincidental that multiple services called it the same thing. Also the internet archive stuff is a s3 work-alike I think, so it would end up with a similar API by default. Anyway, it does seem like a standard key here could work and max_keys might be easiest, but it could just as well be limit if we really wanted. We'll just need to document it and work on adding the appropriate aliases.

[mtekel]

I suggest it is done this way. The suggested implementation doesn't force anyone to start using max_keys. You can still use limit the old way, so it will not break any existing code, but it adds option to use max_keys instead of explicit limit, making it possible to write code that's compatible with multiple cloud storage providers (which I believe is the goal of Fog - write one code that works on multiple providers without changes, because all the differences are implemented in Fog...).

[Ladas]

Having alias sounds reasonable( until we reach such that conflicts :-) ). I wonder where to put this information, so people know what attributes are supported across all models?

For more fun, not even OpenStack is consistent, e.g. keystone v3 uses per_page instead of limit and it doesn't use marker based pagination. :-)

[mtekel]

It would be nice if someone could pick up that PR and add tests so that it is mergable. Without this, functionality for GCS is still broken in some projects which want to be compatible with Openstack, like the referenced cloud foundry controller.

[plribeiro3000]

Closing in favor of fog/fog-openstack#51