Promote ex_prefix to top-level storage SDK argument

[c-w]

Promote ex_prefix to top-level storage SDK argument

Description

Most storage drivers implement ex_prefix for list_container_objects and iterate_container_objects. As such, this pull request promotes ex_prefix to a top-level argument. For each driver that didn't previously implement ex_prefix, a fallback to filter objects client-side is has also been added.

Status

• done, ready for review

Checklist

• Code linting (build passed)
• Documentation (updated existing docstrings)
• Tests (updated existing tests)
• ICLA (committer)

[Kami]

Overall, I'm fine with this change, but another option would be to simply name the argument prefix (aka promote it to be part of the base API).

Then we could also have a fallback, if driver doesn't implement _filter_objects_by_prefix method (or similar) we would simply fall back to the implementation on the base driver which performs simple local filtering on the whole result set.

Since that method would perform filtering locally, it would probably also be a good idea to document that behavior (perhaps also emit a warning?) to avoid surprises.

[c-w]

@Kami Would promoting ex_prefix to prefix be a breaking API change or would we also keep the ex_prefix argument? If we're considering a breaking API change, I'd suggest to first merge this pull request (which is backwards compatible so potentially releasable via a patch version) and consider the argument promotion for a future change.

[Kami]

@c-w That's correct, but since it wasn't previously part of the base API, we probably should only update drivers and methods which supported ex_prefix argument before, to also support it for backward compatibility reasons going forward (at appears those are azure, cloudfiles, oss, dummy and s3 drivers).

And inside those methods we should probably just do something like prefix = prefix or ex_prefix and document in upgrade notes that the new argument name is prefix which has precedence over ex_prefix, which was left in place for backward compatibility reasons.

In short - I would just like to avoid adding ex_prefix to the base API since base API shouldn't really have any ex_ prefixed arguments and methods.

If you won't get a chance to make those changes in the near future, I can also look into it.

[c-w]

Thanks for the additional context @Kami. I'll work on the change and update this PR.

[codecov-io]

Codecov Report

Merging #1397 into trunk will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##            trunk    #1397      +/-   ##
==========================================
+ Coverage   86.57%   86.58%   +<.01%     
==========================================
  Files         364      364              
  Lines       76209    76224      +15     
  Branches     7441     7443       +2     
==========================================
+ Hits        65981    65996      +15     
  Misses       7402     7402              
  Partials     2826     2826

Impacted Files	Coverage Δ
libcloud/test/compute/test_ec2.py	97.97% <100%> (+0.01%)	⬆️
libcloud/compute/drivers/ec2.py	75.27% <100%> (+0.06%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ea7b36...1420a55. Read the comment docs.

[Kami]

I was thinking about doing this a bit differently, but that approach also works.

We just need to make sure that all the drivers which don't implement custom / server side based filtering call that method.

[Kami]

Nice work, LGTM 👍

When you get a chance it would also be good to add upgrade notes entry (document this new argument and advise users to migrate from ex_prefix to prefix if they currently use ex_prefix).

[c-w]

Added upgrade note in 16c692f.