Add integration tests for S3 storage driver

[c-w]

Add integration tests for S3 storage driver

Description

This pull request adds integration tests for the S3 storage driver.

Given that there are existing buckets in the test AWS account, this pull request also updates the base tests to ignore any containers that weren't created as part of the test.

Unlike for Azure and MinIO, on S3, create_container for an already existing container seems to not throw ContainerAlreadyExistsError so the assertion has been disabled in this driver's test suite. In the future we may want to consider fixing all these minor inconsistencies that are brought to light by the tests.

Status

• done, ready for review

Checklist (tick everything that applies)

• Code linting
• Tests
• ICLA

[codecov-commenter]

Codecov Report

Merging #1580 (57f794f) into trunk (fcab0c6) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##            trunk    #1580   +/-   ##
=======================================
  Coverage   83.01%   83.01%           
=======================================
  Files         394      394           
  Lines       84999    84999           
  Branches     9040     9040           
=======================================
  Hits        70558    70558           
  Misses      11374    11374           
  Partials     3067     3067           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fcab0c6...57f794f. Read the comment docs.

[Kami]

Yeah, I think including a special prefix on which we can filter on in the "clean up phase" is a good idea.

[Kami]

Just curious why we use boto3 for the clean up phase and not the Libcloud itself?

[c-w]

I figured that if we have a bug/regression in libcloud, we don't want to keep stray resources around so might as well use the official library to perform the cleanup.

[Kami]

I see the overall integration tests duration is ~14 minutes and test_objects_stream_iterable takes ~ 90 seconds.

Is there any way we could speed that up a bit? E.g. use a smaller object for that test or similar. Or we are intentionally testing large object size and we can't avoid slower duration?

[Kami]

Unlike for Azure and MinIO, on S3, create_container for an already existing container seems to not throw ContainerAlreadyExistsError so the assertion has been disabled in this driver's test suite. In the future we may want to consider fixing all these minor inconsistencies that are brought to light by the tests.

Yeah I agree, this seems like something we should eventually fix for consistency across drivers. Please feel free to open an issue for it if you get a chance.

On that note, what is the current behavior? It just returns an existing container with that name?

[c-w]

On that note, what is the current behavior? It just returns an existing container with that name?

Yes, create_container on an existing container returns a container with the same properties as the existing one. Here's a repro:

from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver

cls = get_driver(Provider.S3)
driver = cls(AWS_ACCESS_KEY_ID, AWS_ACCESS_KEY_SECRET)

containers = driver.list_containers()
print(driver.create_container(containers[0].name).name == containers[0].name)