Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing/plugins: migrate cloud fixtures to use fsspec filesystems #8917

Open
pmrowla opened this issue Jan 31, 2023 · 0 comments
Open

testing/plugins: migrate cloud fixtures to use fsspec filesystems #8917

pmrowla opened this issue Jan 31, 2023 · 0 comments
Labels
p3-nice-to-have It should be done this or next sprint testing Related to the tests and the testing infrastructure

Comments

@pmrowla
Copy link
Contributor

pmrowla commented Jan 31, 2023

Cloud fixtures currently depend on each plugin to implement the cloud/path-like utility methods (read_text/bytes, write_text/bytes, etc) and we do this by using the underlying libraries ourselves directly in dvc plugins (i.e. dvc_s3.tests.cloud uses boto directly). It would be easier to add new functionality if the plugin specified an fsspec filesystem to use, and then the generic dvc.testing.Cloud used the fsspec api calls instead. That way adding new functionality for things like retrieving specific version ID's can be added without having to use the underlying aws/gcs/adlfs libraries ourselves.

Basically, since we already added these things in the underlying fsspec implementations, it doesn't make sense for us to be re-implementing them all again in each plugin's Cloud test fixtures.

This would also let us implement a generic Cloud.remove() (currently we use DVC utils.fs.remove() for removing local files, but have no equivalent for the cloud/remote fixtures)

related: #8850

@pmrowla pmrowla added p3-nice-to-have It should be done this or next sprint testing Related to the tests and the testing infrastructure labels Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
p3-nice-to-have It should be done this or next sprint testing Related to the tests and the testing infrastructure
Projects
None yet
Development

No branches or pull requests

1 participant