[MIG] storage_backend_s3: Migration to 16.0 #235
Conversation
- refactor the way to build the url (use a generic base_url). - make more generic the storage backend by moving specific feature in storage file - better name for variable "name" in store and retrieve method use "relative_path" instead - extra amazon S3 storage component in a separated module with test using vcrpy
…t with the specifiation of the type of file binary or base64
* support custom endpoint * refactor bucket handling * re-register vcrpy tests
Every time the selection field was loaded S3 API were called to retrieve regions. This can have a big impact on computing URLs and in any case is useless if we assume that regions do not change every now and then.
rven
force-pushed
the
16.0-mig-storage_backend_s3
branch
6 times, most recently
from
a3ea5e6
to
0e1f0ef
Compare
rven
force-pushed
the
16.0-mig-storage_backend_s3
branch
from
0e1f0ef
to
df8b8e7
Compare
|
/ocabot migration storage_backend_s3 |
| "license": "LGPL-3", | ||
| "installable": True, | ||
| "external_dependencies": {"python": ["boto3"]}, | ||
| "depends": ["storage_backend"], |
There was a problem hiding this comment.
@abdounasser202 Since the development of the new approach based of fsspecs takes a lot of time, I decided to keep storage_* addons in 16 next to the new fs_* addons even if they cover the same functionalities... It might also be a good idea to facilitate migration from one to another and not block projects that need these features now without being able to wait for the fs_* modules to be finalised.
There was a problem hiding this comment.
Thanks, @lmignon
then as I'm migrating storage_backend_ftp, it will then be better to have one depending on storage_backend and other one depending on fs_storage to keep both ?
There was a problem hiding this comment.
Thanks, @lmignon then as I'm migrating storage_backend_ftp, it will then be better to have one depending on storage_backend and other one depending on fs_storage to keep both ?
fs_storage supports natively ftp/ sftp / , S3, azure, DB, ... backends since it uses the fsspec python lib as client layer to communicate with external storage. We don't need additionnal addon by storage we want to support.
|
@vince-dynapps What's the status of this PR? |
|
There hasn't been any activity on this pull request in the past 4 months, so it has been marked as stale and it will be closed automatically if no further activity occurs in the next 30 days. |
github-actions
bot
added
the
stale
No description provided.