Add support for Amazon s3 bucket package storage "paths"

[bpabel]

Has anyone tried distributing rez packages using s3 buckets?

Has anyone tried using other cloud storage systems to distribute rez packages?

Amazon s3 buckets are a fairly inexpensive, accessible, and secure way to distribute files to multiple locations, especially compared to some of the other cloud storage solutions. Plus, they support billing to the receiver, allowing one group to publish buckets, but not have to be on the hook for all the access and transfer.

[wizofe]

That is something I really want to look into it. It would be great if other people had experience with it :)

[bpabel]

When I get some time, I'm going to try to create a package repository plugin for s3. There was some discussion on the google group about possible local caching in the future, and this might be a good instance to try it out.

[JeanChristopheMorinRodeoFX]

@bpabel Do you think we could setup a little "meeting" with interested people to see if we can come up with a set of common requirements? Maybe @nerdvegas have an opinion on this. I have the feeling that if we all put an effort into it, we could have something pretty cool and flexible.

What do you guys think?

[bpabel]

@JeanChristopheMorinRodeoFX I'm open to that, but it's probably a bit premature for that until there's at least a working proof of concept. If you have any ideas or thoughts in particular, feel free to add them to this ticket.

I think the first step would just be to implement the existing Package Repository API -- https://github.com/nerdvegas/rez/blob/master/src/rez/package_repository.py -- using s3 buckets, even if it's slow and inefficient, and then maybe talk about ways to extend rez with things like caching to improve performance.

[lambdaclan]

Hello @bpabel and @JeanChristopheMorinRodeoFX what is the current status of adding support for S3 buckets as package repositories? Did any of you play with the idea further?

As we are also exploring the possibility of using S3 buckets as a non-file based target for rez-cp I will be interested to hear more details and help towards building a proof of concept.

Are you still willing to have a small meeting or a more in-depth chat via slack or something? I would love to hear your insight before diving deeper into this.

[alexey-pelykh]

Has anything changed here? PoC/researches/etc.

[JeanChristopheMorinPerso]

@alexey-pelykh nothing for now. But there's been some positive momentum recently. For example, #1242.

[alexey-pelykh]

Indeed! I would guess it would be a major refactor prior to actually diving into the S3 subject?

[nerdvegas]

Imho yes, otherwise you'd be shoehorning in that functionality. A

…

On Fri, Mar 4, 2022 at 12:49 AM Alexey Pelykh ***@***.***> wrote: Indeed! I would guess it would be a major refactor prior to actually diving into the S3 subject? — Reply to this email directly, view it on GitHub <#571 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSTIYZRTVANI4WNKO3TU6C7QNANCNFSM4GT46IWQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[JeanChristopheMorinPerso]

Linking to #1380

[JeanChristopheMorinPerso]

Well, the previous PR was closed. New PR is #1453.