Azure Storage integration

[v-a-s-a]

Prework

• Read and agree to the code of conduct and contributing guidelines.
• If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.
• Format your code according to the tidyverse style guide.

Proposal

I've seen hints (e.g. in #769) at this while reading through the saga of GCP integration. For us, the day has come — we have a workflow running on Azure that we are trying to port to targets.

I'd be happy to start just as @MarkEdmondson1234 did in #722, and produce Azure Storage utility functions based on the existing AWS S3 functions. I can also look at the tests, and try to replicate a similar harness for Azure. Based on discussion in #720, it looks like there is then work to be done integrating these into the targets class hierarchy.

How does this sound for an initial plan to PR?

[wlandau]

Yes, I am happy to support Azure storage if you would be willing to help with the utilities. I am on vacation until July 20, and I will be able to work through the technical details and PRs with you then. Looking forward to it.

[v-a-s-a]

Thanks Will — enjoy the time away!

[wlandau]

@noamross mentioned http://min.io/ (example: https://github.com/noamross/targets-minio-versioning), which claims to support Azure. @v-a-s-a, is this a reasonable workaround for you? It would be nice to try this before implementing a native solution in targets.

[v-a-s-a]

This is an interesting alternative. The licensing and pricing structure looks like a significant barrier for us. The alternative that is more realistic is simply checking everything into our repository with gittargets. Though going through AzureStor seems like the most sensible thing to do in the long term.

I've just come back to looking at this today, and I believe the implementation should be fairly lightweight. I've sorted out some decisions around the authentication, and I'll have some utility functions for you to look at soon. From there, you be able to comment as to whether it is worth continuing with the integration.

[v-a-s-a]

I've submitted an intiall pull request with a test and some utilities. It took a bit of time to decide how to to authentication, and I settled on using a service principal (similar to GCP's service account). I've been testing using Azure's free $200 tier on a private account.

It's a bare start, but I thought I would get something into the pull request to start. It does look very similar to GCP, and it would have been great if we could use a single API to handle both.

Take a look at the initial setup and test and let me know what you think. If it's too much to integrate into targets directly, perhaps we can do a separate package, similar to gittargets?

Otherwise, I'll continue trying to replicate the work done with GCP. Following the changes there has been helpful. The AzureR family of packages is decent, and almost all the functionality —e.g. file versioning and meta-data — we need is exposed there.

[alexkyllo]

Hi @v-a-s-a I am a data scientist at Microsoft and active user of some of the Azure-related R packages like {AzureStor}. I am interested in adopting {targets} in my workflow, backed by Azure Storage, and I came across this Issue and PR. It looks to me like it hasn't had activity for about five months. Are you still planning to work on it, and if so, would you like any help?

[JamesHWade]

I'm happy to help, too. Azure + targets would make my work life a lot easier.

[wlandau]

Thank you both. As @v-a-s-a said at the top, it would be great to have Azure storage utilities just like the ones for AWS and GCP (using AzureAuth and AzureStor ?). A PR would really help.

I am naive to Azure Storage, and I am wondering about a few specifics:

• For blob storage (I assume that's what we'll use, unless file storage is better), can we rely on strong read-after-write consistency?
• Would anything special need to happen for large files (> 5GB) or multi-part uploads? @MarkEdmondson1234's packages handle this automatically, but for AWS the solution was manual.

[alexkyllo]

I am naive to Azure Storage, and I am wondering about a few specifics:

• For blob storage (I assume that's what we'll use, unless file storage is better), can we rely on strong read-after-write consistency?

According to this, Azure Blob Storage "was designed to embrace a strong consistency model which guarantees that when the Storage service commits a data insert or update operation all further accesses to that data will see the latest update." So that's good news.

• Would anything special need to happen for large files (> 5GB) or multi-part uploads? @MarkEdmondson1234's packages handle this automatically, but for AWS the solution was manual.

For Block Blobs the {AzureStor} package uses the PUT blocklist operation: blob_upload_internal.R#L1 and according to this documentation the max size for an upload using PUT blocklist is about 190.7 TiB, so I don't believe we'll need to implement any special handling for large files.

[wlandau]

This issue has been inactive for a few months, and I was planning to crowd-source key parts of the implementation, so I am converting it to a discussion.

[jeffery-leirness-noaa]

I am very interested in Azure Storage integration with {targets}. I see this discussion was closed. Was there a resolution or was the idea abandoned? If abandoned, were there any workable alternative solutions that others found?

[wlandau]

Long-term, there are a couple different potential routes for this. If you have interest in native integration, I would be happy to review a pull request that implements the basic utility functions targets would need. (Examples: https://github.com/ropensci/targets/blob/main/R/utils_aws.R, https://github.com/ropensci/targets/blob/main/R/utils_gcp.R.) Otherwise, #1232, is going in a direction that would allow third-party storage systems to use custom cloud backends.

[jeffery-leirness-noaa]

Thanks for your response, Will, and for pointing out the specific AWS and GCP examples. I plan to take a closer look at the code and will make a pull request if can implement a feasible solution for Azure. I appreciate how comprehensive targets is and look forward to utilizing it more in my daily workflows. Well done.