ADLS Gen 2 spout

[lukemarsden]

It's increasingly obvious that ADLS gen 2 is a must-have for Microsoft sales folks. Need a spout that copies ADLS gen 2 data into pachyderm as it comes in.

How we can do ADLS gen 2 spouts into pachyderm-AML: get the terraform to:

1. set up an ADLS gen 2 storage account (i.e. "enable hierarchical namespace when creating a storage account")
2. create a service bus instance and a topic within it
3. configure an ADLS gen 2 event to push the appropriate events to the service bus
4. have terraform output appropriate credentials for both of the above that we can pass into pachyderm (have terraform write them into a k8s secret? or can we get the pachyderm spout to inherit an Azure service account somehow?)
5. create a pachyderm spout which reads from the queue and uses the notifications to go and download data from ADLS gen 2 into pachyderm (with caveats below)
6. tada! demo dropping some json into ADLS gen 2 and the spout runs and a pachyderm commit appears magically as an immutable AML dataset version that the data scientist can read - this is cool because ADLS gen 2 doesn't itself support versioning

making it production-grade will be hard, considerations:

• what happens when there's data in ADLS before the spout starts listening
• what happens when the spout gets disconnected
• what happens when there's terabytes of data and billions of files
• what happens if there are conflicting changes on both sides

[albscui]

Caveat from Andrei: ADLS gen 2 events only include blob creation rename and destroy, not modification.

[JoeyZwicker]

Caveat from Andrei: ADLS gen 2 events only include blob creation rename and destroy, not modification.

Is this a feature that we want ADLS to add. I know the VP fo Product for ADLS and can try to get it escalated. Are there any more specifics about the feature I might need to give him the exact info we're looking for?

[albscui]

Caveat from Andrei: ADLS gen 2 events only include blob creation rename and destroy, not modification.

Is this a feature that we want ADLS to add. I know the VP fo Product for ADLS and can try to get it escalated. Are there any more specifics about the feature I might need to give him the exact info we're looking for?

Events for blob/file modification would be useful for versioning specific objects. However, I don't think we need to escalate this until we have a concrete solution for consuming these events. So let's wait until we start building this out?