Integation with Infinidash

[abegong]

It's still early days, but reactive encryption for data transmission is emerging as an important part of many data stacks, especially as the streaming and batch use cases grow closer together.

We're starting this thread to capture requirements and begin planning for an upcoming Infinidash integration.

If you need more background on Infinidash, check out Ellen Körbes' overview here.

[timsankara]

I agree there's a need for reactive encryption for stream data as well. AWS Infinidash is the way to go.🤝

[abegong]

Proposal: for a v1 of this integration, what if we leave tight integration of the infinidash transpiling and hashing layer out of scope? Instead, we can add data validation pre- and post- those steps, without trying to explicitly bridge them.

That'll lower the level of technical work, and also give the Infinidash team time to resolve the delta sharding issue that the community is split over atm.