One reason interoperability is cool

[skrawcz]

Hi.

I just wanted to highlight a cool use case using the interoperability between pandas and dask (and koalas on spark) to scale a DS workflow without anyone having to rewrite any code.

Blog post for context: https://multithreaded.stitchfix.com/blog/2022/02/22/scaling-hamilton/ & repo - https://github.com/stitchfix/hamilton/

The TL;DR:

1. dask [and koalas] implement a good portion of the pandas API
2. one can use this fact to write code agnostic of where it is run
3. this therefore allows out of the box scaling for free -- not to mention, avoiding vendor lock in.

I know 100% interoperability isn't a goal, but where there is interoperability it can lead to:

1. users not having to know/reimplement their code for a new system.
2. reduce vendor lock in -- giving end users more choice how their code is executed and where.

Thought I would surface in case it's relevant at all :)

[rgommers]

Thanks for sharing @skrawcz!

[GenevieveBuckley]

Really interesting, thanks!

[skrawcz]

Thanks. Happy to chat if anyone wants more detail; Hamilton isn't limited to creating pandas dataframes, you could use it to create numpy matrices, or any array type for that matter... :)