distributed used in production?

[rizplate]

Can someone point out to places which are using this in production?
Is it mature for large clusters doing out-of-core processing on terabyte size data?

[mrocklin]

Can someone point out to places which are using this in production?

Heh, I wish I had a good answer to this question. It's hard to gather this kind of information with open source software. I personally work with several groups that use dask.distributed in production, and many more people approach dask devs at conferences with stories (both positive and negative). Unfortunately I've learned not to divulge this kind of information unless explicitly told that it's ok.

A couple good examples would be the UK Met office (they handle operational weather forecasting) and BlackRock (which published a public blogpost at one point).

Is it mature for large clusters doing out-of-core processing on terabyte size data?

Yes, the larger datasets I've seen people play with are in the tens to hundred TB scale. I don't think I've ever seen anyone touch a petabyte (though the climate science folks are getting there).