Input from cluster memory

[nils-braun]

This PR allows to input published datasets from the cluster memory.

This means, if you have published a dataset, e.g. with a distributed client

client.publish_dataset(my_table=df)

you can now register it with

CREATE TABLE some_name WITH (location='my_table', format='memory')

and start querying it as normal.

[mrocklin]

This is a nice intersection of an old Dask feature (published datasets) and the Dask-SQL context. I'm curious if there are things we should consider doing on the Dask side that we could learn from the context approach.

…

On Mon, Nov 2, 2020 at 2:50 AM Nils Braun ***@***.***> wrote: This PR allows to input published datasets from the cluster memory. This means, if you have published a dataset, e.g. with a distributed client client.publish_dataset(my_table=df) you can now register it with CREATE TABLE some_name WITH (location='my_table', format='memory') and start querying it as normal. ------------------------------ You can view, comment on, or merge this pull request online at: #68 Commit Summary - Create tables from cluster memory - Merge remote-tracking branch 'origin/main' into feature/input-from-cluster-memory - Use pytest for all the tests instead of unittest - Merge branch 'feature/use-pytest' into feature/input-from-cluster-memory - Add a test - Faster tests and stylefix - Merge branch 'feature/use-pytest' into feature/input-from-cluster-memory File Changes - *A* conftest.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-a31c7ed5d35f5ed8233994868c54d625b18e6bacb6794344c4531e62bd9dde59> (3) - *M* dask_sql/physical/rel/custom/create.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-ea4889d273c7235ad3aed884e19e0a0f956d4666327ecd80eb017463b9afe35a> (15) - *M* tests/integration/fixtures.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-4f5217b1bb9f04f8b152369ba94dd849d33ff76074ddee3fdcc3ed20596ce233> (247) - *M* tests/integration/test_complex.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-f9a14faecc13b797a4d8892e62524f8f4eb39193de4f57a8daa59857afa20d4e> (65) - *M* tests/integration/test_create.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-edd89c19b3a29a66ede8bf3f6ae73bf112424f9aecb56452aed34ff66b7469fa> (129) - *M* tests/integration/test_filter.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-bf277f19147f11cc459bd40233bc4d483506dfe040c42f804a885855fd1993aa> (28) - *M* tests/integration/test_function.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-793bddb97c18452095627edb532308f0d610b4d9f5c62961ccc5f306e51ae3f9> (48) - *M* tests/integration/test_groupby.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-c30044443c6a109e7f8a5b4d217bce126b4b30d0c6632950abd8760f3170b3ab> (269) - *M* tests/integration/test_join.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-76ace9120e49ad56eede78cf09b9b336fefdb596616b4308989077710fb06df7> (339) - *M* tests/integration/test_postgres.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-8f0ea23313d918723ccb63528d7ebf00d5a52746c95d455d7f20eeba8e0e7216> (416) - *M* tests/integration/test_rex.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-d3a939ae7d1b49dbff4a8d12130ab82ca3a81b9dfeed28c0ad5abba7cb21148b> (676) - *M* tests/integration/test_select.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-4343d025fcdecc8cf159f6058fd1dfe418f67034a3087dc2fc25e3689d854a61> (163) - *M* tests/integration/test_server.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-b768c0081084c8ef7f0778657cd21d2d1bd75074c286d404a71dd7a9cf138b0b> (176) - *M* tests/integration/test_show.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-527338f6f2def591874dac4690b72e1b99356fcf0bd5ec0ce5bc7393e9c8f98a> (131) - *M* tests/integration/test_sort.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-5382427d935cb819d13d0b315e172bfd4e7069b485bc8b09a0c656469546a7f7> (174) - *M* tests/integration/test_sqlite.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-2e7f95b7d7be270ff1a8118f817ea3e6663cdad273592e536a116c24e6d23c18> (212) - *M* tests/integration/test_union.py <https://github.com/nils-braun/dask-sql/pull/68/files#diff-72a1e87f22629dee4fbb6f7337d813af50f78e60ac92071b135515aece4aa095> (100) Patch Links: - https://github.com/nils-braun/dask-sql/pull/68.patch - https://github.com/nils-braun/dask-sql/pull/68.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#68>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTHZUQ4DYJZG4WNO6KTSN2FIDANCNFSM4THJ7RQQ> .

[codecov-io]

Codecov Report

Merging #68 into main will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main       #68   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           31        31           
  Lines         1196      1200    +4     
  Branches       156       157    +1     
=========================================
+ Hits          1196      1200    +4     

Impacted Files	Coverage Δ
dask_sql/physical/rel/custom/create.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7e22282...b354d62. Read the comment docs.

[nils-braun]

Thanks for commenting, @mrocklin (I am always amazed how fast you find out about what is going on in the ecosystem).

I am not sure if these things can be compared so easily because the context object is local to the process it is running on (or, if running a SQL server, still local to the application but can be controlled via SQL) and the published datasets are always shared among the full cluster.