Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement more plugins #12

Open
12 of 21 tasks
nils-braun opened this issue Aug 27, 2020 · 1 comment
Open
12 of 21 tasks

Implement more plugins #12

nils-braun opened this issue Aug 27, 2020 · 1 comment

Comments

@nils-braun
Copy link
Collaborator

nils-braun commented Aug 27, 2020

So far, basically all standard SELECT statements can be handled, because most of the RelNodes and RexNodes Apache Calcite will produce are already covered. However, there exist more of them and more advanced use cases might trigger them (leading to a NotImplementedError so far).

I would suggest to first find use cases of SQL, where those other classes are triggered, and then use them as a test case for the implementation.

Here is the list of java classes in the Apache Calcite project under rel/logical
and if they still need to be implemented:

  • LogicalAggregate
  • LogicalCalc -> a combination of project and filter, which is not in use as the corresponding optimization is not included (and also not needed)
  • LogicalCorrelate
  • LogicalExchange
  • LogicalFilter
  • LogicalIntersect
  • LogicalJoin
  • LogicalMatch
  • LogicalMinus
  • LogicalProject
  • LogicalRepeatUnion
  • LogicalSnapshot
  • LogicalSortExchange -> not needed with the current optimizations
  • LogicalSort
  • LogicalTableFunctionScan
  • LogicalTableModify -> not needed, as we do not want to have support table edits
  • LogicalTableScan
  • LogicalTableSpool -> no needed, as we do not want to have support table edits
  • LogicalUnion
  • LogicalValues
  • LogicalWindow
@nils-braun
Copy link
Collaborator Author

Some information on how to tackle such a task:

  1. Make sure to understand what the class does. This can be done by looking up the definition in the Apache Calcite source doe.
  2. Build a SQL statement use case for it and debug it in the dask-sql command line tool, with debug turned on. This will give you the full generated algebra and where exactly you still need to implement something.
  3. Add a new plugin in the rel/logical folder of dask_sql and also register it at the context.
  4. For testing, maybe add a code.interact(local=locals()) instead of a real convert function and find out, which parameters the java instance has. Then implement the logic with Dask API calls. Have a look into other plugins.
  5. Create some tests and do a PR :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant