Apache Drill vs Apache Calcite

[ashishmittal13]

I have been exploring about apache drill and apache calcite. I know apache drill uses calcite internally for sql parsing and creating query plan, I wanted to know what is the major difference between the two. I see apache calcite is also able to make sql queries to multiple datasource.

[jnturton]

Calcite plans queries but cannot execute those plans. Drill adds this piece, a query execution engine.

[ashishmittal13]

That`s what I thought will be the difference, But here in the calcite documentation
https://calcite.apache.org/docs/tutorial.html
I see , it allows to even execute queries.

[jnturton]

It doesn't, you're looking at a tutorial based on a demo adapter.

Calcite-example-CSV is a fully functional adapter for Calcite that reads text files in CSV (comma-separated values) format.

It's a worked example to show you how you can add your own execution engine to Calcite, not production code that they ship.

[jnturton]

P.S. please feel welcome to visit to our Slack channels or mailing lists if you'd like to chat more.

[cgivre]

I'd add that Drill is a massively parallel processing (MPP) query layer which can interact with MANY different file formats and storage systems. As @jnturton said, Calcite is a query planner and lacks that execution engine. Drill in fact does use Calcite for it's query planning, but if you're looking to actually query data out of the box, you'll need Drill or some other tool.

Drill can also parallelize queries so it can query data at scale. Bottom line, to actually query data, you'll need a query engine, not just the planner.

[ashishmittal13]

Got it, Thanks for the detailed explanation. I would join slack channel, I have some more things which I want to discuss.

[paul-rogers]

Here's another way to look at it. Many engines use Calcite: Drill, Druid, Hive, ... Calcite handles the parsing and planning part. Each of those tools has their own way to run a query, as needed by the environment in which they run.