Schema Adapter: Supporting customized data models in Calcite

I would like to start from adapters since they are used to represent (and interpret) data. For distributed computing I always find only two issues are interesting (data abstraction and layout, and computation model), where data definitely are the most fundamental.

Why we want an adapter?

A schema adapter allows Calcite to read particular kinds of data, presenting the data as tables within a schema

ʕ•ᴥ•ʔ We can translate records (computation results) of various data formats into table records which Calcite can understand and use in query evaluation.

Basics: Model, Schema and Table

Model file: Provides metadata of the data model

Calcite uses JSON file to specify the model, e.g. toy model

SchemaFactory: Instantiates custom schema

1. Implements the interface org.apache.calcite.schema.SchemaFactory
2. Usually specified in the model file to power a custom schema ("factory": XXXSchemaFactory)

TableFactory: Instantiates custom tables (e.g. PartsTableFactory)

1. Implements the interface org.apache.calcite.schema.TableFactory. The create() method returns a custom table
2. Usually specified in the model file (under a default schema) to instantiate custom tables ("factory": PartsTableFactory)

Schema: Produces a list of tables

1. Implement a schema by extending org.apache.calcite.schema.impl.AbstractSchema

 The schema has no tables unless you override getTableMap()
 The schema has no functions unless you override getFunctionMultimap()
 The schema has no sub-schemas unless you override getSubSchemaMap()
 The schema is mutable unless you override isMutable()
 The name and parent schema are as specified in the constructor arguments.

2. What are the name and the parent schema used for?

   The two attributes can be used to form an expression to reference this schema in generated codes

Table: Provides iterators over records (e.g. PartsTable)

1. Implement the interface org.apache.calcite.schema.Table, or extends org.apache.calcite.schema.impl.AbstractTable for a quick start
2. The simplest: The ScannableTable interface
   1. Implement the scan() method to return an enumerator over the rows in the table
   2. The Enumerator interface: current(), moveNext(), reset(), close()

More on Tables and Views

Table interfaces: Scannable, translatable, and filterable

1. ScannableTable: see above
2. FilterableTable
   1. Implement the scan(DataContext root, List<RexNode> filters) method to return an enumerator over the rows in the table
      1. The list of filters is mutable. If the table can implement a particular filter, it should remove that filter from the list, else it should leave it in the list. Any filters remaining will be implemented by the consuming Calcite operator.
      2. How to support filtering using RexNode? (TODO)
3. TranslatableTable: specifies how the table is to be translated to a Relnode relational expression
   1. If a Table does not implement this interface, it will be converted to an org.apache.calcite.adapter.enumerable.EnumerableTableScan.
   2. Usually a Table implementing the interface will be used with custom rules that act on the corresponding Relnode.
      1. Implement toRel() to convert the table to a TableScan and return
      2. Like any table scan, it serves as a leaf node of a query tree
      3. Implement project() to return an enumerable over a given projection of the fields; This is called by generated codes

Creating an view

A view looks like a table when you are writing a query, but it doesn’t store data. It derives its result by executing a query. The view is expanded while the query is being planned.

Example

"tables": [
  {
    "name": "TABLE_NAME",
    "type": "view",
    "sql": "SELECT * FROM parts WHERE color = 'aka'"
  }
]