clickhouse dialect implementation

[cameronbraid]

Implementation of a ClickHouseQuery class

In:

packages/cubejs-schema-compiler/test/DataSchemaCompilerTest.js
packages/cubejs-schema-compiler/test/GraphBuilderTest.js :

I made some slight mods with the aim to make the DbRunner based tests plugable. However I ended up cloning these two tests for ClickHouse as there were some other changes required.

• to make it easier to read the geneated sql strings
• to not render all the debug logging by default but require you to set env.DEBUG_LOG=true
• to make DbRunner to construct the Query class and provide the adapter name

I added a docker-compose.yml to provide a local instance of postgres and clickhouse to run the graph builder and schema compiler tests. This required changing DbRunner to specify a user and pass as I couldn't get it working with a default user.

ClickhouseDbRunner

• basically re-implements ClickHouseDriver - would be nice to use the real driver here, however I just based the implementation on DnRunner which uses the pg api natively.

The expected failing tests are marked as skipped

[inikolaev]

Why numbers have to be converted to strings?

[cameronbraid]

lots of the cubejs-schema-compiler tests that deal with measures assert that they are strings.

examples

• https://github.com/statsbotco/cube.js/blob/73db73f874582ad86b3b01d4e93a04c3385e7003/packages/cubejs-schema-compiler/test/DataSchemaCompilerTest.js#L158
• https://github.com/statsbotco/cube.js/blob/73db73f874582ad86b3b01d4e93a04c3385e7003/packages/cubejs-schema-compiler/test/GraphBuilderTest.js#L318
• https://github.com/statsbotco/cube.js/blob/73db73f874582ad86b3b01d4e93a04c3385e7003/packages/cubejs-schema-compiler/test/GraphBuilderTest.js#L365
• https://github.com/statsbotco/cube.js/blob/73db73f874582ad86b3b01d4e93a04c3385e7003/packages/cubejs-schema-compiler/test/GraphBuilderTest.js#L400
• https://github.com/statsbotco/cube.js/blob/73db73f874582ad86b3b01d4e93a04c3385e7003/packages/cubejs-schema-compiler/test/GraphBuilderTest.js#L534

[cameronbraid]

I believe that this is due to the fact that the range of values that the javascript number type supports can overflow. Clickhouse caters for this by returning strings for UInt64,Int64 already. I assume postgres does the same, and assumed that this was the protocol for cube.js

[inikolaev]

As I understand this is only an issue with Int64 and UInt64 data types. How is it handled for Postgres which also has support for 64-bit numbers?

[cameronbraid]

@paveltiunov what should we be doing here with numbers vs strings ?

[paveltiunov]

We probably have some variations from driver to driver but as a rule of thumb all numeric types should be returned as strings due to JS can't handle correctly big integers and decimal numbers at all.

Also regarding Timestamp format: driver should always return only local dates in YYYY-DD-MMTHH:MM:SS.sss format as all conversions are done in SQL. There's only one case driver can return UTC: it's UTC timestamp representation in cube.js server local timezone. Otherwise there should be no 'Z' at the end.

[inikolaev]

Seems like something to be added to driver development guidelines :)

[paveltiunov]

@inikolaev Yep. Definitely worth adding it.

[cameronbraid]

Re timestamps - I am confused as for example

The following test uses timezone America/Los_Angeles

https://github.com/statsbotco/cube.js/blob/bde5ea1dc493f4af599c6fcdcaecbcd399dfcebd/packages/cubejs-schema-compiler/test/GraphBuilderTest.js#L302

However the timestamp asserted is UTC (including a Z)

https://github.com/statsbotco/cube.js/blob/bde5ea1dc493f4af599c6fcdcaecbcd399dfcebd/packages/cubejs-schema-compiler/test/GraphBuilderTest.js#L315

Actually - Every date in that test is formatted the same.

Is there an example

[paveltiunov]

@cameronbraid Yep. It's confusing. The thing is Postgres is that exception from the rule: it provides UTC timestamp representation in cube.js server local timezone. Actually it causing problems so we're considering to fix this in the Postgres driver itself.

[cameronbraid]

I've re-worked the PR

It now no longer changes any existing tests, just adds a copy of them specific to clickhouse.

I fixed the timestamp formatting for clickhouse

Lots of passing tests :

ClickHouse DataSchemaCompiler
✓ gutter (83ms)
✓ error
✓ duplicate member
✓ calculated metrics (157ms)
✓ dimension case (200ms)
✓ filtered dates in_date_range (58ms)
✓ filtered dates not_in_date_range (49ms)
✓ filtered dates on_the_date (50ms)
✓ filtered dates before_date (55ms)
✓ filtered dates after_date (54ms)
✓ export import
✓ contexts
✓ dashboard templates (50ms)

ClickHouse JoinGraph
- simple join
✓ simple join total (270ms)
- running total
- rolling
- rolling multiplied
- rolling month
- rolling count
✓ sql utils (177ms)
✓ running total total (162ms)
- running total ratio
- hll rolling
✓ calculated join (159ms)
✓ filter join (115ms)
✓ filter join not multiplied (137ms)
✓ having filter (92ms)
✓ having filter without measure (91ms)
- having filter without measure with join
✓ having filter without measure single multiplied (113ms)
✓ subquery (110ms)
- average subquery
✓ subquery without measure (59ms)
✓ min date subquery (142ms)
✓ min date subquery with error (77ms)
✓ subquery dimension with join (134ms)
- join rollup pre-aggregation
- join rollup total pre-aggregation
✓ user context (119ms)
✓ user context array (136ms)
✓ reference cube sql (111ms)
✓ Filtered count without primaryKey (107ms)
✓ builds geo dimension (85ms)
✓ applies measure_filter type filter (94ms)
✓ contains filter (101ms)
✓ year granularity (100ms)

35 passing (4s)
12 pending

✨ Done in 5.06s.

[paveltiunov]

@cameronbraid Looks great overall! I think we're ready to merge. Idea of creating test suite for drivers is great. Need to think about what's the best way to organize it so all tests can be reused.