Clickhouse query backend #1127

kszucs · 2017-08-21T20:10:41Z

This is the first draft for clickhouse backend #1120, based on impala.

cpcloud · 2017-08-22T13:11:07Z

@kszucs Thanks! Will do a deeper review this weekend.

Couple of minor things:

If you enable CircleCI and Appveyor on your fork you can run builds faster because you get your own set of docker containers to run (as opposed to using those allocated to the ibis-project user).
It looks like there might be conflicting ports. The Impala docker image has fs.default.name set to port 9000, which it looks like is also the port that clickhouse uses. I'm happy to change the default in the Impala docker image, or if it's easier for you to set a different port then that's also a viable solution. I can do whatever requires less work for you.

cpcloud

@kszucs This is a good start, thanks for doing this!

My main concern is that there appears to be a lot of copy paste from the impala backend. I understand why you took this approach, but it's very difficult to review since it's hard to pick out what's Clickhouse specific vs not.

I would suggest implementing the bare minimum to get a viable SELECT operation. So, leave out DDL, special types like arrays, and window function support.

The main focus of this PR should be the client, scalar types, and a bare bones SQL translator.

cpcloud · 2017-08-27T14:12:36Z

ci/datamgr.py

+        df = pd.read_csv(path, index_col=None, header=0, dtype=dtype)
+        if table == 'functional_alltypes':
+            df = df.rename(columns={'Unnamed: 0': 'Unnamed_0'})
+            cols = ['date_string_col', 'string_col']


These should already be strings.

This is just a temporary workaround, I'd like to use zkostyan/clickhouse-sqlalchemy with pandas to_sql, but there are a couple of issues.

cpcloud · 2017-08-27T14:12:47Z

ci/datamgr.py

+            df[cols] = df[cols].astype(str)
+            df.timestamp_col = df.timestamp_col.astype('datetime64[s]')
+        elif table == 'batting':
+            cols = ['playerID', 'teamID', 'lgID']


These should already be strings.

functional_alltypes.string_col is actually int64 which raises error on clickhouse side

cpcloud · 2017-08-27T14:13:04Z

ci/datamgr.py

+            cols = df.select_dtypes([float]).columns
+            df[cols] = df[cols].fillna(0).astype(int)
+        elif table == 'awards_players':
+            cols = ['playerID', 'awardID', 'lgID', 'tie', 'notes']


These should already be strings.

cpcloud · 2017-08-27T14:17:22Z

ibis/clickhouse/api.py

+    >>> client = ibis.clickhouse.connect(
+    ...     host=clickhouse_host,
+    ...     port=clickhouse_port,
+    ...     hdfs_client=hdfs,


There's no hdfs_client argument in this function.

cpcloud · 2017-08-27T14:19:26Z

ibis/clickhouse/api.py

+    Parameters
+    ----------
+    host : str, optional
+        Host name of the clickhoused or HiveServer2 in Hive


This is just clickhoused, correct?

Correct, c&p...

cpcloud · 2017-08-27T14:49:56Z

ibis/clickhouse/identifiers.py

@@ -0,0 +1,169 @@
+# Copyright 2014 Cloudera Inc.


This copyright header isn't necessary.

cpcloud · 2017-08-27T14:53:30Z

ibis/clickhouse/identifiers.py

+# limitations under the License.
+
+
+_identifiers = (


This should be a frozenset so there isn't a linear time average case lookup for identifier quoting.

cpcloud · 2017-08-27T14:54:44Z

ibis/clickhouse/tests/test_array.py

@@ -0,0 +1,115 @@
+# @pytest.fixture


Please remove this file and we'll add support for arrays in a follow up PR.

cpcloud · 2017-08-27T14:55:07Z

ibis/clickhouse/tests/test_client.py

@@ -0,0 +1,343 @@
+# Copyright 2014 Cloudera Inc.


No need for this copyright header.

cpcloud · 2017-08-27T14:57:02Z

ibis/clickhouse/client.py

@@ -0,0 +1,951 @@
+# Copyright 2014 Cloudera Inc.


Remove the copyright header.

kszucs · 2017-08-28T17:23:28Z

@cpcloud Thanks for the review! I'll fix these. I'm on vacation until Sunday. I intend to implement as much as I can during the next week, including basic ddl operations too (I can split this up to multiple PRs though).
At the end of the next week I'll clean up every todos and commented parts.

kszucs · 2017-09-03T19:08:15Z

I've resolved most of requests. Also I left a couple of commented test cases and TODO notes, because organizing them was a time consuming task. I want to implement or drop them before merging.

kszucs · 2017-09-05T07:03:02Z

Clickhouse supports a lot of hash functions, and I've implemented them in the compiler. The only problem is ibis expr is currently support only fnv.

wesm · 2017-09-05T13:58:36Z

I think @cpcloud is out of e-mail access this week so I'll take a look through this today and let you know if I have any more feedback, and then merge. Thanks for all your work on this

kszucs · 2017-09-05T15:45:57Z

@wesm Thanks in advance! I'm not sure that we can consider this PR ready to merge though.

wesm · 2017-09-05T17:44:42Z

I'd be OK with the clickhouse backend remaining WIP for a while as long as incremental patches do not break master. Let me review a bit later and see how things look

cpcloud · 2017-09-10T15:27:25Z

ibis/expr/operations.py

@@ -1472,22 +1472,20 @@ class OuterJoin(Join):
    pass


-class LeftSemiJoin(Join):
+class InnerSemiJoin(Join):


How is this different from a LEFT SEMI JOIN (or ANY LEFT JOIN in clickhouse parlance)?

If I understand the clickhouse JOIN documentation correctly it isn't.

If INNER is specified, the result will contain only those rows that have a matching row in the right table.

If ANY is specified and there are multiple matching rows in the right table, only the first one will be joined.

Composing those two statements yields (paraphrasing):

If INNER is specified and ANY is specified the result will contain only the first row from the left table where there's a matching row in the right table.

Adding LEFT into the mix:

LEFT is specified, any rows in the left table that don’t have matching rows in the right table will be assigned the default value - zeros or empty rows.

Combining that with ANY from above would yield the same statement composed from the definitions of ANY and INNER that I wrote above.

Is there something I'm missing here?

Actually I've got confused, but here are the test cases from 49 to 53

There are four type of joins so must be a difference.

So it turns out there's a difference :)

ANY INNER JOIN is most similar to a LEFT SEMI JOIN (which ibis supports)

ANY LEFT JOIN has no equivalent single node operation in ibis.

I say "most similar to" instead of "equivalent" because the test suite shows that you can select rows from the right table.

In every relational database that I've ever used that supports a LEFT SEMI JOIN syntax you can only select fields from the left table. Databases that don't implement a syntax for this operation implement it through an EXISTS query, which precludes selecting from the right table (since there is no right table in that case).

The reason for this difference is probably because the relational algebraic definition of a semijoin only includes tuples from the left table.

This means that to fully support clickhouse joins we need to introduce new types of joins that are only available to the clickhouse backend.

You can implement this kind of join in other databases using ROW_NUMBER() OVER () AS i on the left table and filtering out only those rows where i = 1 (the first row to be joined).

I think the solution here is to bring back the any_left_join etc methods and we can implement them in other databases down the line as desired.

kszucs · 2017-09-12T08:18:32Z

@cpcloud @wesm please review

cpcloud · 2017-09-14T12:59:50Z

@kszucs I will give this another review today.

cpcloud · 2017-09-14T16:21:55Z

ibis/expr/operations.py


-    """
+    def _get_schema(self):
+        return self.left.schema()


In light of the clickhouse test suite the schema of this operation isn't known until you select columns out of it, similar to the rest of ibis's join operations.

cpcloud · 2017-09-14T16:23:41Z

ibis/expr/api.py

@@ -2472,6 +2473,8 @@ def _table_drop(self, fields):
    join=join,
    cross_join=cross_join,
    inner_join=_regular_join_method('inner_join', 'inner'),
+    inner_semi_join=_regular_join_method('inner_semi_join', 'inner_semi'),
+    left_semi_join=_regular_join_method('left_semi_join', 'left_semi'),


Per my below comments, remove these and bring back the any_left/any_inner (all_* isn't necessary because those have the conventional join semantics.)

kszucs · 2017-09-18T18:08:08Z

@cpcloud done

kszucs · 2017-10-05T07:34:04Z

@cpcloud There is some packaging issue, setup.py wants to download toolz despite it's defined in the recipe.

Also, clickhouse packages have just landed in conda-forge, we might define them in ibis builds.

kszucs · 2017-10-05T07:40:45Z

Just to recap, here are a couple of todos I have in mind:

a better pandas wrapper around clickhouse-driver to prevent double changing of data orientation (columnar->row-wise->columnar)
support passing external tables to clickhouse
clickhouse async connection (thanks to @xzkostyan we now have aioch)
DDL operations
support arrays

Should I implement all of those in separate PRs?

cpcloud · 2017-10-10T14:49:50Z

@kszucs It appears there are multiple issues here with the most recent CI run:

Python 2.7 - no dependency issues, looks like a transient failure.
Python 3.4 - The SQLAlchemy clickhouse driver conda package isn't compatible with anything less than Python 3.5, this will also show up with Python 2.7 once we get to that point in the build.
Python 3.5 - Even though the SQLAlchemy clickhouse driver conda package is only compatible with Python 3.5+ it appears to require the enum34 package which is already in the stdlib as the enum package in 3.4 and up.
Python 3.6 - Same issue as Python 3.5.

kszucs · 2017-10-10T14:52:01Z

@cpcloud see xzkostyan/clickhouse-sqlalchemy#7

cpcloud · 2017-10-10T14:52:54Z

@kszucs Sweet, thanks for the reference.

cpcloud · 2017-10-10T14:54:20Z

@kszucs Is it possible to run the tests only on Python 2.7 for now?

cpcloud · 2017-10-10T14:54:37Z

Using the pip installed version, that is.

kszucs · 2017-10-10T14:56:14Z

I'll revert then, just a couple of minutes.

cpcloud · 2017-10-10T14:56:28Z

Alternatively, you can install enum34 with Python 3.4+ but I'm not sure if this kind of setup has well-defined behavior.

kszucs · 2017-10-10T16:30:51Z

@cpcloud It's green now.

cpcloud · 2017-10-10T16:38:32Z

@kszucs Is it possible for you to rebase on top of master?

…ings

…ally supported table aliases

…on conda forge

kszucs · 2017-10-13T18:14:54Z

@cpcloud Rebase done. Thanks for the help!

cpcloud · 2017-10-15T23:10:11Z

@kszucs Thanks for doing this! Excited to see use cases like this. Please keep the PRs and issues coming!

cpcloud self-assigned this Aug 22, 2017

cpcloud added the feature Features or general enhancements label Aug 22, 2017

cpcloud added this to the 0.12 milestone Aug 22, 2017

cpcloud self-requested a review August 22, 2017 13:11

cpcloud requested changes Aug 27, 2017

View reviewed changes

cpcloud reviewed Sep 10, 2017

View reviewed changes

cpcloud reviewed Sep 14, 2017

View reviewed changes

kszucs changed the title ~~[WIP] Clickhouse backend~~ Clickhouse backend Sep 26, 2017

kszucs added 24 commits October 13, 2017 20:13

clickhouse: removed importorskip from conftest

52806fe

clickhouse: minor cleanup

2a20131

clickhouse: fix parenthesize or condition in select builder

f7f4c31

clickhouse: added parse_url

60e8814

clickhouse: hash functions

e787d88

clickhouse: timestamp from integer

caa0b2e

clcikhouse: cleanup identifiers

a1c9d66

clickhouse: rename semi joins to any prefixed ones

a95a807

clickhouse: basic timedelta support; more verbose names for type mapp…

5e73b8a

…ings

flake8

b13e5da

clickhouse: turn off table prefixes in column formatting due to parti…

2809bbf

…ally supported table aliases

clickhouse: support timestamp truncate (Y, M, D, H, MI)

637dedd

clickhouse: zeroifnull; nullifzero

6656801

clickhouse: flake8

b9cba61

clickhouse: fixed client unsupported test case

8abe2a3

clickhouse removed accidentally committed ctags file

27bdc37

clickhouse: update requirements in setup.py

ff41290

clickhouse: extras require

c07a3c6

clickhouse: updated development dependencies, packages now available …

82c76ea

…on conda forge

clickhouse: removed unused client code

b638c0b

clickhouse: unused import

609aa9e

clickhouse: retrigger circleci builds

bab6ffb

clickhouse: install dependencies via pip instead of conda

2c5339b

clickhouse: flake8

4986f93

kszucs force-pushed the clickhouse branch from b5627c3 to 4986f93 Compare October 13, 2017 18:13

cpcloud approved these changes Oct 15, 2017

View reviewed changes

cpcloud closed this in 3324d1f Oct 15, 2017

kszucs deleted the clickhouse branch December 29, 2017 14:09

Clickhouse query backend #1127

Clickhouse query backend #1127

Conversation

kszucs commented Aug 21, 2017 • edited Loading

cpcloud commented Aug 22, 2017

cpcloud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs commented Aug 28, 2017

kszucs commented Sep 3, 2017 • edited Loading

kszucs commented Sep 5, 2017

wesm commented Sep 5, 2017

kszucs commented Sep 5, 2017

wesm commented Sep 5, 2017

cpcloud Sep 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs commented Sep 12, 2017

cpcloud commented Sep 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs commented Sep 18, 2017

kszucs commented Oct 5, 2017

kszucs commented Oct 5, 2017

cpcloud commented Oct 10, 2017 • edited Loading

kszucs commented Oct 10, 2017

cpcloud commented Oct 10, 2017

cpcloud commented Oct 10, 2017

cpcloud commented Oct 10, 2017

kszucs commented Oct 10, 2017

cpcloud commented Oct 10, 2017 • edited Loading

kszucs commented Oct 10, 2017

cpcloud commented Oct 10, 2017

kszucs commented Oct 13, 2017

cpcloud commented Oct 15, 2017

kszucs commented Aug 21, 2017 •

edited

Loading

kszucs commented Sep 3, 2017 •

edited

Loading

cpcloud Sep 10, 2017 •

edited

Loading

cpcloud commented Oct 10, 2017 •

edited

Loading

cpcloud commented Oct 10, 2017 •

edited

Loading