External database access unification (jdbc/odbc bridges) by alex-krash · Pull Request #3210 · ClickHouse/ClickHouse

alex-krash · 2018-09-25T15:02:36Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

The purpose of this PR is to implement connect to external databases via odbc/jdbc bridges, spawned as separate daemons.
I am working in parallel on JDBC-bridge here - https://github.com/alex-krash/clickhouse-jdbc-bridge

The final result of this PR would be like:

Table functions "odbc" and "jdbc" exists
Engines "ODBC" and "JDBC" exists
(optional) support for JDBC dictionaries implemented

While working on this PR, I've figured out following problems, which I would like to solve as well:

Existing ODBC bridge forces using of double-quotes as escape sequence for identifiers. It breaks compatibility with MySQL, particularly. I propose to add API method to odbc/jdbc bridge, that will return value of escape character (or add it to existsing /column_info handler, and rename it)
Nullability support abscence. For JDBC bridge I've implemented it (see below), maybe we should implement it for ODBC as well?

Example of jdbc interaction with MySQL:

mysql> CREATE TABLE `test`.`test` (
    ->   `int_id` INT NOT NULL AUTO_INCREMENT,
    ->   `int_nullable` INT NULL DEFAULT NULL,
    ->   `float` FLOAT NOT NULL,
    ->   `float_nullable` FLOAT NULL DEFAULT NULL,
    ->   `double` DOUBLE NOT NULL,
    ->   `double_nullable` DOUBLE NULL DEFAULT NULL,
    ->   `timestamp` TIMESTAMP NOT NULL,
    ->   `timestamp_nullable` TIMESTAMP NULL DEFAULT NULL,
    ->   `date` DATE NOT NULL,
    ->   `date_nullable` DATE NULL DEFAULT NULL,
    ->   PRIMARY KEY (`int_id`));
Query OK, 0 rows affected (0,09 sec)

mysql> insert into test (`int_id`, `float`, `double`, `timestamp`, `date`) VALUES (1,2,3, '2018-01-02 03:04:05', '2019-05-06');Query OK, 1 row affected (0,00 sec)

mysql> select * from test;
+--------+--------------+-------+----------------+--------+-----------------+---------------------+--------------------+------------+---------------+
| int_id | int_nullable | float | float_nullable | double | double_nullable | timestamp           | timestamp_nullable | date       | date_nullable |
+--------+--------------+-------+----------------+--------+-----------------+---------------------+--------------------+------------+---------------+
|      1 |         NULL |     2 |           NULL |      3 |            NULL | 2018-01-02 03:04:05 | NULL               | 2019-05-06 | NULL          |
+--------+--------------+-------+----------------+--------+-----------------+---------------------+--------------------+------------+---------------+
1 row in set (0,00 sec)

From clickhouse-client:

DESCRIBE TABLE jdbc('jdbc:mysql://localhost:3306/?user=root&password=root', 'test', 'test')

┌─name───────────────┬─type───────────────┬─default_type─┬─default_expression─┐
│ int_id             │ Int32              │              │                    │
│ int_nullable       │ Nullable(Int32)    │              │                    │
│ float              │ Float32            │              │                    │
│ float_nullable     │ Nullable(Float32)  │              │                    │
│ double             │ Float64            │              │                    │
│ double_nullable    │ Nullable(Float64)  │              │                    │
│ timestamp          │ DateTime           │              │                    │
│ timestamp_nullable │ Nullable(DateTime) │              │                    │
│ date               │ Date               │              │                    │
│ date_nullable      │ Nullable(Date)     │              │                    │
└────────────────────┴────────────────────┴──────────────┴────────────────────┘

10 rows in set. Elapsed: 0.031 sec.

SELECT *
FROM jdbc('jdbc:mysql://localhost:3306/?user=root&password=root', 'test', 'test') 

┌─int_id─┬─int_nullable─┬─float─┬─float_nullable─┬─double─┬─double_nullable─┬───────────timestamp─┬─timestamp_nullable─┬───────date─┬─date_nullable─┐
│      1 │         ᴺᵁᴸᴸ │     2 │           ᴺᵁᴸᴸ │      3 │            ᴺᵁᴸᴸ │ 2018-01-02 03:04:05 │               ᴺᵁᴸᴸ │ 2019-05-06 │          ᴺᵁᴸᴸ │
└────────┴──────────────┴───────┴────────────────┴────────┴─────────────────┴─────────────────────┴────────────────────┴────────────┴───────────────┘

1 rows in set. Elapsed: 0.055 sec.

alex-krash · 2018-09-25T15:03:49Z

Currently, new implementation of ODBC, via unified bridge interface is registered via "idbc" name, in order to being able to perform comparison between them.

alexey-milovidov · 2018-09-26T02:41:42Z

I propose to add API method to odbc/jdbc bridge, that will return value of escape character

Ok.

maybe we should implement it for ODBC as well?

Yes. There is no reason, why it is not implemented.

Currently, new implementation of ODBC, via unified bridge interface is registered via "idbc" name, in order to being able to perform comparison between them.

You can replace old ODBC to your implementation as soon as integration test passes.

alex-krash · 2018-10-01T09:26:16Z

Looks like I've finished with this PR.

A new API endpoint /identifier_quote has been added to odbc bridge. Now queries are escaped with appropriate character, and should work for MySQL ODBC (when segfault there disappears :) )
A support for Nullable was not implemented due to conflict with ODBCBlockInputStream and ExternalResultDescription (looks like nullability support there is something out of current PR, and can have significant side-effects):

DESCRIBE TABLE odbc('DSN=test', '', 'test')

┌─name─┬─type────────────┬─default_type─┬─default_expression─┐
│ id   │ Int32           │              │                    │
│ two  │ Nullable(Int32) │              │                    │
└──────┴─────────────────┴──────────────┴────────────────────┘

2 rows in set. Elapsed: 0.076 sec. 

krash :) select * from odbc('DSN=test', '', 'test')

SELECT *
FROM odbc('DSN=test', '', 'test') 

Received exception from server (version 18.14.0):
Code: 86. DB::Exception: Received from localhost:9000, 127.0.0.1. DB::Exception: Received error from remote server /?connection_string=DSN%3Dtest&columns=columns%20format%20version%3A%201%0A2%20columns%3A%0A%60id%60%20Int32%0A%60two%60%20Nullable%28Int32%29%0A&max_block_size=65536. HTTP status code: 500 Internal Server Error, body: ��Code: 50, e.displayText() = DB::Exception: Unsupported type Nullable(Int32), e.what() = DB::Exception,

Unfortunately, I've failed to run integration tests (did not start, python/Docket/whatever problems)
I propose to review the code, and merge it, turning off jdbc() function and JDBC engine, till clickhouse-jdbc-bridge is ready for production.

alesapin · 2018-10-03T09:10:03Z

+        std::string identifier = "";
+
+        SQLSMALLINT t;
+        SQLRETURN r = POCO_SQL_ODBC_CLASS::SQLGetInfo(hdbc, SQL_IDENTIFIER_QUOTE_CHAR, NULL, 0, &t);


Actually we know that the size of SQL_IDENTIFIER_QUOTE_CHAR is one or zero bytes. So maybe here we can allocate buffer (identifier) and request ODBC only once?

I propose to keep it consistent with bloody ODBC-interface, and not make too tricky and fragile.

alesapin · 2018-10-03T10:38:33Z

Style Erros:

dbms/src/TableFunctions/ITableFunctionXDBC.h:38: const std::string & connection_string_) const override { 
dbms/src/TableFunctions/ITableFunctionXDBC.h:55: const std::string & connection_string_) const override { 
dbms/src/Common/XDBCBridgeHelper.h:238:struct ODBCBridgeMixin {

Build logs:
https://pastebin.com/2iGXGYpJ

alesapin · 2018-10-04T08:58:56Z

Build still fails with errors: https://pastebin.com/EeVHeViY

alex-krash · 2018-10-04T13:20:13Z

@alesapin , sorry, my bad.
I don't have reproduce for this error while build.

alesapin · 2018-10-04T15:14:48Z

Ok, thank you. Now travis build fails (you can see error in "Details" link) and two ODBC integration tests fail.
https://pastebin.com/FuMwJTy4

alex-krash · 2018-10-04T15:29:31Z

Got it, thanks! Will spend more time on trying to launch integrations.

alesapin · 2018-10-04T15:35:05Z

Maybe it's possible to recognize error from stack trace:

E           QueryRuntimeException: Client failed! Return code: 232, stderr: Received exception from server (version 18.14.2):
E           Code: 1000. DB::Exception: Received from 172.18.0.3:9000. DB::Exception: Not found: dictionary.source.connection_string. Stack trace:
E          
E           0. /usr/bin/clickhouse(StackTrace::StackTrace()+0x13) [0x6873223]
E           1. /usr/bin/clickhouse(DB::TCPHandler::runImpl()+0xfbe) [0x2d6192e]
E           2. /usr/bin/clickhouse(DB::TCPHandler::run()+0x1a) [0x2d68cfa]
E           3. /usr/bin/clickhouse(Poco::Net::TCPServerConnection::start()+0xc) [0x6976acc]
E           4. /usr/bin/clickhouse(Poco::Net::TCPServerDispatcher::run()+0x118) [0x6976f98]
E           5. /usr/bin/clickhouse(Poco::PooledThread::run()+0xad) [0x6a3dfad]
E           6. /usr/bin/clickhouse(Poco::ThreadImpl::runnableEntry(void*)+0x3a) [0x6a3b72a]
E           7. /usr/bin/clickhouse() [0x705cdcf]
E           8. /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f8568228184]
E           9. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8567a4703d]

Intergration tests instruction https://github.com/yandex/ClickHouse/blob/master/dbms/tests/integration/README.md

alexey-milovidov · 2018-10-09T23:28:06Z

Nullability support abscence. For JDBC bridge I've implemented it (see below), maybe we should implement it for ODBC as well?

We want to add the support for Nullable types in ODBC and MySQL storages.

The logic will be, for MySQL and ODBC table engine:

convert NULL to default value, if it is not Nullable in ClickHouse table;
read NULL value if it is Nullable in ClickHouse table.

For mysql and odbc table functions:

introduce a setting to use Nullable types (by default) or to convert everything to non-Nullable (current behaviour).

alexey-milovidov · 2018-10-09T23:31:14Z

As I understand, it is disabled right now.
Nevertheless, I've merged this PR to make further development easier.

Things to do:

enable this feature;
prepare a manual on how to start using it after installing ClickHouse:
-- how to install JDK;
-- how to install JDBC bridge;
-- how to install example database with JDBC driver (maybe it will be ClickHouse itself);
add integration test.

alexey-milovidov · 2018-10-09T23:36:36Z

We should think how to make this feature available for users with absolute minimum installation steps.

create a jar with all dependencies inside?
package it in selfcontained application (is it possible)?
add it to the clickhouse-common-static package?

alex-krash · 2018-10-10T00:22:27Z

@alexey-milovidov , I've wrapped JDBC bridge into .deb package.
https://github.com/alex-krash/clickhouse-jdbc-bridge#installation
Maybe we should move this repo to Yandex one, and distribute as clickhouse-jdbc-bridge ?

Application consists of single jar with all the dependencies, and requires JDK to start this jar.

alexey-milovidov · 2018-10-12T02:53:22Z

We want to add the support for Nullable types in ODBC and MySQL storages.
The logic will be, for MySQL and ODBC table engine:
convert NULL to default value, if it is not Nullable in ClickHouse table;
read NULL value if it is Nullable in ClickHouse table.

This is implemented in #3362

For mysql and odbc table functions:
introduce a setting to use Nullable types (by default) or to convert everything to non-Nullable (current behaviour).

This is to do.

Alexander Krasheninnikov and others added 2 commits September 25, 2018 03:21

Initial

fe8affa

Implement jdbc/idbc bridge connectors

647d104

alex-krash and others added 3 commits September 27, 2018 18:23

Implement quote getter from ODBC

4b43126

New bridge interaction, quote identifier via api

ceea4b9

Add doc for jdbc table function

c3f10e7

alex-krash changed the title ~~[WIP] External database access unification (jdbc/odbc bridges)~~ External database access unification (jdbc/odbc bridges) Oct 1, 2018

alexey-milovidov requested a review from alesapin October 1, 2018 18:49

alexey-milovidov assigned alesapin Oct 1, 2018

alesapin reviewed Oct 3, 2018

View reviewed changes

alesapin added the can be tested label Oct 3, 2018

alex-krash added 2 commits October 3, 2018 13:44

fix formatting errors according code review

c7fee93

Added error codes, bridge startup, fixed destructor in XDBC stuff

82200bc

Removing destructor

d94b574

alex-krash and others added 3 commits October 9, 2018 18:03

Fix config reference, header inclusion and failed tests

27f2886

Update getIdentifierQuote.cpp

fe2c4e4

Update DictionarySourceFactory.cpp

54e933f

alexey-milovidov merged commit 1472e3a into ClickHouse:master Oct 9, 2018

alex-krash deleted the jdbc-bridge branch October 10, 2018 14:55

BayoNet mentioned this pull request May 16, 2019

DOCAPI-6206 ODBC table engine and odbc table functions descriptions #5299

Merged

Conversation

alex-krash commented Sep 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alex-krash commented Sep 25, 2018

Uh oh!

alexey-milovidov commented Sep 26, 2018

Uh oh!

alex-krash commented Oct 1, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alesapin Oct 3, 2018

Choose a reason for hiding this comment

Uh oh!

alex-krash Oct 3, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alesapin commented Oct 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alesapin commented Oct 4, 2018

Uh oh!

alex-krash commented Oct 4, 2018

Uh oh!

alesapin commented Oct 4, 2018

Uh oh!

alex-krash commented Oct 4, 2018

Uh oh!

alesapin commented Oct 4, 2018

Uh oh!

alexey-milovidov commented Oct 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Oct 9, 2018

Uh oh!

alexey-milovidov commented Oct 9, 2018

Uh oh!

alex-krash commented Oct 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Oct 12, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alex-krash commented Sep 25, 2018 •

edited

Loading

alesapin commented Oct 3, 2018 •

edited

Loading

alexey-milovidov commented Oct 9, 2018 •

edited

Loading

alex-krash commented Oct 10, 2018 •

edited

Loading