DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding #4456

ermakus · 2019-02-20T12:44:38Z

Failure on load dictionary from PostgreSQL over unixODBC connector

ClickHouse client version 19.3.4.
ClickHouse server version 19.3.4

File /etc/odbcinst.ini

[DEFAULT]
Driver = staffcop
[staffcop]
Description         = PostgreSQL connection to staffcop
Driver              = PostgreSQL Unicode
Database            = staffcop
Servername          = 127.0.0.1
UserName            = staffcop
Password            = XXX
Port                = 5432
Protocol            = 9.3
ReadOnly            = No
RowVersioning       = No
ShowSystemTables    = No
ConnSettings        =

File /etc/odbcinst.ini

[PostgreSQL ANSI]
Description = PostgreSQL ODBC driver (ANSI version)
Driver		= /usr/lib/x86_64-linux-gnu/odbc/psqlodbca.so
Setup		= /usr/lib/x86_64-linux-gnu/odbc/libodbcpsqlS.so
Debug       = 0
CommLog		= 1
UsageCount  = 1

[PostgreSQL Unicode]
Description = PostgreSQL ODBC driver (Unicode version)
Driver      = /usr/lib/x86_64-linux-gnu/odbc/psqlodbcw.so
Setup       = /usr/lib/x86_64-linux-gnu/odbc/libodbcpsqlS.so
Debug		= 0
CommLog		= 1
UsageCount  = 1

error log

2019.02.20 12:42:49.499761 [ 26 ] {43f63aab-34fc-11e9-ae85-f403435768d8} <Error> executeQuery: Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding (from [::1]:57808) (in query: SELECT dictGetString('analytics_filter', 'name_ru', arrayJoin(agent_event_filters)) AS "analytics_filter.name", min(agent_event_local_time) AS local_time_min, max(agent_event_local_time) AS local_time_max, sum(agent_event_activity_time) AS activity_time_sum, sum(agent_event_idle_time) AS idle_time_sum, sum(agent_event_call_duration) AS call_duration_sum, count() AS record_count  FROM agent_event  WHERE agent_event_local_time <= 1550699999 AND agent_event_local_time >= 1550613600 GROUP BY dictGetString('analytics_filter', 'name_ru', arrayJoin(agent_event_filters)) ORDER BY record_count DESC  LIMIT 0, 50 FORMAT TabSeparatedWithNamesAndTypes), Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x6f13346]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3399d82]
2. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::startBridgeSync() const+0x3f8) [0x3504fb8]
3. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::getIdentifierQuotingStyle()+0x6d) [0x35039dd]
4. /usr/bin/clickhouse-server(DB::XDBCDictionarySource::XDBCDictionarySource(DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Block const&, DB::Context const&, std::shared_ptr<DB::IXDBCBridgeHelper>)+0x498) [0x5d35828]
5. /usr/bin/clickhouse-server() [0x5d36909]
6. /usr/bin/clickhouse-server() [0x5d36ad0]
7. /usr/bin/clickhouse-server(DB::DictionarySourceFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::DictionaryStructure const&, DB::Context&) const+0x642) [0x6768202]
8. /usr/bin/clickhouse-server(DB::DictionaryFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&) const+0x292) [0x6766e32]
9. /usr/bin/clickhouse-server(DB::ExternalDictionaries::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x4a) [0x62ba96a]
10. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1248) [0x62be078]
11. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFiles(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xab) [0x62c081b]
12. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadAndUpdate(bool)+0x5e) [0x62c0f4e]
13. /usr/bin/clickhouse-server(DB::ExternalLoader::init(bool)+0x6a) [0x62c2a3a]
14. /usr/bin/clickhouse-server(DB::ExternalDictionaries::ExternalDictionaries(std::unique_ptr<DB::IExternalLoaderConfigRepository, std::default_delete<DB::IExternalLoaderConfigRepository> >, DB::Context&, bool)+0x1d0) [0x62bb0d0]
15. /usr/bin/clickhouse-server(DB::Context::getExternalDictionariesImpl(bool) const+0x107) [0x628e9a7]
16. /usr/bin/clickhouse-server(DB::SyntaxAnalyzer::analyze(std::shared_ptr<DB::IAST>&, DB::NamesAndTypesList const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<DB::IStorage>) const+0x12f3) [0x640abb3]
17. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::shared_ptr<DB::IBlockInputStream> const&, std::shared_ptr<DB::IStorage> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x448) [0x62f60c8]
18. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x56) [0x62f7136]
19. /usr/bin/clickhouse-server(DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x7e7) [0x6303427]
20. /usr/bin/clickhouse-server(DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, DB::Context&, DB::QueryProcessingStage::Enum)+0x368) [0x62de6a8]
21. /usr/bin/clickhouse-server() [0x643bc24]
22. /usr/bin/clickhouse-server(DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)+0xfa) [0x643dc3a]
23. /usr/bin/clickhouse-server(DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&)+0x3344) [0x33b79f4]
24. /usr/bin/clickhouse-server(DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&)+0x3ca) [0x33ba23a]
25. /usr/bin/clickhouse-server(Poco::Net::HTTPServerConnection::run()+0x332) [0x7051412]
26. /usr/bin/clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x7049cbf]
27. /usr/bin/clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x16a) [0x704a09a]
28. /usr/bin/clickhouse-server(Poco::PooledThread::run()+0x77) [0x71265a7]
29. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0x7122468]
30. /usr/bin/clickhouse-server() [0xacbfecf]
31. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f53e68986ba]

2019.02.20 12:42:49.499875 [ 26 ] {43f63aab-34fc-11e9-ae85-f403435768d8} <Error> HTTPHandler: Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x6f13346]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3399d82]
2. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::startBridgeSync() const+0x3f8) [0x3504fb8]
3. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::getIdentifierQuotingStyle()+0x6d) [0x35039dd]
4. /usr/bin/clickhouse-server(DB::XDBCDictionarySource::XDBCDictionarySource(DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Block const&, DB::Context const&, std::shared_ptr<DB::IXDBCBridgeHelper>)+0x498) [0x5d35828]
5. /usr/bin/clickhouse-server() [0x5d36909]
6. /usr/bin/clickhouse-server() [0x5d36ad0]
7. /usr/bin/clickhouse-server(DB::DictionarySourceFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::DictionaryStructure const&, DB::Context&) const+0x642) [0x6768202]
8. /usr/bin/clickhouse-server(DB::DictionaryFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&) const+0x292) [0x6766e32]
9. /usr/bin/clickhouse-server(DB::ExternalDictionaries::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x4a) [0x62ba96a]
10. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1248) [0x62be078]
11. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFiles(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xab) [0x62c081b]
12. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadAndUpdate(bool)+0x5e) [0x62c0f4e]
13. /usr/bin/clickhouse-server(DB::ExternalLoader::init(bool)+0x6a) [0x62c2a3a]
14. /usr/bin/clickhouse-server(DB::ExternalDictionaries::ExternalDictionaries(std::unique_ptr<DB::IExternalLoaderConfigRepository, std::default_delete<DB::IExternalLoaderConfigRepository> >, DB::Context&, bool)+0x1d0) [0x62bb0d0]
15. /usr/bin/clickhouse-server(DB::Context::getExternalDictionariesImpl(bool) const+0x107) [0x628e9a7]
16. /usr/bin/clickhouse-server(DB::SyntaxAnalyzer::analyze(std::shared_ptr<DB::IAST>&, DB::NamesAndTypesList const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<DB::IStorage>) const+0x12f3) [0x640abb3]
17. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::shared_ptr<DB::IBlockInputStream> const&, std::shared_ptr<DB::IStorage> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x448) [0x62f60c8]
18. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x56) [0x62f7136]
19. /usr/bin/clickhouse-server(DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x7e7) [0x6303427]
20. /usr/bin/clickhouse-server(DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, DB::Context&, DB::QueryProcessingStage::Enum)+0x368) [0x62de6a8]
21. /usr/bin/clickhouse-server() [0x643bc24]
22. /usr/bin/clickhouse-server(DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)+0xfa) [0x643dc3a]
23. /usr/bin/clickhouse-server(DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&)+0x3344) [0x33b79f4]
24. /usr/bin/clickhouse-server(DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&)+0x3ca) [0x33ba23a]
25. /usr/bin/clickhouse-server(Poco::Net::HTTPServerConnection::run()+0x332) [0x7051412]
26. /usr/bin/clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x7049cbf]
27. /usr/bin/clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x16a) [0x704a09a]
28. /usr/bin/clickhouse-server(Poco::PooledThread::run()+0x77) [0x71265a7]
29. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0x7122468]
30. /usr/bin/clickhouse-server() [0xacbfecf]
31. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f53e68986ba]

Additional context

Same configuration works for most of users, but some server installs have persistent error clickhouse-odbc-bridge is not responding

clickhouse-odbc-bridge process available

ps aux | grep clickhouse-odbc-bridge
clickho+ 3121 0.0 0.0 150780 2864 ? Sl 12:42 0:00 /usr/bin/clickhouse-odbc-bridge odbc-bridge --http-port 9018 --listen-host localhost --http-timeout 1800000000

odbc connection works:

root@SC:/home/root777# isql -v "staffcop"
+---------------------------------------+
| Connected!
..

Any hint how to debug issue?

The text was updated successfully, but these errors were encountered:

alesapin · 2019-02-20T13:13:10Z

Same configuration works for most of users, but some server installs have persistent error clickhouse-odbc-bridge is not responding

This means that some hosts are OK and some have clickhouse-odbc-bridge is not responding?

You can set logging files for odbc-bridge with following lines in config.xml <logger> section:

        <odbc_bridge_log>/var/log/clickhouse-server/clickhouse-odbc-bridge.log</odbc_bridge_log>
        <odbc_bridge_errlog>/var/log/clickhouse-server/clickhouse-odbc-bridge.err.log</odbc_bridge_errlog>
        <odbc_bridge_level>trace</odbc_bridge_level>

ermakus · 2019-02-21T04:32:00Z

Yes, most of hosts works correctly.
We use same OS (Ubuntu 16.04LTS), same postgresql & clickhouse versions, but VMs/servers is very different (deployed on end users hardware as part of enterprise product).

Also there is propagated error in clickhouse.server.err.log
looks like clickhouse tried call startBridgeSync() somewhere in retry loop / exception handler and got huge trace (~500Mb)

Its happened not always, sometime there is only 1-10 loops and then exception reported.

2019.02.05 13:44:20.508827 [ 25 ] {} <Error> HTTPHandler: Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception: 
...
many times more

Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.wh
at() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x9fde646]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3317a42]
2. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::startBridgeSync() const+0x3f8) [0x3476658]
3. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::getIdentifierQuotingStyle()+0x6d) [0x347507d]
4. /usr/bin/clickhouse-server(DB::XDBCDictionarySource::XDBCDictionarySource(DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Block const&, DB::Context const&, std::shared_ptr<DB::IXDBCBridgeHelper>)+0x498) [0x9facd88]
5. /usr/bin/clickhouse-server() [0x9fade69]
6. /usr/bin/clickhouse-server() [0x9fae030]
7. /usr/bin/clickhouse-server(DB::DictionarySourceFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::DictionaryStructure const&, DB::Context&) const+0x642) [0x973d352]
8. /usr/bin/clickhouse-server(DB::DictionaryFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&) const+0x28e) [0x973bcee]
9. /usr/bin/clickhouse-server(DB::ExternalDictionaries::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x4a) [0x6f2fa5a]
10. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x16f8) [0x6f33678]
11. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFiles(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xab) [0x6f35a2b]
12. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadAndUpdate(bool)+0x5e) [0x6f3615e]
13. /usr/bin/clickhouse-server(DB::ExternalLoader::init(bool)+0x65) [0x6f37be5]
14. /usr/bin/clickhouse-server(DB::ExternalDictionaries::ExternalDictionaries(std::unique_ptr<DB::IExternalLoaderConfigRepository, std::default_delete<DB::IExternalLoaderConfigRepository> >, DB::Context&, bool)+0x1d0) [0x6f301c0]
15. /usr/bin/clickhouse-server(DB::Context::getExternalDictionariesImpl(bool) const+0xf0) [0x6f052e0]
16. /usr/bin/clickhouse-server(std::shared_ptr<DB::IFunctionBuilder> DB::FunctionFactory::createDefaultFunction<DB::FunctionDictGetString>(DB::Context const&)+0x27) [0x3836df7]
17. /usr/bin/clickhouse-server(std::_Function_handler<std::shared_ptr<DB::IFunctionBuilder> (DB::Context const&), std::shared_ptr<DB::IFunctionBuilder> (*)(DB::Context const&)>::_M_invoke(std::_Any_data const&, DB::Context const&)+0x23) [0x356a1a3]
18. /usr/bin/clickhouse-server(DB::FunctionFactory::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context const&) const+0x1d1) [0x7356381]
19. /usr/bin/clickhouse-server(DB::ActionsVisitor::visit(std::shared_ptr<DB::IAST> const&)+0xf01) [0x75d5e21]
20. /usr/bin/clickhouse-server(DB::ActionsVisitor::visit(std::shared_ptr<DB::IAST> const&)+0x1312) [0x75d6232]
21. /usr/bin/clickhouse-server(DB::ActionsVisitor::visit(std::shared_ptr<DB::IAST> const&)+0x1312) [0x75d6232]
22. /usr/bin/clickhouse-server(DB::ActionsVisitor::visit(std::shared_ptr<DB::IAST> const&)+0x1312) [0x75d6232]
23. /usr/bin/clickhouse-server(DB::ActionsVisitor::visit(std::shared_ptr<DB::IAST> const&)+0x1747) [0x75d6667]
24. /usr/bin/clickhouse-server(DB::ExpressionAnalyzer::getRootActions(std::shared_ptr<DB::IAST> const&, bool, std::shared_ptr<DB::ExpressionActions>&, bool)+0xd8) [0x7508d28]
25. /usr/bin/clickhouse-server(DB::ExpressionAnalyzer::getConstActions()+0xc9) [0x75097a9]
26. /usr/bin/clickhouse-server(DB::KeyCondition::getBlockWithConstants(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::SyntaxAnalyzerResult const> const&, DB::Context const&)+0x24f) [0x71e667f]
27. /usr/bin/clickhouse-server(DB::MergeTreeWhereOptimizer::MergeTreeWhereOptimizer(DB::SelectQueryInfo&, DB::Context const&, DB::MergeTreeData const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, Poco::Logger*)+0x62a) [0x728385a]
28. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::executeImpl(DB::InterpreterSelectQuery::Pipeline&, std::shared_ptr<DB::IBlockInputStream> const&, bool)+0x16db) [0x6f6b2fb]
29. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::shared_ptr<DB::IBlockInputStream> const&, std::shared_ptr<DB::IStorage> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x732) [0x6f6bd82]
30. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x56) [0x6f6caf6]
31. /usr/bin/clickhouse-server(DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x7e7) [0x6f78837]

2019.02.11 10:02:46.048353 [ 2 ] {} <Error> BaseDaemon: (from thread 90) Terminate called without an active exception
2019.02.11 10:02:46.071932 [ 2 ] {} <Error> BaseDaemon: ########################################
2019.02.11 10:02:46.105713 [ 2 ] {} <Error> BaseDaemon: (from thread 90) Received signal Aborted (6).
2019.02.11 10:02:46.207453 [ 2 ] {} <Error> BaseDaemon: 0. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f25cc394428]
2019.02.11 10:02:46.207472 [ 2 ] {} <Error> BaseDaemon: 1. /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f25cc39602a]
2019.02.11 10:02:46.207478 [ 2 ] {} <Error> BaseDaemon: 2. /usr/bin/clickhouse-server() [0x6dd6b8d]
2019.02.11 10:02:46.207670 [ 2 ] {} <Error> BaseDaemon: 3. /usr/bin/clickhouse-server(__cxxabiv1::__terminate(void (*)())+0x6) [0xaa01e26]
2019.02.11 10:02:46.207698 [ 2 ] {} <Error> BaseDaemon: 4. /usr/bin/clickhouse-server() [0xaa01e71]
2019.02.11 10:02:46.207705 [ 2 ] {} <Error> BaseDaemon: 5. /usr/bin/clickhouse-server(__cxa_rethrow+0x49) [0xaa017d9]
2019.02.11 10:02:46.207715 [ 2 ] {} <Error> BaseDaemon: 6. /usr/bin/clickhouse-server(DB::getCurrentExceptionMessage[abi:cxx11](bool, bool)+0x19f) [0x9fbf5df]
2019.02.11 10:02:46.218675 [ 2 ] {} <Error> BaseDaemon: 7. /usr/bin/clickhouse-server(DB::tryLogCurrentException(Poco::Logger*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x219) [0x9fbff19]
2019.02.11 10:02:46.218708 [ 2 ] {} <Error> BaseDaemon: 8. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadAndUpdate(bool)+0xa57) [0x6f36b57]
2019.02.11 10:02:46.218716 [ 2 ] {} <Error> BaseDaemon: 9. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadPeriodically()+0x2a) [0x6f37b5a]
2019.02.11 10:02:46.218722 [ 2 ] {} <Error> BaseDaemon: 10. /usr/bin/clickhouse-server() [0xaa3e73f]
2019.02.11 10:02:46.218733 [ 2 ] {} <Error> BaseDaemon: 11. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f25cce456ba]

EDIT: Probably second exception "Received signal Aborted (6)" not related to first one

ermakus · 2019-02-21T08:48:11Z

cat /var/log/clickhouse-server/clickhouse-odbc-bridge.log

2019.02.21 10:26:10.172293 [ 1 ] {} <Debug> ApplicationStartup: Initializing subsystem: Logging Subsystem
2019.02.21 10:26:10.207094 [ 1 ] {} <Information> ApplicationStartup: Starting up
2019.02.21 10:26:10.311844 [ 1 ] {} <Information> ApplicationStartup: Listening http://[::1]:9018
2019.02.21 10:29:23.572722 [ 2 ] {} <Information> Application: Received termination signal (Terminated)
2019.02.21 10:29:23.579108 [ 1 ] {} <Debug> ApplicationStartup: Received termination signal.
2019.02.21 10:29:23.579465 [ 1 ] {} <Debug> ApplicationStartup: Waiting for current connections to close.
2019.02.21 10:29:23.801719 [ 1 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem
2019.02.21 10:29:23.807449 [ 2 ] {} <Information> BaseDaemon: Stop SignalListener thread
2019.02.21 10:30:18.267115 [ 1 ] {} <Debug> ApplicationStartup: Initializing subsystem: Logging Subsystem
2019.02.21 10:30:18.297625 [ 1 ] {} <Information> ApplicationStartup: Starting up
2019.02.21 10:30:18.420133 [ 1 ] {} <Information> ApplicationStartup: Listening http://[::1]:9018

error log is empty

Looks like problem at clickhouse side. Or, maybe, ipv6 bound server?

alesapin · 2019-02-21T08:58:42Z

clickhouse-odbc-bridge always dies with server. Seems like server get abortion signal in ExternalLoader (class which load dictionaries including odbc) while loading dictionaries, after that it restarts and again loading dictionaries and get abortion. Can you collect some statistics about uptime of troublesome server (SELECT uptime())?

ermakus · 2019-02-21T10:34:24Z

Looks like abort is unrelated to this issue, server do not crash on this error and correctly return it to client

ODBC bridge is also available

root@SC:/home/root777# curl http://localhost:9018/ping
Ok.

Looks like that
checkBridgeIsRunning() can't ping bridge somehow, meanwhile curl can.

ermakus · 2019-02-21T10:52:40Z

Finally, solved by downgrading to 19.1.6

Unfortunately, can't figure out exact version with regression yet

alesapin · 2019-02-21T12:40:33Z

Looks like abort is unrelated to this issue, server do not crash on this error and correctly return it to client.

This is impossible, server always shutdown after abort signal.

ermakus · 2019-02-22T02:41:48Z

I think abort caused by another error, not related to clickhouse-odbc-bridge is not responding because it heppend 6 days later (see timestumps)
So it coincidence i think. Probably...

2019.02.05 13:44:20.508827 [ 25 ] {} <Error> HTTPHandler: Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding
...
2019.02.11 10:02:46.048353 [ 2 ] {} <Error> BaseDaemon: (from thread 90) Terminate called without an active exception

ODBCBridgeHelper: clickhouse-odbc-bridge is not responding do not crash server, just return error message to HTTP client

Example of such error from client side:

500 Error at /analytics/aggregate/agent_event/

Traceback (most recent call last):
  File "/usr/share/staffcop/lib/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "./util/decorators.py", line 13, in _wrapped_view
  File "./analytics/views.py", line 430, in aggregate
  File "./cubes/backends/clickhouse/flat.py", line 414, in aggregate
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2075, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1416, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
  File "./clickhouse/connector.py", line 224, in execute
  File "./clickhouse/connector.py", line 355, in _process_response
  File "/usr/share/staffcop/lib/infi/clickhouse_orm/database.py", line 163, in select
    r = self._send(query, settings, True)
  File "./clickhouse/connector.py", line 117, in _send
Exception: Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding

Number of lines varies from one to thousands.

AntonVozdvizhenskiy · 2019-04-23T14:31:05Z

On 19.4.2 revision 54416, with same simptoms, helped command system reload dictionaries.

k0st1an · 2019-06-03T12:34:25Z

On version 19.7.3.9 (official build) with same symptoms

k0st1an · 2019-07-01T11:14:05Z

The error occurs when you restart the ClickHouse server or restart the server itself.

alexey-milovidov · 2020-07-12T05:57:43Z

@alesapin Do I remember correctly that we have observed similar issue recently?

alesapin · 2020-07-15T15:28:12Z

Yes, probably it's a problem with a too-small timeout. It's fixed in #10990. Also we had a lot of fixes in the dictionaries loader.

alesapin · 2020-07-15T16:08:49Z

I think this issue can be closed.

MrSsunlight · 2021-03-09T03:53:24Z

clickhouse-odbc-bridge always dies with server. Seems like server get abortion signal in ExternalLoader (class which load dictionaries including odbc) while loading dictionaries, after that it restarts and again loading dictionaries and get abortion. Can you collect some statistics about uptime of troublesome server (SELECT uptime())?

I also encountered the same problem in the newer version, the specific link is https://github.com/ClickHouse/ClickHouse/issues/21322

ermakus added bug Confirmed user-visible misbehaviour in official release issue labels Feb 20, 2019

abyss7 removed the issue label Mar 6, 2019

alexey-milovidov assigned alesapin Apr 22, 2020

alexey-milovidov added the comp-foreign-db Integrations with other databases label Apr 22, 2020

filimonov added the comp-odbc label May 4, 2020

alesapin closed this as completed Jul 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding #4456

DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding #4456

ermakus commented Feb 20, 2019

alesapin commented Feb 20, 2019

ermakus commented Feb 21, 2019 •

edited

ermakus commented Feb 21, 2019

alesapin commented Feb 21, 2019

ermakus commented Feb 21, 2019

ermakus commented Feb 21, 2019

alesapin commented Feb 21, 2019 •

edited

ermakus commented Feb 22, 2019

AntonVozdvizhenskiy commented Apr 23, 2019

k0st1an commented Jun 3, 2019

k0st1an commented Jul 1, 2019

alexey-milovidov commented Jul 12, 2020

alesapin commented Jul 15, 2020 •

edited

alesapin commented Jul 15, 2020

MrSsunlight commented Mar 9, 2021

DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding #4456

DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding #4456

Comments

ermakus commented Feb 20, 2019

alesapin commented Feb 20, 2019

ermakus commented Feb 21, 2019 • edited

ermakus commented Feb 21, 2019

alesapin commented Feb 21, 2019

ermakus commented Feb 21, 2019

ermakus commented Feb 21, 2019

alesapin commented Feb 21, 2019 • edited

ermakus commented Feb 22, 2019

AntonVozdvizhenskiy commented Apr 23, 2019

k0st1an commented Jun 3, 2019

k0st1an commented Jul 1, 2019

alexey-milovidov commented Jul 12, 2020

alesapin commented Jul 15, 2020 • edited

alesapin commented Jul 15, 2020

MrSsunlight commented Mar 9, 2021

ermakus commented Feb 21, 2019 •

edited

alesapin commented Feb 21, 2019 •

edited

alesapin commented Jul 15, 2020 •

edited