Clickhouse: ServerErrorHandler: Code: 101, e.displayText() = DB::NetException: Unexpected packet from client

[pythoninja]

Version

Sentry 21.8.0.dev0 (f0bbaa6)

Steps to Reproduce

I am not sure, I just updated to latest nightly (based on 2021-08-09) and checked the logs.

Expected Result

Not sure about that log message, may be it's ok.

Actual Result

Here is docker-compose logs -f output:

clickhouse_1                                | 2021.08.09 08:57:26.654912 [ 10255 ] {} <Error> ServerErrorHandler: Code: 101, e.displayText() = DB::NetException: Unexpected packet from client, Stack trace (when copying this message, always include the lines below):
clickhouse_1                                | 
clickhouse_1                                | 0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
clickhouse_1                                | 1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f4172d in /usr/bin/clickhouse
clickhouse_1                                | 2. ? @ 0x90263ad in /usr/bin/clickhouse
clickhouse_1                                | 3. DB::TCPHandler::runImpl() @ 0x90239e0 in /usr/bin/clickhouse
clickhouse_1                                | 4. DB::TCPHandler::run() @ 0x9025470 in /usr/bin/clickhouse
clickhouse_1                                | 5. Poco::Net::TCPServerConnection::start() @ 0xe3ac69b in /usr/bin/clickhouse
clickhouse_1                                | 6. Poco::Net::TCPServerDispatcher::run() @ 0xe3acb1d in /usr/bin/clickhouse
clickhouse_1                                | 7. Poco::PooledThread::run() @ 0x105c3317 in /usr/bin/clickhouse
clickhouse_1                                | 8. Poco::ThreadImpl::runnableEntry(void*) @ 0x105bf11c in /usr/bin/clickhouse
clickhouse_1                                | 9. ? @ 0x105c0abd in /usr/bin/clickhouse
clickhouse_1                                | 10. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
clickhouse_1                                | 11. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
clickhouse_1                                |  (version 20.3.9.70 (official build))
clickhouse_1                                | 2021.08.09 08:57:29.819605 [ 2384 ] {} <Error> ServerErrorHandler: Code: 101, e.displayText() = DB::NetException: Unexpected packet from client, Stack trace (when copying this message, always include the lines below):
clickhouse_1                                | 
clickhouse_1                                | 0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
clickhouse_1                                | 1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f4172d in /usr/bin/clickhouse
clickhouse_1                                | 2. ? @ 0x90263ad in /usr/bin/clickhouse
clickhouse_1                                | 3. DB::TCPHandler::runImpl() @ 0x90239e0 in /usr/bin/clickhouse
clickhouse_1                                | 4. DB::TCPHandler::run() @ 0x9025470 in /usr/bin/clickhouse
clickhouse_1                                | 5. Poco::Net::TCPServerConnection::start() @ 0xe3ac69b in /usr/bin/clickhouse
clickhouse_1                                | 6. Poco::Net::TCPServerDispatcher::run() @ 0xe3acb1d in /usr/bin/clickhouse
clickhouse_1                                | 7. Poco::PooledThread::run() @ 0x105c3317 in /usr/bin/clickhouse
clickhouse_1                                | 8. Poco::ThreadImpl::runnableEntry(void*) @ 0x105bf11c in /usr/bin/clickhouse
clickhouse_1                                | 9. ? @ 0x105c0abd in /usr/bin/clickhouse
clickhouse_1                                | 10. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
clickhouse_1                                | 11. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
clickhouse_1                                |  (version 20.3.9.70 (official build))

[BYK]

Not sure about that log message, may be it's ok.

Are you experiencing any issues? We don't get any of this on our CI pipeline so it might be a specific case for your system.

[pythoninja]

@BYK I do not know about issues at the moment because this is my first test instance. Maybe just a lot of spam messages from clickhouse. I can share ssh access if you want to inspect this error from logs. Or we can close this issue

[BYK]

I can share ssh access if you want to inspect this error from logs. Or we can close this issue

Unfortunately, that's not something we can do. Will close the issue for now as I don't see anything we can do right now. Let us know if you experience any issues please.

[bellini666]

Hi @BYK

Just noticed that exact issue being raised over and over again by clickhouse on my installation after upgrading to the latest version.

It doesn't seem to be affecting anything feature-wise (at least not from what I use), but it is there spamming on my logs every ~5 seconds

clickhouse_1                                | 2021.08.16 18:11:32.392769 [ 496 ] {} <Error> ServerErrorHandler: Code: 101, e.displayText() = DB::NetException: Unexpected packet from client, Stack trace (when copying this message, always include the lines below):
clickhouse_1                                | 
clickhouse_1                                | 0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
clickhouse_1                                | 1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f4172d in /usr/bin/clickhouse
clickhouse_1                                | 2. ? @ 0x90263ad in /usr/bin/clickhouse
clickhouse_1                                | 3. DB::TCPHandler::runImpl() @ 0x90239e0 in /usr/bin/clickhouse
clickhouse_1                                | 4. DB::TCPHandler::run() @ 0x9025470 in /usr/bin/clickhouse
clickhouse_1                                | 5. Poco::Net::TCPServerConnection::start() @ 0xe3ac69b in /usr/bin/clickhouse
clickhouse_1                                | 6. Poco::Net::TCPServerDispatcher::run() @ 0xe3acb1d in /usr/bin/clickhouse
clickhouse_1                                | 7. Poco::PooledThread::run() @ 0x105c3317 in /usr/bin/clickhouse
clickhouse_1                                | 8. Poco::ThreadImpl::runnableEntry(void*) @ 0x105bf11c in /usr/bin/clickhouse
clickhouse_1                                | 9. ? @ 0x105c0abd in /usr/bin/clickhouse
clickhouse_1                                | 10. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
clickhouse_1                                | 11. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
clickhouse_1                                |  (version 20.3.9.70 (official build))

[pythoninja]

Fresh installation (from scratch on clean vps (Almalinux 8.4)) produces same errors from clickhouse in docker-compose logs -f.

@BYK please debug it on your side, just run ./install.sh as mentioned in your instructions.

[BYK]

This looks like a known Clickhouse issue so I suggest following up with them there: ClickHouse/ClickHouse#1534

[codegain]

Same issue for me. From scratch with no data installed on Almalinux 8.4 - same errors in docker-compose logs -f. I'm not sure what this is causing, but I first checked out the 21.8.0 tag (was on master previously), did a ./install.sh: same error.
Then checked 21.7.0 out, ./install.sh and the error went away.

I noticed that the installation and docker-compose up -d of 21.8.0 took far longer (like ~5min) than the 21.7.0 install (like ~1min).

[BYK]

I noticed that the installation and docker-compose up -d of 21.8.0 took far longer (like ~5min) than the 21.7.0 install (like ~1min).

This is due to the newly introduced health checks on the self-hosted setup and requiring all dependencies to be healthy (instead of just the container being up).

Exploring the Clickhouse issues internally.