Log message for 'Connection reset by peer' could be improved #5183

citrin · 2019-05-03T14:28:15Z

We see a lot of messages like [ 23 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 104, e.displayText() = Connection reset by peer (version 19.5.3.1). It is almost impossible to trouble shoot them now.

It would be great if remote port:ip will be logged with this message.

The text was updated successfully, but these errors were encountered:

isaachui · 2019-07-05T20:47:10Z

We're also seeing a lot of these. Using 19.5.3.1 Really not sure where they're coming from

ghuname · 2020-02-03T09:01:49Z

Where we can find more info about this and all other errors, what is causing them and how to get more info about them?

alexey-milovidov · 2020-04-16T19:32:02Z

I've checked on master and I see that the issue is already solved. Now it looks like this:

2020.04.16 22:31:17.903683 [ 4920 ] {} ServerErrorHandler: Code: 210, e.displayText() = DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:35372), Stack trace (when copying this message, always include the lines below):

/home/milovidov/work/ClickHouse/build/../contrib/libcxx/include/exception:129: std::exception::capture() @ 0x1524a438 in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../contrib/libcxx/include/exception:109: std::exception::exception() @ 0x1524a405 in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../contrib/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) @ 0x20162223 in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../src/Common/Exception.cpp:29: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) @ 0x15244713 in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../src/Common/NetException.h:12: DB::NetException::NetException(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) @ 0x153861fd in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../src/IO/ReadBufferFromPocoSocket.cpp:38: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x153a0db7 in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../src/IO/ReadBuffer.h:53: DB::ReadBuffer::next() @ 0x1524c32b in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../src/IO/ReadBuffer.h:81: DB::TCPHandler::runImpl() @ 0x153733df in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../programs/server/TCPHandler.cpp:1237: DB::TCPHandler::run() @ 0x1537bc52 in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../contrib/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x2003748c in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../contrib/poco/Net/src/TCPServerDispatcher.cpp:114: Poco::Net::TCPServerDispatcher::run() @ 0x20037c9c in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../contrib/poco/Foundation/src/ThreadPool.cpp:199: Poco::PooledThread::run() @ 0x201e7493 in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../contrib/poco/Foundation/src/Thread.cpp:56: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x201e3e0d in /home/milovidov/work/ClickHouse/build/programs/clickhouse
/home/milovidov/work/ClickHouse/build/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345: Poco::ThreadImpl::runnableEntry(void*) @ 0x201e2c98 in /home/milovidov/work/ClickHouse/build/programs/clickhouse
start_thread @ 0x9669 in /usr/lib/x86_64-linux-gnu/libpthread-2.30.so
__clone @ 0x122323 in /usr/lib/x86_64-linux-gnu/libc-2.30.so
(version 20.4.1.1)

alexey-milovidov · 2020-04-16T19:33:58Z

Two changes:

socket address is printed;
stack trace is provided.

filimonov · 2020-04-17T15:30:16Z

But it looks like you tested some other scenario (it was already working properly before).
(Pay attention on Code: 1000, e.code() = 104)

2020.03.24 00:00:00.067459 [ 176 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 104, e.displayText() = Connection reset by peer
0. 0x100b49eb Poco::IOException::IOException(int) in /usr/bin/clickhouse 
1. 0xde90ea9 Poco::Net::ConnectionResetException::ConnectionResetException(int) in /usr/bin/clickhouse
2. 0xdea575a ? in /usr/bin/clickhouse
3. 0xdea3d13 Poco::Net::SocketImpl::receiveBytes(void*, int, int) in /usr/bin/clickhouse
4. 0xde81634 Poco::Net::HTTPSession::receive(char*, int) in /usr/bin/clickhouse
5. 0xde8169b Poco::Net::HTTPSession::get() in /usr/bin/clickhouse
6. 0xde76fd2 Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long) in /usr/bin/clickhouse
7. 0xde73aa8 Poco::BasicBufferedStreamBuf<char, std::__1::char_traits<char>, Poco::Net::HTTPBufferAllocator>::underflow() in /usr/bin/clickhouse
8. 0x10b3adba std::__1::basic_streambuf<char, std::__1::char_traits<char> >::uflow() ?
9. 0x10b41fa6 std::__1::basic_istream<char, std::__1::char_traits<char> >::get() ?
10. 0xde7a123 Poco::Net::HTTPRequest::read(std::__1::basic_istream<char, std::__1::char_traits<char> >&) in /usr/bin/clickhouse
11. 0xde7f8b1 Poco::Net::HTTPServerRequestImpl::HTTPServerRequestImpl(Poco::Net::HTTPServerResponseImpl&, Poco::Net::HTTPServerSession&, Poco::Net::HTTPServerParams*) in /usr/bin/c
lickhouse
12. 0xde7eab1 Poco::Net::HTTPServerConnection::run() in /usr/bin/clickhouse
13. 0xdeac637 Poco::Net::TCPServerConnection::start() in /usr/bin/clickhouse
14. 0xdeaca9d Poco::Net::TCPServerDispatcher::run() in /usr/bin/clickhouse
15. 0x101364bf Poco::PooledThread::run() in /usr/bin/clickhouse
16. 0x10132508 Poco::ThreadImpl::runnableEntry(void*) in /usr/bin/clickhouse
17. 0x10133da9 ? in /usr/bin/clickhouse
18. 0x7ea5 start_thread in /usr/lib64/libpthread-2.17.so
19. 0xfe8cd clone in /usr/lib64/libc-2.17.so

filimonov · 2020-04-17T16:01:18Z

Steps to reproduce:

# open a http connection
exec 3<>/dev/tcp/127.0.0.1/8123
# start sending request 
echo "GET / HTTP/1.0" >&3
# now kill the connection (to mimic connection reset)
sudo  ss -K dst 127.0.0.1 dport = 8123

winter7 · 2020-06-12T03:04:36Z

@filimonov can you please tell me what does Code: 1000, e.code() = 104 means? I found my server keep printing this error and stack trace like this (btw, I'm in eks)

2020.06.12 02:59:11.229712 [ 498 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 104, e.displayText() = Connection reset by peer, Stack trace (when copying this message, always include the lines below):

0. Poco::IOException::IOException(int) @ 0x104218bf in /usr/bin/clickhouse
1. Poco::Net::ConnectionResetException::ConnectionResetException(int) @ 0x102dfffd in /usr/bin/clickhouse
2. ? @ 0x102fd89a in /usr/bin/clickhouse
3. Poco::Net::SocketImpl::receiveBytes(void*, int, int) @ 0x102fbf93 in /usr/bin/clickhouse
4. Poco::Net::HTTPSession::receive(char*, int) @ 0x102cd128 in /usr/bin/clickhouse
5. Poco::Net::HTTPSession::get() @ 0x102cd193 in /usr/bin/clickhouse
6. Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long) @ 0x102c1daa in /usr/bin/clickhouse
7. Poco::BasicBufferedStreamBuf<char, std::__1::char_traits<char>, Poco::Net::HTTPBufferAllocator>::underflow() @ 0x102be6f8 in /usr/bin/clickhouse
8. std::__1::basic_streambuf<char, std::__1::char_traits<char> >::uflow() @ 0x10f41c8e in ?
9. std::__1::basic_istream<char, std::__1::char_traits<char> >::get() @ 0x10f49286 in ?
10. Poco::Net::HTTPRequest::read(std::__1::basic_istream<char, std::__1::char_traits<char> >&) @ 0x102c4e77 in /usr/bin/clickhouse
11. Poco::Net::HTTPServerRequestImpl::HTTPServerRequestImpl(Poco::Net::HTTPServerResponseImpl&, Poco::Net::HTTPServerSession&, Poco::Net::HTTPServerParams*) @ 0x102cb2a5 in /usr/bin/clickhouse
12. Poco::Net::HTTPServerConnection::run() @ 0x102c9c6c in /usr/bin/clickhouse
13. Poco::Net::TCPServerConnection::start() @ 0x10304f4b in /usr/bin/clickhouse
14. Poco::Net::TCPServerDispatcher::run() @ 0x103053db in /usr/bin/clickhouse
15. Poco::PooledThread::run() @ 0x104b2fa6 in /usr/bin/clickhouse
16. Poco::ThreadImpl::runnableEntry(void*) @ 0x104ae260 in /usr/bin/clickhouse
17. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
18. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
 (version 20.4.4.18 (official build))

adadfa2 · 2020-06-12T03:26:35Z

A large number of the following logs are printed on our logs, what are the solutions to solve this

2020.06.12` 11:12:13.865814 [ 138025 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 104, e.displayText() = Connection reset by peer, Stack trace (when copying this message, always include the lines below):

0. Poco::IOException::IOException(int) @ 0x102e95bb in /usr/bin/clickhouse
1. Poco::Net::ConnectionResetException::ConnectionResetException(int) @ 0xe07a369 in /usr/bin/clickhouse
2. ? @ 0xe08ec1a in /usr/bin/clickhouse
3. Poco::Net::SocketImpl::receiveBytes(void*, int, int) @ 0xe08d1d3 in /usr/bin/clickhouse
4. Poco::Net::HTTPSession::receive(char*, int) @ 0xe06aaf4 in /usr/bin/clickhouse
5. Poco::Net::HTTPSession::get() @ 0xe06ab5b in /usr/bin/clickhouse
6. Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long) @ 0xe060482 in /usr/bin/clickhouse
7. Poco::BasicBufferedStreamBuf<char, std::__1::char_traits<char>, Poco::Net::HTTPBufferAllocator>::underflow() @ 0xe05cf58 in /usr/bin/clickhouse
8. std::__1::basic_streambuf<char, std::__1::char_traits<char> >::uflow() @ 0x10dcc8ca in ?
9. std::__1::basic_istream<char, std::__1::char_traits<char> >::get() @ 0x10dd3ab6 in ?
10. Poco::Net::HTTPRequest::read(std::__1::basic_istream<char, std::__1::char_traits<char> >&) @ 0xe0635d3 in /usr/bin/clickhouse
11. Poco::Net::HTTPServerRequestImpl::HTTPServerRequestImpl(Poco::Net::HTTPServerResponseImpl&, Poco::Net::HTTPServerSession&, Poco::Net::HTTPServerParams*) @ 0xe068d71 in /usr/bin/clickhouse
12. Poco::Net::HTTPServerConnection::run() @ 0xe067f71 in /usr/bin/clickhouse
13. Poco::Net::TCPServerConnection::start() @ 0xe095ab7 in /usr/bin/clickhouse
14. Poco::Net::TCPServerDispatcher::run() @ 0xe095f1d in /usr/bin/clickhouse
15. Poco::PooledThread::run() @ 0x1036af7f in /usr/bin/clickhouse
16. Poco::ThreadImpl::runnableEntry(void*) @ 0x10366fc8 in /usr/bin/clickhouse
17. ? @ 0x10368869 in /usr/bin/clickhouse
18. start_thread @ 0x7dc5 in /usr/lib64/libpthread-2.17.so
19. clone @ 0xf776d in /usr/lib64/libc-2.17.so
 (version 20.3.4.10 (official build))

jacobrandall · 2020-09-24T22:12:56Z

Seeing a ton of these messages in all our clusters (a few every second per node). Would it make sense to change these to trace log level instead of error? (also running 20.3.4.10)

2020.09.25 03:32:09.024210 [ 327 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 104, e.displayText() = Connection reset by peer, Stack trace (when copying this message, always include the lines below):
0. Poco::IOException::IOException(int) @ 0x102e95bb in /usr/bin/clickhouse
1. Poco::Net::ConnectionResetException::ConnectionResetException(int) @ 0xe07a369 in /usr/bin/clickhouse
2. ? @ 0xe08ec1a in /usr/bin/clickhouse
3. Poco::Net::SocketImpl::receiveBytes(void*, int, int) @ 0xe08d1d3 in /usr/bin/clickhouse
4. Poco::Net::HTTPSession::receive(char*, int) @ 0xe06aaf4 in /usr/bin/clickhouse
5. Poco::Net::HTTPSession::get() @ 0xe06ab5b in /usr/bin/clickhouse
6. Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long) @ 0xe060482 in /usr/bin/clickhouse
7. Poco::BasicBufferedStreamBuf<char, std::__1::char_traits<char>, Poco::Net::HTTPBufferAllocator>::underflow() @ 0xe05cf58 in /usr/bin/clickhouse
8. std::__1::basic_streambuf<char, std::__1::char_traits<char> >::uflow() @ 0x10dcc8ca in ?
9. std::__1::basic_istream<char, std::__1::char_traits<char> >::get() @ 0x10dd3ab6 in ?
10. Poco::Net::HTTPRequest::read(std::__1::basic_istream<char, std::__1::char_traits<char> >&) @ 0xe0635d3 in /usr/bin/clickhouse
11. Poco::Net::HTTPServerRequestImpl::HTTPServerRequestImpl(Poco::Net::HTTPServerResponseImpl&, Poco::Net::HTTPServerSession&, Poco::Net::HTTPServerParams*) @ 0xe068d71 in /usr/bin/clickhouse
12. Poco::Net::HTTPServerConnection::run() @ 0xe067f71 in /usr/bin/clickhouse
13. Poco::Net::TCPServerConnection::start() @ 0xe095ab7 in /usr/bin/clickhouse
14. Poco::Net::TCPServerDispatcher::run() @ 0xe095f1d in /usr/bin/clickhouse
15. Poco::PooledThread::run() @ 0x1036af7f in /usr/bin/clickhouse
16. Poco::ThreadImpl::runnableEntry(void*) @ 0x10366fc8 in /usr/bin/clickhouse
17. ? @ 0x10368869 in /usr/bin/clickhouse
18. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
19. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
 (version 20.3.4.10 (official build))

John-belt · 2020-09-25T10:18:45Z

Same here tons of these messages, running 20.6.3. what can be possible reason for it??? and can we move it to trace logs.

filimonov · 2021-08-13T12:56:06Z

The error message was improved in the newer versions (most probably starting from in 21.3), and now it includes the ip.

2021.08.13 14:49:40.894239 [ 500767 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket (127.0.0.1:54520). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0.  /contrib/libcxx/include/exception:133: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0xc79b9cc in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
1.  /src/Common/Exception.cpp:59: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x56e30da in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
2.  /src/Common/NetException.h:12: DB::ReadBufferFromPocoSocket::nextImpl() @ 0xbfeb8c5 in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
3.  /src/IO/ReadBuffer.h:63: DB::skipToNextLineOrEOF(DB::ReadBuffer&) @ 0x5729e1c in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
4.  /src/Server/HTTP/HTTPServerRequest.cpp:116: DB::HTTPServerRequest::readRequest(DB::ReadBuffer&) @ 0xc0677c1 in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
5.  /src/Server/HTTP/HTTPServerRequest.cpp:41: DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0xc066e3f in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
6.  /src/Server/HTTP/HTTPServerConnection.cpp:0: DB::HTTPServerConnection::run() @ 0xc06630c in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
7.  /contrib/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0xc78a6a7 in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
8.  /contrib/poco/Net/src/TCPServerDispatcher.cpp:114: Poco::Net::TCPServerDispatcher::run() @ 0xc78ab3c in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
9.  /contrib/poco/Foundation/include/Poco/ScopedLock.h:36: Poco::PooledThread::run() @ 0xc7f6a79 in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
10. /contrib/poco/Foundation/include/Poco/SharedPtr.h:156: Poco::ThreadImpl::runnableEntry(void*) @ 0xc7f4786 in /home/mfilimonov/workspace/ClickHouse/kafka-producer-fix/build/programs/clickhouse
11. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
12. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
 (version 21.9.1.1)

filimonov · 2022-01-21T12:18:44Z

Still happening, see examples in #23858

filimonov added the usability label May 7, 2019

stale bot added the stale label Oct 20, 2019

alexey-milovidov removed the stale label Oct 23, 2019

filimonov mentioned this issue Apr 16, 2020

Avoid error message on RST #10274

Merged

ClickHouse deleted a comment from stale bot Apr 16, 2020

alexey-milovidov closed this as completed Apr 16, 2020

alexey-milovidov added the st-fixed label Apr 16, 2020

filimonov reopened this Apr 17, 2020

filimonov removed the st-fixed label Apr 17, 2020

filimonov closed this as completed Aug 13, 2021

filimonov reopened this Jan 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log message for 'Connection reset by peer' could be improved #5183

Log message for 'Connection reset by peer' could be improved #5183

citrin commented May 3, 2019

isaachui commented Jul 5, 2019

ghuname commented Feb 3, 2020

alexey-milovidov commented Apr 16, 2020 •

edited

alexey-milovidov commented Apr 16, 2020 •

edited

filimonov commented Apr 17, 2020 •

edited

filimonov commented Apr 17, 2020

winter7 commented Jun 12, 2020 •

edited

adadfa2 commented Jun 12, 2020 •

edited

jacobrandall commented Sep 24, 2020 •

edited

John-belt commented Sep 25, 2020

filimonov commented Aug 13, 2021 •

edited

filimonov commented Jan 21, 2022

Log message for 'Connection reset by peer' could be improved #5183

Log message for 'Connection reset by peer' could be improved #5183

Comments

citrin commented May 3, 2019

isaachui commented Jul 5, 2019

ghuname commented Feb 3, 2020

alexey-milovidov commented Apr 16, 2020 • edited

alexey-milovidov commented Apr 16, 2020 • edited

filimonov commented Apr 17, 2020 • edited

filimonov commented Apr 17, 2020

winter7 commented Jun 12, 2020 • edited

adadfa2 commented Jun 12, 2020 • edited

jacobrandall commented Sep 24, 2020 • edited

John-belt commented Sep 25, 2020

filimonov commented Aug 13, 2021 • edited

filimonov commented Jan 21, 2022

alexey-milovidov commented Apr 16, 2020 •

edited

alexey-milovidov commented Apr 16, 2020 •

edited

filimonov commented Apr 17, 2020 •

edited

winter7 commented Jun 12, 2020 •

edited

adadfa2 commented Jun 12, 2020 •

edited

jacobrandall commented Sep 24, 2020 •

edited

filimonov commented Aug 13, 2021 •

edited