Do not mark file for distributed send as broken on EOF

[azat]

Changelog category (leave one):

• Bug Fix

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Do not mark file for distributed send as broken on EOF

Detailed description / Documentation draft:
ATTEMPT_TO_READ_AFTER_EOF should not be ignored if we receive it from remote (receiver), since:

• the sender will got ATTEMPT_TO_READ_AFTER_EOF when the client just go away,
  i.e. server had been restarted
• since Improve checksum checks for async INSERT into Distributed on the sender #18853 the file will be checked on the sender locally, and
  if there is something wrong with the file itself, we will receive
  ATTEMPT_TO_READ_AFTER_EOF not from the remote at first
  and mark batch as broken.

2021.01.15 08:59:57.379143 [ 8271 ] {} <Error> default.dist.DirectoryMonitor: Failed to send batch due to: Code: 32, e.displayText() = DB::Exception: Attempt to read after eof: while receiving packet from ch2.net:9000, Stack trace (when copying this message, always include the lines below): 
0. DB::throwReadAfterEOF() @ 0x86e5020 in /usr/bin/clickhouse
1. ? @ 0x870934d in /usr/bin/clickhouse
2. DB::Connection::receivePacket(std::__1::function<void (Poco::Net::Socket&)>) @ 0xf965556 in /usr/bin/clickhouse
3. DB::RemoteBlockOutputStream::RemoteBlockOutputStream(DB::Connection&, DB::ConnectionTimeouts const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Settings const&, DB::ClientInfo const&) @ 0xf6ecdbc in /usr/bin/clickhou
se
4. DB::StorageDistributedDirectoryMonitor::Batch::send() @ 0xf6e7775 in /usr/bin/clickhouse
5. DB::StorageDistributedDirectoryMonitor::processFilesWithBatching(std::__1::map<unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<unsigned long>, std::__1::allocator<std::__1::pair<unsigned long const, std 
::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > const&) @ 0xf6e2e04 in /usr/bin/clickhouse
6. DB::StorageDistributedDirectoryMonitor::run() @ 0xf6e0fd6 in /usr/bin/clickhouse
7. DB::BackgroundSchedulePoolTaskInfo::execute() @ 0xecc31e0 in /usr/bin/clickhouse
8. DB::BackgroundSchedulePool::threadFunction() @ 0xecc51d2 in /usr/bin/clickhouse
9. ? @ 0xecc5f92 in /usr/bin/clickhouse
10. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x870df5e in /usr/bin/clickhouse
11. ? @ 0x87119d3 in /usr/bin/clickhouse
12. start_thread @ 0x8ea7 in /lib/x86_64-linux-gnu/libpthread-2.31.so
13. clone @ 0xfdeaf in /lib/x86_64-linux-gnu/libc-2.31.so
 (version 21.1.1.5646)
2021.01.15 08:59:57.379211 [ 8271 ] {} <Error> default.dist.DirectoryMonitor: Marking a batch of 28 files as broken.

Cc: @alexey-milovidov (maybe you have some details on why ATTEMPT_TO_READ_AFTER_EOF has been added?)

[azat]

and I don't see how ATTEMPT_TO_READ_AFTER_EOF can be received while reading local file

Ok, I see them now (tons of throwReadAfterEOF in ReadHelpers.h and ReadBuffer.h), will rewrite the patch a little

[alexey-milovidov]

AST fuzzer (ASan) — Lost connection to server. See the logs

#19108