Distributed table can't write data to remote shard (v18.14.9)

[PhantomPhreak]

We're experiencing problems with writing data to tables with Distributed engine after upgrading to version 18.14.9. After inserting data to distributed table, it can be successfully inserted to local shard, but can't be inserted to shard on remote server. I can see following in the log:

On server with data in table with Distributed engine:

2018.10.18 12:10:46.472687 [ 39 ] {} <Trace> distributed_events.Distributed.DirectoryMonitor: Started processing `/var/lib/clickhouse/data/<cluster>/<distributed_table>/<user_name>:<password>@<server_name>:9500/1.bin`
2018.10.18 12:10:46.472780 [ 39 ] {} <Trace> Connection (<remote_server>:9500, <remote_server_ip>): DB::NetException: Unexpected packet from server <remote_server>:9500, <remote_server_ip> (expected Pong, got Data)
2018.10.18 12:10:46.472798 [ 39 ] {} <Trace> Connection (<remote_server>:9500, <remote_server_ip>): Connection was closed, will reconnect.
2018.10.18 12:10:46.472831 [ 39 ] {} <Trace> Connection (<remote_server>:9500, <remote_server_ip>): Connecting. Database: (not specified). User: <user_name>
2018.10.18 12:10:46.473386 [ 39 ] {} <Trace> Connection (<remote_server>:9500, <remote_server_ip>): Connected to ClickHouse server version 18.14.9.
2018.10.18 12:10:46.475212 [ 39 ] {} <Error> <distributed_table>.Distributed.DirectoryMonitor: Code: 49, e.displayText() = DB::Exception: Logical error: empty block received as table structure, e.what() = DB::Exception, Stack trace:

on remote shard

2018.10.18 12:26:16.486439 [ 44 ] {42319601-feec-4ea8-ab11-5f75a6ae50fc} <Error> executeQuery: Code: 210, e.displayText() = DB::NetException: I/O error: Broken pipe:  while reading from socket (<remote_server_ip:54094), e.what() = DB::NetException (from <remote_server_ip>:54094) (in query: INSERT INTO <table> FORMAT Native), Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x99e9626]
1. /usr/bin/clickhouse-server(DB::WriteBufferFromPocoSocket::nextImpl()+0x740) [0x9a1f600]
2. /usr/bin/clickhouse-server(DB::WriteBuffer::next()+0x26) [0x3087466]
3. /usr/bin/clickhouse-server(DB::TCPHandler::runImpl()+0x791) [0x3094c11]
4. /usr/bin/clickhouse-server(DB::TCPHandler::run()+0x2b) [0x3095b1b]
5. /usr/bin/clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x9b67d7f]
6. /usr/bin/clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x16a) [0x9b6815a]
7. /usr/bin/clickhouse-server(Poco::PooledThread::run()+0x77) [0x9c84f37]
8. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0x9c80aa8]
9. /usr/bin/clickhouse-server() [0xa3da90f]
10. /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7feca00b80a4]
11. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fec9f6e087d]

2018.10.18 12:26:16.486613 [ 44 ] {42319601-feec-4ea8-ab11-5f75a6ae50fc} <Warning> TCPHandler: Client has gone away.
2018.10.18 12:26:16.486703 [ 44 ] {} <Information> TCPHandler: Processed in 30.001 sec.
2018.10.18 12:26:16.486724 [ 44 ] {} <Information> TCPHandler: Done processing connection.

[denisov-vlad]

We are having the same issue.

ch 18.14.9 / ubuntu 18.04.

[MykhailoSobol]

We are having the same issue.
ClickHouse server version 18.14.10 revision 54409
ubuntu 18.04.

[ztlpn]

Somehow I can't reproduce the issue, but judging from the number of upvotes it should be easy. Can anyone affected share their cluster configuration (<remote_servers>, what kind of ENGINE local tables use)?

[denisov-vlad]

@ztlpn
engine = ReplicatedMergeTree
cluster:

<CLUSTER_NAME>
      <shard>
        <internal_replication>true</internal_replication>
        <weight>0</weight>
        <replica>
          <host>hdd1</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>hdd2</host>
          <port>9000</port>
        </replica>
      </shard>
      <shard>
        <internal_replication>true</internal_replication>
        <weight>1</weight>
        <replica>
          <host>ssd1</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>ssd2/host>
          <port>9000</port>
        </replica>
      </shard>
      <shard>
        <internal_replication>true</internal_replication>
        <weight>1</weight>
        <replica>
          <host>ssd3</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>ssd4</host>
          <port>9000</port>
        </replica>
      </shard>
    </CLUSTER_NAME>
  </remote_servers>

[ztlpn]

@denisov-vlad Thanks, but still no luck.

Some additional questions:

1. Are the local tables replicated (I assume they are, because internal_replication is true)
2. Are there any materialized views attached to local/distributed tables?
3. Is the set of columns the same for local and distributed tables? Do the tables contain something other than ordinary columns (aliases, etc.)? Were the tables recently altered?
4. Were all servers in the cluster the same version (18.14.9) at the moment of error? What was the previous version?
5. What additional settings did you enable?
6. Does the error appear right away or did some data get successfully inserted? What is the time interval between the last success and the appearance of error?
7. Any additional details that you can think of.

[denisov-vlad]

1. Yes.
2. No
3. Same; no; they were recently created.
4. Yes.
5. I can show you config-preprocessed file via telegram.
6. No.
7. I'll send you a message through telegram.

[ztlpn]

The bug appeared when data was inserted into a Distributed table in the Native format. It is fixed in master and in the upcoming v18.14.11 fix-release.