Cannot read all data. Bytes read: xxx. Bytes expected:xxx

[yishanchuan]

Hi guys. The exception just exist when I insert data into one table, other tables are ok.

Describe the bug
when I insert some data into a distributed table, I got "Cannot read all data." exeception

How to reproduce
I use 21.3.4.25 lts

Error message and/or stacktrace

 ru.yandex.clickhouse.except.ClickHouseException: ClickHouse exception, code: 33, host: localhost, port: 80; Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 10970. Bytes expected: 173359084.: (at row 4)
 (version 21.3.4.25 (official build))

	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:59)
	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:29)
	at ru.yandex.clickhouse.ClickHouseStatementImpl.checkForErrorAndThrow(ClickHouseStatementImpl.java:1052)
	at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:1022)
	at ru.yandex.clickhouse.Writer.send(Writer.java:151)
	at ru.yandex.clickhouse.Writer.send(Writer.java:205)
	at com.ctrip.hotel.settlement.exchange.common.clickhouse.CkHelper.bulkInsertRowBinary(CkHelper.java:322)
	at com.ctrip.hotel.settlement.exchange.business.DBCleaner.ClickhouseSyncTask.sendData(ClickhouseSyncTask.java:115)
	at com.ctrip.hotel.settlement.exchange.business.DBCleaner.AbstractSync.loopSendData(AbstractSync.java:525)
	at com.ctrip.hotel.settlement.exchange.business.DBCleaner.AbstractSync.loadData(AbstractSync.java:434)
	at com.ctrip.hotel.settlement.exchange.business.DBCleaner.AbstractSync.loadDataSplit(AbstractSync.java:335)
	at com.ctrip.hotel.settlement.exchange.business.DBCleaner.AbstractSync.loopSend(AbstractSync.java:285)
	at com.ctrip.hotel.settlement.exchange.business.DBCleaner.AbstractSync.syncDataTask(AbstractSync.java:129)
	at com.ctrip.hotel.settlement.exchange.business.DBCleaner.ClickhouseSyncTask.lambda$execute$0(ClickhouseSyncTask.java:49)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Throwable: Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 10970. Bytes expected: 173359084.: (at row 4)
 (version 21.3.4.25 (official build))

	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:54)
	... 19 more

0. DB::ReadBuffer::readStrict(char*, unsigned long) @ 0x864c94b in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
1. DB::DataTypeString::deserializeBinary(DB::IColumn&, DB::ReadBuffer&) const @ 0xe979e6a in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
2. DB::DataTypeNullable::deserializeBinary(DB::IColumn&, DB::ReadBuffer&) const @ 0xe968584 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
3. DB::BinaryRowInputFormat::readRow(std::__1::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, DB::RowReadExtension&) @ 0xf97bdaf in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
4. DB::IRowInputFormat::generate() @ 0xf970c58 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
5. DB::ISource::tryGenerate() @ 0xf8ff205 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
6. DB::ISource::work() @ 0xf8fedfa in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
7. DB::InputStreamFromInputFormat::readImpl() @ 0xd002aaf in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
8. DB::IBlockInputStream::read() @ 0xe8b0065 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
9. DB::InputStreamFromASTInsertQuery::readImpl() @ 0xec66810 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
10. DB::IBlockInputStream::read() @ 0xe8b0065 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
11. DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::__1::atomic<bool>*) @ 0xe8d579f in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
12. DB::NullAndDoCopyBlockInputStream::readImpl() @ 0xec58249 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
13. DB::IBlockInputStream::read() @ 0xe8b0065 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
14. void DB::copyDataImpl<std::__1::function<bool ()> const&, std::__1::function<void (DB::Block const&)> const&>(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::__1::function<bool ()> const&, std::__1::function<void (DB::Block const&)> const&) @ 0xe8d5ac6 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
15. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) @ 0xf1372d4 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
16. DB::HTTPHandler::processQuery(DB::Context&, DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::__1::optional<DB::CurrentThread::QueryScope>&) @ 0xf84da4a in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
17. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0xf851bde in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
18. DB::HTTPServerConnection::run() @ 0xf8daa0f in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
19. Poco::Net::TCPServerConnection::start() @ 0x11f84caf in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
20. Poco::Net::TCPServerDispatcher::run() @ 0x11f866c1 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
21. Poco::PooledThread::run() @ 0x120bcde9 in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
22. Poco::ThreadImpl::runnableEntry(void*) @ 0x120b8c4a in /usr/lib/debug/.build-id/23/ea76bdd7b9132c8fb55ec397db8bafbeb73c2b.debug
23. start_thread @ 0x7dd5 in /usr/lib64/libpthread-2.17.so
24. __clone @ 0xfdead in /usr/lib64/libc-2.17.so

[CurtizJ]

Please, can you show your insert query and the schema of table. From stacktrace seems that you use https://github.com/ClickHouse/clickhouse-jdbc, right?

[yishanchuan]

yes, I use clickhouse-jdbc the BinaryRow format for inserting data

[jhettler]

Hey guys, but the problem is not related just to JDBC, we have the same problem in ODBC integration engine. I will send details later today.

[yishanchuan]

sql:
INSERT INTO accountdb.ods_htl_accountemployee_all FORMAT RowBinary

ddl:

CREATE TABLE test.ods_htl_accountemployee
(

    `accountemployeeid` Int32,

    `eid` String,

    `employeename` Nullable(String),

    `email` Nullable(String),

    `mobile` Nullable(String),

    `telphone` Nullable(String),

    `fax` Nullable(String),

    `ccdesk` Nullable(String),

    `accountgroupid` Nullable(Int32),

    `accountpostinfoid` Int32,

    `operator` Nullable(String),

    `isvalid` Int8,

    `merchantids` Nullable(String),

    `datachange_lasttime` DateTime,

    `qq` Nullable(String),

    `foreman` Nullable(String),

    `supervisor` Nullable(String),

    `manager` Nullable(String)
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{layer}-{shard}/test/ods_htl_accountemployee',
 '{replica}')
PRIMARY KEY accountemployeeid
ORDER BY accountemployeeid
SETTINGS index_granularity = 8192

the ods_htl_accountemployee_all is a distributed table on multiple ods_htl_accountemployee table.

[jhettler]

Hey guys, but the problem is not related just to JDBC, we have the same problem in ODBC integration engine. I will send details later today.

In my case it was probably because of exhausted memory - I tried to load table with 62 million rows and it consumed at the end 52 GB of RAM. After adding few more GB of RAM everything looks fine. Question is, is it usual that it consumes such a big amount of memory?

[filimonov]

Question is, is it usual that it consumes such a big amount of memory?

No.

[filimonov]

For sure the reason is the same as described here: #23778 (comment)

As a workaround - you can try to slice the data by some simple WHERE condition to smaller chunks - it may help.
I.e. instead of

INSERT INTO clickhouse_table SELECT * FROM odbc(...) 

Try to do

INSERT INTO clickhouse_table SELECT * FROM odbc(...) where date = '2020-01-01';
INSERT INTO clickhouse_table SELECT * FROM odbc(...) where date = '2020-01-02';
INSERT INTO clickhouse_table SELECT * FROM odbc(...) where date = '2020-01-03';
-- ... 
-- etc.

Closing issue as a duplicate of #23778 (there is a lot more debug information there).

[jhettler]

This will probably work. When I tried to find the cause of this trouble, I tried to load smaller datasets and it worked without any troubles. Usually the trouble starts around higher tens of milions. I have never had this issues under ten milions of rows.