Swarm joins unexpected `bad get: has Int64, requested Decimal64`

[alsugiliazova]

Got this exception on arm once.

Code: 170. DB::Exception: Received from localhost:9000. DB::Exception: Received from clickhouse3:9000. DB::Exception: Bad get: has Int64, requested Decimal64: While executing ReadFromObjectStorage. (BAD_GET)
(query: SELECT *
        FROM database_ede509c8_d48a_11f0_84b0_920006daec54.`namespace_f0865cf6_d48a_11f0_90ed_920006daec54.table_f0865e1a_d48a_11f0_8763_920006daec54` AS t1
        RIGHT ANTI JOIN s3('http://minio:9000/warehouse/data1/data/**.parquet', '[masked]:Secret(name='minio_root_user')', '[masked]:Secret(name='minio_root_password')') AS t2
         ON t1.timestamp_col = t2.timestamp_col ORDER BY tuple(*) SETTINGS object_storage_cluster_join_mode='allow', object_storage_cluster='replicated_cluster' FORMAT Values
)

Trace:

2025.12.09 00:10:10.568682 [ 1585 ] {dfb617ee-4c2c-43fc-8cdb-7f992d72e501} <Error> executeQuery: Code: 170. DB::Exception: Bad get: has Int64, requested Decimal64: While executing ReadFromObjectStorage. (BAD_GET) (version 25.8.9.20496.altinityantalya (altinity build)) (from 172.18.0.7:59630) (query 1, line 2) (in query: SELECT __table1.boolean_col AS boolean_col, __table1.long_col AS long_col, __table1.double_col AS double_col, __table1.string_col AS string_col, __table1.timestamp_col AS timestamp_col, __table1.date_col AS date_col, __table1.time_col AS time_col, __table1.timestamptz_col AS timestamptz_col, __table1.integer_col AS integer_col, __table1.float_col AS float_col, __table1.decimal_col AS decimal_col, __table2.boolean_col AS `t2.boolean_col`, __table2.long_col AS `t2.long_col`, __table2.double_col AS `t2.double_col`, __table2.string_col AS `t2.string_col`, __table2.timestamp_col AS `t2.timestamp_col`, __table2.date_col AS `t2.date_col`, __table2.time_col AS `t2.time_col`, __table2.timestamptz_col AS `t2.timestamptz_col`, __table2.integer_col AS `t2.integer_col`, __table2.float_col AS `t2.float_col`, __table2.decimal_col AS `t2.decimal_col` FROM icebergS3Cluster('replicated_cluster', 'http://minio:9000/warehouse/data3/', 'admin', '[HIDDEN]', 'Parquet', '`boolean_col` Nullable(Bool), `long_col` Nullable(Int64), `double_col` Nullable(Float64), `string_col` Nullable(String), `timestamp_col` Nullable(DateTime64(6)), `date_col` Nullable(Date), `time_col` Nullable(Int64), `timestamptz_col` Nullable(DateTime64(6, \'UTC\')), `integer_col` Nullable(Int32), `float_col` Nullable(Float32), `decimal_col` Nullable(Decimal(10, 2))', SETTINGS iceberg_metadata_file_path = 'metadata/00004-8104b757-641c-4291-84de-afd9c47886c1.metadata.json') AS __table1 ANTI RIGHT JOIN s3('http://minio:9000/warehouse/data1/data/**.parquet', 'admin', '[HIDDEN]') AS __table2 ON __table1.timestamp_col = __table2.timestamp_col ORDER BY __table1.boolean_col ASC, __table1.long_col ASC, __table1.double_col ASC, __table1.string_col ASC, __table1.timestamp_col ASC, __table1.date_col ASC, __table1.time_col ASC, __table1.timestamptz_col ASC, __table1.integer_col ASC, __table1.float_col ASC, __table1.decimal_col ASC, __table2.boolean_col ASC, __table2.long_col ASC, __table2.double_col ASC, __table2.string_col ASC, __table2.timestamp_col ASC, __table2.date_col ASC, __table2.time_col ASC, __table2.timestamptz_col ASC, __table2.integer_col ASC, __table2.float_col ASC, __table2.decimal_col ASC SETTINGS object_storage_cluster_join_mode = 'allow'), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000010cce5b0
1. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000b76ac98
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000b76a768
3. DB::Exception::Exception<std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, std::basic_string_view<char, std::char_traits<char>>&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x0000000013ebefbc
4. DB::NearestFieldTypeImpl<__decay(DB::DateTime64), void>::Type& DB::Field::safeGet<DB::DateTime64>() & @ 0x0000000013ec0344
5. DB::ColumnDecimal<DB::DateTime64>::insert(DB::Field const&) @ 0x000000000b5fee98
6. DB::DataTypeNullable::createColumnConst(unsigned long, DB::Field const&) const @ 0x0000000014236634
7. DB::StorageObjectStorageSource::generate() @ 0x00000000137d8c30
8. DB::ISource::tryGenerate() @ 0x00000000167e836c
9. DB::ISource::work() @ 0x00000000167e7f80
10. DB::ExecutionThreadContext::executeTask() @ 0x0000000016802360
11. DB::PipelineExecutor::executeStepImpl(unsigned long, DB::IAcquiredSlot*, std::atomic<bool>*) @ 0x00000000167f5d70
12. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x00000000167f94d4
13. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x0000000010dffa7c
14. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000010e05e60
15. ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x0000000010dfcfcc
16. void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x0000000010e03fdc
17. ? @ 0x0000000000080398
18. ? @ 0x00000000000e9e9c

2025.12.09 00:10:10.577929 [ 1585 ] {} <Error> TCPHandler: Code: 170. DB::Exception: Bad get: has Int64, requested Decimal64: While executing ReadFromObjectStorage. (BAD_GET), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000010cce5b0
1. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000b76ac98
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000b76a768
3. DB::Exception::Exception<std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, std::basic_string_view<char, std::char_traits<char>>&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x0000000013ebefbc
4. DB::NearestFieldTypeImpl<__decay(DB::DateTime64), void>::Type& DB::Field::safeGet<DB::DateTime64>() & @ 0x0000000013ec0344
5. DB::ColumnDecimal<DB::DateTime64>::insert(DB::Field const&) @ 0x000000000b5fee98
6. DB::DataTypeNullable::createColumnConst(unsigned long, DB::Field const&) const @ 0x0000000014236634
7. DB::StorageObjectStorageSource::generate() @ 0x00000000137d8c30
8. DB::ISource::tryGenerate() @ 0x00000000167e836c
9. DB::ISource::work() @ 0x00000000167e7f80
10. DB::ExecutionThreadContext::executeTask() @ 0x0000000016802360
11. DB::PipelineExecutor::executeStepImpl(unsigned long, DB::IAcquiredSlot*, std::atomic<bool>*) @ 0x00000000167f5d70
12. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x00000000167f94d4
13. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x0000000010dffa7c
14. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000010e05e60
15. ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x0000000010dfcfcc
16. void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x0000000010e03fdc
17. ? @ 0x0000000000080398
18. ? @ 0x00000000000e9e9c

[alsugiliazova]

SHOW CREATE TABLE `namespace_e380ef31_d511_11f0_923d_e0c26496f172.table_e380ef65_d511_11f0_8a18_e0c26496f172`

Query id: 9df4e4a3-697a-4fee-945e-0f7ed3a5490d

   ┌─statement──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
1. │ CREATE TABLE database_e377e16e_d511_11f0_84b0_e0c26496f172.`namespace_e380ef31_d511_11f0_923d_e0c26496f172.table_e380ef65_d511_11f0_8a18_e0c26496f172`↴│
   │↳(                                                                                                                                                     ↴│
   │↳    `boolean_col` Nullable(Bool),                                                                                                                     ↴│
   │↳    `long_col` Nullable(Int64),                                                                                                                       ↴│
   │↳    `double_col` Nullable(Float64),                                                                                                                   ↴│
   │↳    `string_col` String,                                                                                                                              ↴│
   │↳    `timestamp_col` Nullable(DateTime64(6)),                                                                                                          ↴│
   │↳    `date_col` Nullable(Date),                                                                                                                        ↴│
   │↳    `time_col` Nullable(Int64),                                                                                                                       ↴│
   │↳    `timestamptz_col` Nullable(DateTime64(6, 'UTC')),                                                                                                 ↴│
   │↳    `integer_col` Nullable(Int32),                                                                                                                    ↴│
   │↳    `float_col` Nullable(Float32),                                                                                                                    ↴│
   │↳    `decimal_col` Nullable(Decimal(10, 2))                                                                                                            ↴│
   │↳)                                                                                                                                                     ↴│
   │↳ENGINE = Iceberg('http://minio:9000/warehouse/data1/', 'admin', '[HIDDEN]')                                                                            │
   └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.010 sec. 

SELECT *
FROM `namespace_e380ef31_d511_11f0_923d_e0c26496f172.table_e380ef65_d511_11f0_8a18_e0c26496f172`

Query id: 4bd5f518-01e1-4840-997a-e780e730e006

   ┌─boolean_col─┬─long_col─┬─double_col─┬─string_col─┬──────────────timestamp_col─┬───date_col─┬────time_col─┬────────────timestamptz_col─┬─integer_col─┬─float_col─┬─decimal_col─┐
1. │ true        │     1000 │     456.78 │ Alice      │ 2024-01-01 13:00:00.000000 │ 2024-01-01 │ 43200000000 │ 2024-01-01 12:00:00.000000 │        1000 │    456.78 │      456.78 │
2. │ false       │     2000 │     456.78 │ Bob        │ 2024-01-01 13:00:00.000000 │ 2024-01-01 │ 43200000000 │ 2024-01-01 12:00:00.000000 │        2000 │    456.78 │      456.78 │
3. │ true        │     1000 │     456.78 │ Alice      │ 2024-01-01 13:00:00.000000 │ 2024-01-01 │ 43200000000 │ 2024-01-01 12:00:00.000000 │        1000 │    456.78 │      456.78 │
4. │ false       │     2000 │     456.78 │ Bob        │ 2024-01-01 13:00:00.000000 │ 2024-01-01 │ 43200000000 │ 2024-01-01 12:00:00.000000 │        2000 │    456.78 │      456.78 │
5. │ true        │     1000 │     456.78 │ Alice      │ 2024-01-01 13:00:00.000000 │ 2024-01-01 │ 43200000000 │ 2024-01-01 12:00:00.000000 │        1000 │    456.78 │      456.78 │
6. │ false       │     2000 │     456.78 │ Bob        │ 2024-01-01 13:00:00.000000 │ 2024-01-01 │ 43200000000 │ 2024-01-01 12:00:00.000000 │        2000 │    456.78 │      456.78 │
7. │ true        │     1000 │     456.78 │ Alice      │ 2024-01-01 13:00:00.000000 │ 2024-01-01 │ 43200000000 │ 2024-01-01 12:00:00.000000 │        1000 │    456.78 │      456.78 │
8. │ false       │     2000 │     456.78 │ Bob        │ 2024-01-01 13:00:00.000000 │ 2024-01-01 │ 43200000000 │ 2024-01-01 12:00:00.000000 │        2000 │    456.78 │      456.78 │
   └─────────────┴──────────┴────────────┴────────────┴────────────────────────────┴────────────┴─────────────┴────────────────────────────┴─────────────┴───────────┴─────────────┘

8 rows in set. Elapsed: 0.031 sec. 

clickhouse1 :) 

[alsugiliazova]

Same issue on new version of 25.8 antalya:

2026.05.04 12:26:14.787237 [ 1239 ] {5331997a-c494-44a7-bab6-a49ac66d6eb9} <Error> executeQuery: Code: 170. DB::Exception: Received from clickhouse4:9000. DB::Exception: Bad get: has Int64, requested Decimal64: While executing ReadFromObjectStorage. Stack trace:

0. ./ci/tmp/build/./base/poco/Foundation/src/Exception.cpp:28: Poco::Exception::Exception(String const&, int) @ 0x000000001c998f38
1. ./ci/tmp/build/./src/Common/Exception.cpp:128: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e74cbac
2. DB::Exception::Exception(String&&, int, String, bool) @ 0x00000000091e8198
3. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000091e7da8
4. ./src/Common/Exception.h:141: DB::Exception::Exception<std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, std::basic_string_view<char, std::char_traits<char>>&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x0000000012f5dd7c
5. ./ci/tmp/build/./src/Core/Field.cpp:874: DB::NearestFieldTypeImpl<__decay(DB::DateTime64), void>::Type& DB::Field::safeGet<DB::DateTime64>() & @ 0x0000000012f5f5d8
6. ./src/Core/Field.h:437: DB::ColumnDecimal<DB::DateTime64>::insert(DB::Field const&) @ 0x0000000016356dd8
7. ./ci/tmp/build/./src/DataTypes/DataTypeNullable.cpp:82: DB::DataTypeNullable::createColumnConst(unsigned long, DB::Field const&) const @ 0x00000000132ab388
8. ./ci/tmp/build/./src/Storages/ObjectStorage/StorageObjectStorageSource.cpp:386: DB::StorageObjectStorageSource::generate() @ 0x00000000128a9de4
9. ./ci/tmp/build/./src/Processors/ISource.cpp:144: DB::ISource::tryGenerate() @ 0x00000000174e9de8
10. ./ci/tmp/build/./src/Processors/ISource.cpp:110: DB::ISource::work() @ 0x00000000174e9abc
11. ./ci/tmp/build/./src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::ExecutionThreadContext::executeTask() @ 0x0000000017500bdc
12. ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::executeStepImpl(unsigned long, DB::IAcquiredSlot*, std::atomic<bool>*) @ 0x00000000174f5708
13. ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:279: void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x00000000174f6a54
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000e8782e8
15. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'()::operator()() @ 0x000000000e87ca60
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000e87622c
17. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000e87b408
18. ? @ 0x000000000007d5b8
19. ? @ 0x00000000000e5edc
: While executing Remote. (BAD_GET) (version 25.8.22.20001.altinityantalya) (from 127.0.0.1:52398) (query 1, line 2) (in query: SELECT * FROM database_a60c8f3a_47a3_11f1_893a_de7b9eea3490.`namespace_a71be5ce_47a3_11f1_893a_de7b9eea3490.table_a71be628_47a3_11f1_893a_de7b9eea3490` AS t1 ANY RIGHT JOIN s3('http://minio:9000/warehouse/data2/data/**.parquet', 'admin', '[HIDDEN]') AS t2 ON (t1.long_col < t2.long_col) AND (t1.string_col = t2.string_col) ORDER BY tuple(*) ASC SETTINGS object_storage_cluster_join_mode = 'allow', object_storage_cluster = 'replicated_cluster_three_nodes' FORMAT Values), Stack trace (when copying this message, always include the lines below):

0. ./ci/tmp/build/./base/poco/Foundation/src/Exception.cpp:28: Poco::Exception::Exception(String const&, int) @ 0x000000001c998f38
1. ./ci/tmp/build/./src/Common/Exception.cpp:128: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e74cbac
2. DB::Exception::Exception(String const&, int, String, bool) @ 0x000000000d451018
3. ./src/Common/Exception.h:130: DB::readException(DB::ReadBuffer&, String const&, bool) @ 0x000000000e81cb08
4. ./ci/tmp/build/./src/Client/Connection.cpp:1506: DB::Connection::receiveException() const @ 0x0000000017301880
5. ./ci/tmp/build/./src/Client/Connection.cpp:1328: DB::Connection::receivePacket() @ 0x00000000173087e0
6. ./ci/tmp/build/./src/Client/MultiplexedConnections.cpp:398: DB::MultiplexedConnections::receivePacketUnlocked(std::function<void (int, Poco::Timespan, DB::AsyncEventTimeoutType, String const&, unsigned int)>) @ 0x0000000017345b8c
7. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutorReadContext.cpp:72: DB::RemoteQueryExecutorReadContext::Task::run(std::function<void (int, Poco::Timespan, DB::AsyncEventTimeoutType, String const&, unsigned int)>, std::function<void ()>) @ 0x000000001327f53c
8. ./ci/tmp/build/./src/Common/AsyncTaskExecutor.cpp:89: DB::AsyncTaskExecutor::Routine::operator()(std::function<void ()>) @ 0x000000001327e9b8
9. ./src/Common/Fiber.h:76: void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, FiberStack&, Fiber::RoutineImpl<DB::AsyncTaskExecutor::Routine>>>(boost::context::detail::transfer_t) @ 0x000000001327e7cc

2026.05.04 12:26:14.789419 [ 1239 ] {} <Error> TCPHandler: Code: 170. DB::Exception: Received from clickhouse4:9000. DB::Exception: Bad get: has Int64, requested Decimal64: While executing ReadFromObjectStorage. Stack trace:

0. ./ci/tmp/build/./base/poco/Foundation/src/Exception.cpp:28: Poco::Exception::Exception(String const&, int) @ 0x000000001c998f38
1. ./ci/tmp/build/./src/Common/Exception.cpp:128: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e74cbac
2. DB::Exception::Exception(String&&, int, String, bool) @ 0x00000000091e8198
3. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000091e7da8
4. ./src/Common/Exception.h:141: DB::Exception::Exception<std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, std::basic_string_view<char, std::char_traits<char>>&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x0000000012f5dd7c
5. ./ci/tmp/build/./src/Core/Field.cpp:874: DB::NearestFieldTypeImpl<__decay(DB::DateTime64), void>::Type& DB::Field::safeGet<DB::DateTime64>() & @ 0x0000000012f5f5d8
6. ./src/Core/Field.h:437: DB::ColumnDecimal<DB::DateTime64>::insert(DB::Field const&) @ 0x0000000016356dd8
7. ./ci/tmp/build/./src/DataTypes/DataTypeNullable.cpp:82: DB::DataTypeNullable::createColumnConst(unsigned long, DB::Field const&) const @ 0x00000000132ab388
8. ./ci/tmp/build/./src/Storages/ObjectStorage/StorageObjectStorageSource.cpp:386: DB::StorageObjectStorageSource::generate() @ 0x00000000128a9de4
9. ./ci/tmp/build/./src/Processors/ISource.cpp:144: DB::ISource::tryGenerate() @ 0x00000000174e9de8
10. ./ci/tmp/build/./src/Processors/ISource.cpp:110: DB::ISource::work() @ 0x00000000174e9abc
11. ./ci/tmp/build/./src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::ExecutionThreadContext::executeTask() @ 0x0000000017500bdc
12. ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::executeStepImpl(unsigned long, DB::IAcquiredSlot*, std::atomic<bool>*) @ 0x00000000174f5708
13. ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:279: void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x00000000174f6a54
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000e8782e8
15. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'()::operator()() @ 0x000000000e87ca60
16. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000e87622c
17. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000e87b408
18. ? @ 0x000000000007d5b8
19. ? @ 0x00000000000e5edc
: While executing Remote. (BAD_GET), Stack trace (when copying this message, always include the lines below):

0. ./ci/tmp/build/./base/poco/Foundation/src/Exception.cpp:28: Poco::Exception::Exception(String const&, int) @ 0x000000001c998f38
1. ./ci/tmp/build/./src/Common/Exception.cpp:128: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e74cbac
2. DB::Exception::Exception(String const&, int, String, bool) @ 0x000000000d451018
3. ./src/Common/Exception.h:130: DB::readException(DB::ReadBuffer&, String const&, bool) @ 0x000000000e81cb08
4. ./ci/tmp/build/./src/Client/Connection.cpp:1506: DB::Connection::receiveException() const @ 0x0000000017301880
5. ./ci/tmp/build/./src/Client/Connection.cpp:1328: DB::Connection::receivePacket() @ 0x00000000173087e0
6. ./ci/tmp/build/./src/Client/MultiplexedConnections.cpp:398: DB::MultiplexedConnections::receivePacketUnlocked(std::function<void (int, Poco::Timespan, DB::AsyncEventTimeoutType, String const&, unsigned int)>) @ 0x0000000017345b8c
7. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutorReadContext.cpp:72: DB::RemoteQueryExecutorReadContext::Task::run(std::function<void (int, Poco::Timespan, DB::AsyncEventTimeoutType, String const&, unsigned int)>, std::function<void ()>) @ 0x000000001327f53c
8. ./ci/tmp/build/./src/Common/AsyncTaskExecutor.cpp:89: DB::AsyncTaskExecutor::Routine::operator()(std::function<void ()>) @ 0x000000001327e9b8
9. ./src/Common/Fiber.h:76: void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, FiberStack&, Fiber::RoutineImpl<DB::AsyncTaskExecutor::Routine>>>(boost::context::detail::transfer_t) @ 0x000000001327e7cc

[alsugiliazova]

Bug: Bad get: has Int64, requested Decimal64 in Distributed Iceberg JOIN

Component: StorageObjectStorageSource / Iceberg constant-column optimisation
Severity: Crash (exception, query aborted)
Versions affected: 25.8.x (altinityantalya); likely any build with allow_experimental_iceberg_read_optimization = true
Reproducibility: Intermittent — see Why it does not always happen

---

Symptom

A distributed RIGHT ANY JOIN between an Iceberg (DataLakeCatalog) table and a plain s3() table function — using object_storage_cluster_join_mode = 'allow' — crashes on a remote cluster node with:

Code: 170. DB::Exception: Bad get: has Int64, requested Decimal64:
While executing ReadFromObjectStorage. (BAD_GET)

Stack frames (condensed):

StorageObjectStorageSource::generate()          ← StorageObjectStorageSource.cpp:386
  DataTypeNullable::createColumnConst(n, field) ← DataTypeNullable.cpp:82
    ColumnDecimal<DateTime64>::insert(field)
      field.safeGet<DateTime64>()
        → THROWS: "has Int64, requested Decimal64"

Failing query shape:

SELECT *
FROM <iceberg_catalog_table> AS t1
RIGHT ANY JOIN s3('<iceberg-location>/data/**.parquet', ...) AS t2
  ON t1.long_col < t2.long_col AND t1.string_col = t2.string_col
ORDER BY tuple(*)
SETTINGS
    object_storage_cluster_join_mode = 'allow',
    object_storage_cluster = 'replicated_cluster_three_nodes'

The same query against MergeTree copies of both tables succeeds.

---

Root Cause

There are two defects that combine to cause the crash.

---

Defect 1 — Iceberg time type maps to Int64; Arrow/Parquet maps it to DateTime64

SchemaProcessor.cpp:245–246

if (type_name == f_time)
    return std::make_shared<DataTypeInt64>();   // Iceberg "time" → Int64

ArrowColumnToCHColumn.cpp:413–418 (Arrow/Parquet reader)

template <typename TimeType, typename TimeArray>
static ColumnWithTypeAndName readColumnWithTimeData(...)
{
    const UInt8 scale = arrow_type.unit() * 3;
    auto internal_type = std::make_shared<DataTypeDateTime64>(scale); // TIME64 → DateTime64(6)
    ...
}

The same physical column (time_col, Iceberg TimeType, Parquet INT64 TIME_MICROS) is seen as two different ClickHouse types depending on who reads the schema:

Reader	ClickHouse type
Iceberg manifest / SchemaProcessor	Nullable(Int64)
Arrow/Parquet inference (used for s3())	Nullable(DateTime64(6))

---

Defect 2 — WhichDataType::isDecimal() excludes DateTime64

In IDataType.h:413:

// isDecimal() does NOT include DateTime64 or Time64
constexpr bool isDecimal() const {
    return isDecimal32() || isDecimal64() || isDecimal128() || isDecimal256();
}

In ManifestFile.cpp:67, Iceberg column bound values are deserialised:

if (DB::WhichDataType(non_nullable_type).isDecimal())
{
    // correct path: big-endian → little-endian + DecimalField<Decimal64>
}
else
{
    // fallback path: raw memcpy → column->get() → returns whatever the column's get() produces
    column->insertData(str.data(), str.length());
    DB::Field result;
    column->get(0, result);   // for Int64 column → Field holds Int64
    return result;
}

DateTime64 falls into the else branch. But Int64 (the type for time) also falls into the else branch and returns a Field(Int64). This is correct for Int64-typed columns, but the caller (StorageObjectStorageSource) later pairs that Int64 bound value with a column type of Nullable(DateTime64(6)) — which is wrong.

---

How the two defects combine

Initiator node                          Remote node (clickhouse4)
──────────────────────────────────────  ─────────────────────────────────────────────────
IcebergIterator reads manifest file
  └─ time_col lower_bound bytes
       → deserializeFieldFromBinaryRepr(expected_type=Nullable(Int64))
       → Field(Int64 = 43200000000)    ─── serialized via ClusterFunctionReadTask ──▶

                                        DataFileMetaInfo.columns_info["time_col"]
                                          .hyperrectangle.left = Field(Int64)

                                        read_from_format_info.requested_columns
                                          built from Arrow/Parquet schema inference
                                          → time_col: Nullable(DateTime64(6))

                                        constant_columns_with_values["time_col"] = {
                                            .name_and_type.type = Nullable(DateTime64(6)) ← parquet
                                            .value              = Int64                    ← iceberg
                                        }

                                        createColumnConst(n, Int64)
                                          for Nullable(DateTime64(6))
                                          → ColumnDecimal<DateTime64>::insert(Int64)
                                          → safeGet<DateTime64>()
                                          → CRASH: "has Int64, requested Decimal64"

DataFileMetaInfo is built from Iceberg column statistics (using Iceberg schema) on the initiator, then sent to remote nodes via ClusterFunctionReadTask.cpp:103–104. On the remote node the file is processed with a read_from_format_info built from Arrow/Parquet schema inference, not from the Iceberg schema. These two see time_col differently.

---

Why it only fires for time_col, not for timestamp columns

timestamp_col (Iceberg TimestampType) is mapped to DateTime64(6) by both the Iceberg SchemaProcessor and the Arrow reader, so there is no disagreement. time_col (Iceberg TimeType) is the only column type mapped to Int64 by the Iceberg SchemaProcessor but to DateTime64 by Arrow.

---

Why it does not always happen

The crash requires all five conditions to hold simultaneously:

Condition 1 — allow_experimental_iceberg_read_optimization = true

This setting (default true, marked EXPERIMENTAL in Settings.cpp) must be enabled. When it is false, the constant_columns_with_values optimisation is entirely skipped:

// StorageObjectStorageSource.cpp:608
if (context_->getSettingsRef()[Setting::allow_experimental_iceberg_read_optimization])
{
    auto file_meta_data = object_info->getFileMetaInfo();
    ...
}

If the setting is disabled in a test run or user config, no crash occurs.

---

Condition 2 — Iceberg manifest must contain column statistics for time_col

The bound values (lower_bounds, upper_bounds) are written into the manifest by the Iceberg writer (pyiceberg in this test). Not all writers include statistics for all column types:

• pyiceberg computes statistics from PyArrow's Parquet metadata
• For TimeType, some pyiceberg versions skip or omit the column statistics if the internal conversion from Parquet INT64 TIME_MICROS bounds to Iceberg time bounds is not implemented
• If lower_bounds / upper_bounds are absent for time_col, hyperrectangle.has_value() is false and the constant-column path is not entered → no crash

This is the primary source of intermittency across different environments or pyiceberg versions.

---

Condition 3 — time_col must be a "point" (min == max) with null_count == 0

// StorageObjectStorageSource.cpp:645–646
if (column.second.hyperrectangle.value().isPoint() &&
    (!column.second.nulls_count.has_value() || column.second.nulls_count.value() <= 0))

Both conditions fire only when all values in a file are identical and none are NULL. For the test data (time(12, 0, 0) repeated for every row, 2 rows per file) this is deterministically true — but only when Condition 2 is also satisfied.

---

Condition 4 — The specific join combination must be exercised

The test runs 675 combinations in parallel (different left/right table types, join clauses, clusters, join conditions). The crash only occurs when:

• Left table: Iceberg catalog table (iceberg_table type)
• Right table: plain s3() table function
• Mode: object_storage_cluster_join_mode = 'allow'
• Cluster: non-None (e.g., replicated_cluster_three_nodes)

Only a subset of all 675 combinations satisfies this. If a given test run does not reach the failing combination before a timeout, or if a prior failure in the same parallel batch interrupts the run, the crash does not appear in the output.

---

Condition 5 — Schema resolution on the remote node takes the Parquet-inference path

With allow mode, the full JOIN query is sent unchanged to remote nodes (IStorageCluster.cpp:297–298). Each remote node executes the complete query independently. When the remote node reads the Iceberg parquet files received via ClusterFunctionReadTask:

• The DataFileMetaInfo (with time_col bounds as Int64) comes from the initiator
• Whether read_from_format_info.requested_columns uses the Iceberg schema (Int64) or the Parquet-inferred schema (DateTime64) depends on which code path builds read_from_format_info on the remote node

If the remote node resolves the Iceberg table via the catalog (using the Iceberg schema, Int64), there is no mismatch and no crash. If the remote node builds read_from_format_info from the Parquet/Arrow reader (e.g., because it receives a plain ObjectInfo without the full Iceberg catalog context), the mismatch occurs.

This path varies depending on how the query is serialised and deserialised across nodes, which can differ between ClickHouse patch versions.

---

Minimal Reproduction

-- Create Iceberg catalog database
CREATE DATABASE iceberg_db ENGINE = DataLakeCatalog(...)

-- iceberg_db.test_table has:
--   time_col TimeType (all rows = time(12,0,0))
--   ...other columns...

-- This query crashes:
SELECT *
FROM iceberg_db.test_table AS t1
RIGHT ANY JOIN s3('http://minio:9000/warehouse/data/**.parquet', 'user', 'pass') AS t2
  ON t1.long_col < t2.long_col
SETTINGS
    allow_experimental_iceberg_read_optimization = 1,
    object_storage_cluster_join_mode = 'allow',
    object_storage_cluster = 'my_cluster'

---

Proposed Fixes

Fix A — Align Iceberg time type mapping with Arrow (recommended)

SchemaProcessor.cpp:245

// Before:
if (type_name == f_time)
    return std::make_shared<DataTypeInt64>();

// After:
if (type_name == f_time)
    return std::make_shared<DataTypeDateTime64>(6);   // matches Arrow TIME64 mapping

This eliminates the type disagreement at the source. It is a semantic change (visible type of time_col changes from Int64 to DateTime64(6)) so downstream effects (casting, parquet write-back, comparisons) must be verified.

---

Fix B — Handle DateTime64/Time64 in deserializeFieldFromBinaryRepr

ManifestFile.cpp:67

// Before:
if (DB::WhichDataType(non_nullable_type).isDecimal())

// After:
if (DB::WhichDataType(non_nullable_type).isDecimal()
    || DB::WhichDataType(non_nullable_type).isDateTime64()
    || DB::WhichDataType(non_nullable_type).isTime64())

And add the correct deserialization for DateTime64/Time64 inside the block: big-endian 8-byte int64 → DecimalField<DateTime64>. This also fixes the byte-order issue for timestamp bounds (currently read via memcpy without endian conversion).

---

Fix C — Defensive guard in StorageObjectStorageSource (crash prevention)

Add a type-compatibility check before inserting into constant_columns_with_values:

// StorageObjectStorageSource.cpp, before line 649
if (column.second.hyperrectangle.value().isPoint() && ...)
{
    const auto & bound_value = column.second.hyperrectangle.value().left;
    const auto * col_type = removeNullable(i_column->second.second.type).get();
    // Only use as constant if the bound field type is compatible
    bool compatible = (col_type->getTypeId() == TypeIndex::Int64 && bound_value.getType() == Field::Types::Int64)
                   || (isDecimalOrDateTime64(col_type) && bound_value.getType() == Field::Types::Decimal64)
                   || ...;
    if (!compatible)
    {
        LOG_WARNING(log, "Skipping constant column '{}': type mismatch between "
                    "Iceberg bounds ({}) and requested column type ({})",
                    column_name, bound_value.getTypeName(), col_type->getName());
    }
    else
    {
        constant_columns_with_values[...] = ...;
    }
}

This prevents the crash without changing semantics, and logs a warning to aid diagnosis. Should be applied as a short-term patch while Fix A or B is evaluated.

---

Files to Change

File	Change
src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp:245	Map Iceberg time → DateTime64(6) instead of Int64
src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp:67	Extend decimal check to include DateTime64/Time64; add big-endian handling
src/Storages/ObjectStorage/StorageObjectStorageSource.cpp:645	Add type-compatibility guard before populating constant_columns_with_values

[ianton-ru]

May be this can help: #1731