Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

23.11+ Grouping is not finalized when a Map element that does not exist in a Map #59402

Closed
SaltTan opened this issue Jan 30, 2024 · 5 comments · Fixed by #59594
Closed

23.11+ Grouping is not finalized when a Map element that does not exist in a Map #59402

SaltTan opened this issue Jan 30, 2024 · 5 comments · Fixed by #59594
Assignees
@Algunenano
Labels
bug Confirmed user-visible misbehaviour in official release crash Crash / segfault / abort

Comments

@SaltTan
Copy link
Contributor

SaltTan commented Jan 30, 2024

I am extracting a value from a Map column and grouping by that value.
If corresponding value does not exist in the Map (the empty value is returned) I find several rows for this empty value in the result.

Broken in 23.11
23.11 https://fiddle.clickhouse.com/43f333fe-7d2b-4e0c-bc2d-09304048a71b
head https://fiddle.clickhouse.com/a05a016a-c182-468b-a35e-13fec358b547

How to reproduce

CREATE TABLE t1
  (f1 Int32,
   f2 Map(LowCardinality(String),LowCardinality(String)),
   f3 Map(String,String),
   f4 Map(Int32,Int32)
   )
engine=Memory as 
SELECT 1 as f1,
  map(number%2,number%10) as f2,
  f2 as f3,
  f2 as f4
from numbers(1000111);

set max_block_size=10;

select f1, f2['2'] as LowCardinality, count() 
from t1 group by 1,2 order by 1,2 format PrettyCompactMonoBlock;
+-f1-+-LowCardinality-+-count()-+
|  1 |                |   65409 |
|  1 |                |   65409 |
|  1 |                |   65409 |
|  1 |                |   65409 |
|  1 |                |  738475 |
+----+----------------+---------+

select f1, f3['2'] as String, count() 
from t1 group by 1,2 order by 1,2 format PrettyCompactMonoBlock;
+-f1-+-String-+-count()-+
|  1 |        |   18976 |
|  1 |        |   65409 |
|  1 |        |   65409 |
|  1 |        |  850317 |
+----+--------+---------+

select f1, f4[2] as Int32, count()
from t1 group by 1,2 order by 1,2 format PrettyCompactMonoBlock;
+-f1-+-Int32-+-count()-+
|  1 |     0 | 1000111 |
+----+-------+---------+
@SaltTan SaltTan added the potential bug To be reviewed by developers and confirmed/rejected. label Jan 30, 2024
@caoyang10
Copy link

I met same problem...

@Algunenano Algunenano added bug Confirmed user-visible misbehaviour in official release crash Crash / segfault / abort and removed potential bug To be reviewed by developers and confirmed/rejected. labels Jan 31, 2024
@Algunenano
Copy link
Member

The report sounded suspicious and made me check under MSAN:

ClickHouse local version 24.1.1.1.

Mordor :) CREATE TABLE t1
  (f1 Int32,
   f2 Map(LowCardinality(String),LowCardinality(String)),
   f3 Map(String,String),
   f4 Map(Int32,Int32)
   )
engine=Memory as 
SELECT 1 as f1,
  map(number%2,number%10) as f2,
  f2 as f3,
  f2 as f4
from numbers(1000111);

CREATE TABLE t1
(
    `f1` Int32,
    `f2` Map(LowCardinality(String), LowCardinality(String)),
    `f3` Map(String, String),
    `f4` Map(Int32, Int32)
)
ENGINE = Memory AS
SELECT
    1 AS f1,
    map(number % 2, number % 10) AS f2,
    f2 AS f3,
    f2 AS f4
FROM numbers(1000111)

Query id: 54cacc5c-d106-4e61-b673-1d49061b4cdd

Ok.

0 rows in set. Elapsed: 1.242 sec. Processed 1.00 million rows, 8.00 MB (805.43 thousand rows/s., 6.44 MB/s.)
Peak memory usage: 44.47 MiB.

Mordor :) select f1, f2['2'] as LowCardinality, count() 
from t1 group by 1,2 order by 1,2 format PrettyCompactMonoBlock;

SELECT
    f1,
    f2['2'] AS LowCardinality,
    count()
FROM t1
GROUP BY
    1,
    2
ORDER BY
    1 ASC,
    2 ASC
FORMAT PrettyCompactMonoBlock

Query id: 2c9dde33-0455-49dd-8b5e-630771597b6e

==39641==WARNING: MemorySanitizer: use-of-uninitialized-value
↘ Progress: 1.00 million rows, 14.00 MB (806.51 thousand rows/s., 11.29 MB/s.)                                                                                                                                                                                  #0 0x5638eac68ff5 in HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>::keyEquals(StringRef const&, unsigned long) const build_msan/./src/Common/HashTable/HashMap.h:176:87
    #1 0x5638eac68ff5 in HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>::keyEquals(StringRef const&, unsigned long, HashTableNoState const&) const build_msan/./src/Common/HashTable/HashMap.h:177:97
    #2 0x5638eac68ff5 in HashTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>::findCell(StringRef const&, unsigned long, unsigned long) const build_msan/./src/Common/HashTable/HashTable.h:481:69
    #3 0x5638eac68ff5 in void HashTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>::emplaceNonZero<DB::SerializedKeyHolder&>(DB::SerializedKeyHolder&, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>*&, bool&, unsigned long) build_msan/./src/Common/HashTable/HashTable.h:1017:30
    #4 0x5638eac68ff5 in void HashTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>::emplace<DB::SerializedKeyHolder&>(DB::SerializedKeyHolder&, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>*&, bool&, unsigned long) build_msan/./src/Common/HashTable/HashTable.h:1097:13
    #5 0x5638eac68ff5 in void HashTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>::emplace<DB::SerializedKeyHolder&>(DB::SerializedKeyHolder&, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>*&, bool&) build_msan/./src/Common/HashTable/HashTable.h:1088:9
    #6 0x5638eac68ff5 in DB::ColumnsHashing::columns_hashing_impl::EmplaceResultImpl<char*> DB::ColumnsHashing::columns_hashing_impl::HashMethodBase<DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>, PairNoInit<StringRef, char*>, char*, false, false, false>::emplaceImpl<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>, DB::SerializedKeyHolder>(DB::SerializedKeyHolder&, HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>&) build_msan/./src/Common/ColumnsHashingImpl.h:291:14
    #7 0x5638eac68ff5 in DB::ColumnsHashing::columns_hashing_impl::EmplaceResultImpl<char*> DB::ColumnsHashing::columns_hashing_impl::HashMethodBase<DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>, PairNoInit<StringRef, char*>, char*, false, false, false>::emplaceKey<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>(HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>&, unsigned long, DB::Arena&) build_msan/./src/Common/ColumnsHashingImpl.h:187:16
    #8 0x5638eac68ff5 in void DB::Aggregator::executeImplBatch<false, false, false, DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>, DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>>(DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>&, DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>&, DB::Arena*, unsigned long, unsigned long, DB::Aggregator::AggregateFunctionInstruction*, bool, char*) const build_msan/./src/Interpreters/Aggregator.cpp:1259:41
    #9 0x5638eac54fa5 in void DB::Aggregator::executeImpl<DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>, DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>>(DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>&, DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>&, DB::Arena*, unsigned long, unsigned long, DB::Aggregator::AggregateFunctionInstruction*, bool, bool, char*) const build_msan/./src/Interpreters/Aggregator.cpp:1120:17
    #10 0x5638ea80fc36 in void DB::Aggregator::executeImpl<DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>>(DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>&, DB::Arena*, unsigned long, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>&, DB::Aggregator::AggregateFunctionInstruction*, DB::ColumnsHashing::LastElementCacheStats&, bool, bool, char*) const build_msan/./src/Interpreters/Aggregator.cpp:1071:9
    #11 0x5638ea5644bb in DB::Aggregator::executeImpl(DB::AggregatedDataVariants&, unsigned long, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>&, DB::Aggregator::AggregateFunctionInstruction*, bool, bool, char*) const build_msan/./src/Interpreters/Aggregator.cpp:1047:5
    #12 0x5638ea572ee4 in DB::Aggregator::executeOnBlock(std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>&, std::__1::vector<std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>, std::__1::allocator<std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>>>&, bool&) const build_msan/./src/Interpreters/Aggregator.cpp:1705:9
    #13 0x5638f239c0f7 in DB::AggregatingTransform::consume(DB::Chunk) build_msan/./src/Processors/Transforms/AggregatingTransform.cpp:684:33
    #14 0x5638f2390d0b in DB::AggregatingTransform::work() build_msan/./src/Processors/Transforms/AggregatingTransform.cpp:643:9
    #15 0x5638f148094d in DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) build_msan/./src/Processors/Executors/ExecutionThreadContext.cpp:47:26
    #16 0x5638f148094d in DB::ExecutionThreadContext::executeTask() build_msan/./src/Processors/Executors/ExecutionThreadContext.cpp:95:9
    #17 0x5638f1451c91 in DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) build_msan/./src/Processors/Executors/PipelineExecutor.cpp:273:26
    #18 0x5638f1455299 in DB::PipelineExecutor::executeSingleThread(unsigned long) build_msan/./src/Processors/Executors/PipelineExecutor.cpp:239:5
    #19 0x5638f1455299 in DB::PipelineExecutor::spawnThreads()::$_0::operator()() const build_msan/./src/Processors/Executors/PipelineExecutor.cpp:373:17
    #20 0x5638f1455299 in decltype(std::declval<DB::PipelineExecutor::spawnThreads()::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&>(DB::PipelineExecutor::spawnThreads()::$_0&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
    #21 0x5638f1455299 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::PipelineExecutor::spawnThreads()::$_0&>(DB::PipelineExecutor::spawnThreads()::$_0&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9
    #22 0x5638f1455299 in std::__1::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>::operator()[abi:v15000]() build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12
    #23 0x5638f1455299 in void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__1::__function::__policy_storage const*) build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16
    #24 0x5638d401894b in std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16
    #25 0x5638d401894b in std::__1::function<void ()>::operator()() const build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12
    #26 0x5638d401894b in ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__1::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) build_msan/./src/Common/ThreadPool.cpp:455:13
    #27 0x5638d4023bd8 in void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_msan/./src/Common/ThreadPool.cpp:217:73
    #28 0x5638d4023bd8 in decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(void&&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
    #29 0x5638d4023bd8 in decltype(auto) std::__1::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_msan/./contrib/llvm-project/libcxx/include/tuple:1789:1
    #30 0x5638d4023bd8 in decltype(auto) std::__1::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&) build_msan/./contrib/llvm-project/libcxx/include/tuple:1798:1
    #31 0x5638d4023bd8 in ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() build_msan/./src/Common/ThreadPool.h:223:13
    #32 0x5638d4023a5e in decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
    #33 0x5638d4023a5e in void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9
    #34 0x5638d4023a5e in std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12
    #35 0x5638d4023a5e in void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16
    #36 0x5638d40104ba in std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16
    #37 0x5638d40104ba in std::__1::function<void ()>::operator()() const build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12
    #38 0x5638d40104ba in ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_msan/./src/Common/ThreadPool.cpp:455:13
    #39 0x5638d401d696 in void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_msan/./src/Common/ThreadPool.cpp:217:73
    #40 0x5638d401d696 in decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
    #41 0x5638d401d696 in void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) build_msan/./contrib/llvm-project/libcxx/include/thread:284:5
    #42 0x5638d401d696 in void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_msan/./contrib/llvm-project/libcxx/include/thread:295:5
    #43 0x768b05be79ea in start_thread /usr/src/debug/glibc/glibc/nptl/pthread_create.c:444:8
    #44 0x768b05c6b7cb in clone3 /usr/src/debug/glibc/glibc/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

  Uninitialized value was created by a heap allocation
↗ Progress: 1.00 million rows, 14.00 MB (110.43 thousand rows/s., 1.55 MB/s.)                                                                                                                                                                                   #0 0x5638ba4a19ce in malloc (/mnt/ch/ClickHouse/build_msan/programs/clickhouse+0x8a489ce) (BuildId: 675876e3c973cdc42aeb7fe04e3f52dd8e6f321b)
    #1 0x5638d3c3dad4 in void* (anonymous namespace)::allocNoTrack<false, false>(unsigned long, unsigned long) build_msan/./src/Common/Allocator.cpp:68:19
    #2 0x5638d3c3dad4 in Allocator<false, false>::alloc(unsigned long, unsigned long) build_msan/./src/Common/Allocator.cpp:115:18
    #3 0x5638bde438d5 in DB::Arena::addMemoryChunk(unsigned long) (/mnt/ch/ClickHouse/build_msan/programs/clickhouse+0xc3ea8d5) (BuildId: 675876e3c973cdc42aeb7fe04e3f52dd8e6f321b)
    #4 0x5638bde42f68 in DB::Arena::allocContinue(unsigned long, char const*&, unsigned long) (/mnt/ch/ClickHouse/build_msan/programs/clickhouse+0xc3e9f68) (BuildId: 675876e3c973cdc42aeb7fe04e3f52dd8e6f321b)
    #5 0x5638ee1ca56b in DB::ColumnVector<int>::serializeValueIntoArena(unsigned long, DB::Arena&, char const*&, char8_t const*) const build_msan/./src/Columns/ColumnVector.cpp:71:21
    #6 0x5638eac614b3 in DB::serializeKeysToPoolContiguous(unsigned long, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>> const&, DB::Arena&) build_msan/./src/Interpreters/AggregationCommon.h:275:37
    #7 0x5638eac614b3 in DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>::getKeyHolder(unsigned long, DB::Arena&) const build_msan/./src/Common/ColumnsHashing.h:708:13
    #8 0x5638eac614b3 in DB::ColumnsHashing::columns_hashing_impl::EmplaceResultImpl<char*> DB::ColumnsHashing::columns_hashing_impl::HashMethodBase<DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>, PairNoInit<StringRef, char*>, char*, false, false, false>::emplaceKey<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>(HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>&, unsigned long, DB::Arena&) build_msan/./src/Common/ColumnsHashingImpl.h:186:57
    #9 0x5638eac614b3 in void DB::Aggregator::executeImplBatch<false, false, false, DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>, DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>>(DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>&, DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>&, DB::Arena*, unsigned long, unsigned long, DB::Aggregator::AggregateFunctionInstruction*, bool, char*) const build_msan/./src/Interpreters/Aggregator.cpp:1259:41
    #10 0x5638eac54fa5 in void DB::Aggregator::executeImpl<DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>, DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>>(DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>&, DB::ColumnsHashing::HashMethodSerialized<PairNoInit<StringRef, char*>, char*>&, DB::Arena*, unsigned long, unsigned long, DB::Aggregator::AggregateFunctionInstruction*, bool, bool, char*) const build_msan/./src/Interpreters/Aggregator.cpp:1120:17
    #11 0x5638ea80fc36 in void DB::Aggregator::executeImpl<DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>>(DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>>&, DB::Arena*, unsigned long, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>&, DB::Aggregator::AggregateFunctionInstruction*, DB::ColumnsHashing::LastElementCacheStats&, bool, bool, char*) const build_msan/./src/Interpreters/Aggregator.cpp:1071:9
    #12 0x5638ea5644bb in DB::Aggregator::executeImpl(DB::AggregatedDataVariants&, unsigned long, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>&, DB::Aggregator::AggregateFunctionInstruction*, bool, bool, char*) const build_msan/./src/Interpreters/Aggregator.cpp:1047:5
    #13 0x5638ea572ee4 in DB::Aggregator::executeOnBlock(std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>&, std::__1::vector<std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>, std::__1::allocator<std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>>>&, bool&) const build_msan/./src/Interpreters/Aggregator.cpp:1705:9
    #14 0x5638f239c0f7 in DB::AggregatingTransform::consume(DB::Chunk) build_msan/./src/Processors/Transforms/AggregatingTransform.cpp:684:33
    #15 0x5638f2390d0b in DB::AggregatingTransform::work() build_msan/./src/Processors/Transforms/AggregatingTransform.cpp:643:9
    #16 0x5638f148094d in DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) build_msan/./src/Processors/Executors/ExecutionThreadContext.cpp:47:26
    #17 0x5638f148094d in DB::ExecutionThreadContext::executeTask() build_msan/./src/Processors/Executors/ExecutionThreadContext.cpp:95:9
    #18 0x5638f1451c91 in DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) build_msan/./src/Processors/Executors/PipelineExecutor.cpp:273:26
    #19 0x5638f1455299 in DB::PipelineExecutor::executeSingleThread(unsigned long) build_msan/./src/Processors/Executors/PipelineExecutor.cpp:239:5
    #20 0x5638f1455299 in DB::PipelineExecutor::spawnThreads()::$_0::operator()() const build_msan/./src/Processors/Executors/PipelineExecutor.cpp:373:17
    #21 0x5638f1455299 in decltype(std::declval<DB::PipelineExecutor::spawnThreads()::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&>(DB::PipelineExecutor::spawnThreads()::$_0&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
    #22 0x5638f1455299 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::PipelineExecutor::spawnThreads()::$_0&>(DB::PipelineExecutor::spawnThreads()::$_0&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9
    #23 0x5638f1455299 in std::__1::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>::operator()[abi:v15000]() build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12
    #24 0x5638f1455299 in void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__1::__function::__policy_storage const*) build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16
    #25 0x5638d401894b in std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16
    #26 0x5638d401894b in std::__1::function<void ()>::operator()() const build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12
    #27 0x5638d401894b in ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__1::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) build_msan/./src/Common/ThreadPool.cpp:455:13
    #28 0x5638d4023bd8 in void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_msan/./src/Common/ThreadPool.cpp:217:73
    #29 0x5638d4023bd8 in decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(void&&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
    #30 0x5638d4023bd8 in decltype(auto) std::__1::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_msan/./contrib/llvm-project/libcxx/include/tuple:1789:1
    #31 0x5638d4023bd8 in decltype(auto) std::__1::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&) build_msan/./contrib/llvm-project/libcxx/include/tuple:1798:1
    #32 0x5638d4023bd8 in ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() build_msan/./src/Common/ThreadPool.h:223:13
    #33 0x5638d4023a5e in decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
    #34 0x5638d4023a5e in void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9
    #35 0x5638d4023a5e in std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12
    #36 0x5638d4023a5e in void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16
    #37 0x5638d40104ba in std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16
    #38 0x5638d40104ba in std::__1::function<void ()>::operator()() const build_msan/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12
    #39 0x5638d40104ba in ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_msan/./src/Common/ThreadPool.cpp:455:13
    #40 0x5638d401d696 in void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_msan/./src/Common/ThreadPool.cpp:217:73
    #41 0x5638d401d696 in decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_msan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
    #42 0x5638d401d696 in void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) build_msan/./contrib/llvm-project/libcxx/include/thread:284:5
    #43 0x5638d401d696 in void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_msan/./contrib/llvm-project/libcxx/include/thread:295:5

SUMMARY: MemorySanitizer: use-of-uninitialized-value build_msan/./src/Common/HashTable/HashMap.h:176:87 in HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>::keyEquals(StringRef const&, unsigned long) const
Exiting
==39641==WARNING: MemorySanitizer: use-of-uninitialized-value

@CurtizJ
Copy link
Member

CurtizJ commented Jan 31, 2024

May be related: #55957.

@Algunenano Algunenano self-assigned this Feb 5, 2024
@Algunenano
Copy link
Member

Reverting #55957 does fix the issue. I'm having a look at the code, but seeing that it was supposed to improve performance and the perf tests show a degradation guides me towards straight revert.

cc @lgbo-ustc

@Algunenano
Copy link
Member

Fix is relatively simple, but I don't quite like the resulting code. I'm verifying if the performance change is worth the hassle it caused.

@Algunenano Algunenano mentioned this issue Feb 5, 2024
Merged
@den-crane den-crane mentioned this issue Feb 12, 2024
Closed
@den-crane den-crane mentioned this issue Feb 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Algunenano Algunenano

Labels
bug Confirmed user-visible misbehaviour in official release crash Crash / segfault / abort
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix incorrect result of arrayElement / map[] on empty value Algunenano/ClickHouse
4 participants
@Algunenano @caoyang10 @SaltTan @CurtizJ