Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add sanity checks for writing number in variable length format (resubmit) #48154

Merged
merged 5 commits into from
Mar 31, 2023
Merged

Add sanity checks for writing number in variable length format (resubmit) #48154

merged 5 commits into from
Mar 31, 2023

Conversation

azat
Copy link
Collaborator

@azat azat commented Mar 29, 2023

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

This is resubmit of #47608 that reverted in #47850 because sanity check had been triggered for Progress packets, this has been addressed in d7d9f0c

Cc: @rschu1ze

@azat
Add sanity checks for writing number in variable length format
36f6408 
And just to double check:

    # var_uint 9223372036854775807
    ffffffffffffffff7f
    ffffffffffffffff7f
    ffffffffffffffff7f
    x: 9223372036854775807, y: 9223372036854775807
    # var_uint 9223372036854775808
    808080808080808080
    808080808080808080
    808080808080808080
    x: 9223372036854775808, y: 0

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
@azat
Use VAR_UINT_MAX for unknown_packet_in_send_data
9457b1c 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
@azat
Fix overflow of VarUInt format in Progress packets
d7d9f0c 
Otherwise query like this, can trigger sanity check:

    WITH x AS (SELECT [], number AS a FROM numbers(9223372036854775807)), y AS (SELECT arrayLastOrNull(x -> (x >= -inf), []), arrayLastOrNull(x -> (x >= NULL), [1]), number AS a FROM numbers(1.)) SELECT [1023], * FROM x WHERE a IN (SELECT a FROM y) ORDER BY arrayLastOrNull(x -> (x >= 1025), [1048577, 1048576]) DESC NULLS LAST, '0.0000000002' ASC NULLS LAST, a DESC NULLS FIRST

CI: https://s3.amazonaws.com/clickhouse-test-reports/0/a9bcd022d5f4a5be530595dbfae3ed177b5c1972/fuzzer_astfuzzermsan/report.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
@azat azat mentioned this pull request Mar 29, 2023
Merged
@robot-ch-test-poll2 robot-ch-test-poll2 added the pr-not-for-changelog This PR should not be mentioned in the changelog label Mar 29, 2023
@rschu1ze rschu1ze self-assigned this Mar 29, 2023
@rschu1ze
Copy link
Member

AST Fuzzer:

ClickHouse Integration Tests (asan) [1/6]

@rschu1ze rschu1ze merged commit eb93ec3 into ClickHouse:master Mar 31, 2023
@azat azat deleted the varuint-v2 branch March 31, 2023 09:02
rschu1ze added a commit that referenced this pull request Apr 5, 2023
@rschu1ze
AST Fuzzer: Fix assertion in TopK serialization
598b050 
Problem:
  https://s3.amazonaws.com/clickhouse-test-reports/0/fa5b2bd4a5b02336bca8837c473a7124f8ecedf2/fuzzer_astfuzzerasan/report.html

The new assertion in the Varint code was introduced with (*). It rejects
values whose serialization cannot be deserialized (and this behavior
cannot be changed due to historical reasons). Such values should be
exceptionally rare in practice but AST fuzzer managers to trigger them.

The fix is similar to (**): Bypass the check by limiting the value to
the maximum allowed value.

(if AST fuzzer triggers finds more violations of the assertion, we might
consider throwing an exception instead)

(*) #48154
(**) https://github.com/ClickHouse/ClickHouse/pull/48154/files#diff-653c0a18dfdaa86262c78dc6b25550add0487f165b4ad053e86f530388f6203a
@rschu1ze rschu1ze mentioned this pull request Apr 5, 2023
Merged
zhiqiang-hhhh pushed a commit to zhiqiang-hhhh/ClickHouse that referenced this pull request Apr 11, 2023
@devcrafter @SmitaRKulkarni
@tavplubix @vdimir @vitlibar @novikd @robot-ch-test-poll @rschu1ze @CurtizJ @bkuschel @FFFFFFFHHHHHHH @serxa @rfraposa @robot-clickhouse-ci-1 @azat @antonio2368 @Avogar @yakov-olkhovskiy @CheSema @nikitamikhaylov @Algunenano @DanRoscigno @kssenii @alexey-milovidov @KochetovNicolai @robot-clickhouse @davenger @evillique @robot-ch-test-poll2 @robot-ch-test-poll1 @Felixoid @robot-ch-test-poll3 @justindeguzman @ltrk2 @robot-clickhouse-ci-2 @alesapin @lzydmxy @al13n321 @pershin-artem-gene @awfeequdng @den-crane @thomoco @robot-ch-test-poll4
REbase (#1)
4e7e47f 
* Disable debug logging

* Added check for backup/restore when they fail and status is not COMPLETED

* Fixed comment

* find big allocations without limits checks

* Fix usage of is_unbound in FileSegment

* Test WriteBufferToFileSegment with several threads

* Update test.py

* fix bad error messages

* Fix 02579_fill_empty_chunk.sql

* add perf test

* Fix krb5 with openssl

* Add statistical aggregate function kolmogorovSmirnovTest (ClickHouse#48325)

* doc entry for SYSTEM WAIT LOADING PARTS

* make CI slack bot less noisy

* Make remote bandwidth settings server settings

Right now those two settings are applied only at start:
- max_remote_read_network_bandwidth_for_server
- max_remote_write_network_bandwidth_for_server

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Move some methods for Context into module part

This will be required later, for per-query bandwidth limiting.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Add per-query network throttling

Controlled with:
- max_remote_read_network_bandwidth
- max_remote_write_network_bandwidth

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Add ability to throttle local IO on per-query/server basis

Server settings:
- max_local_read_bandwidth_for_server
- max_local_write_bandwidth_for_server

Query settings:
- max_local_read_bandwidth
- max_local_write_bandwidth

This is the preparation for adding ability to throttle BACKUPs

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Add ability to throttle BACKUPs on per-server/backup basis

Server settings:
- backup_read_bandwidth_for_server
- backup_write_bandwidth_for_server

Query settings:
- backup_read_bandwidth
- backup_write_bandwidth

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Keep only one throttler for BACKUPs IO (instead of separate read/write)

There is no need in separate read/write throttling, because you cannot
write faster then read anyway, and plus this makes the code less cleaner

(and also it will allow avoid implementing throttling backups to S3,
since it does not use common S3 writer).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Use existing ServerSettings in Context instead of reading it again

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Do not require proper Poco Application for Context

    $ yes | head | clickhouse-obfuscator --structure 'id String' --seed foo --input-format TSV --output-format TSV
    Poco::Exception. Code: 1000, e.code() = 0, Null pointer: _pInstance in file "/src/ch/clickhouse/base/poco/Util/include/Poco/Util/Application.h", line 446, Stack trace (when copying this message, always include the lines below):

    0. /src/ch/clickhouse/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x15605622 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    1. /src/ch/clickhouse/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x156055ed in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    2. /src/ch/clickhouse/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x2ebd2d80 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    3. /src/ch/clickhouse/base/poco/Foundation/src/Exception.cpp:132: Poco::LogicException::LogicException(String const&, int) @ 0x2ebd3667 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    4. /src/ch/clickhouse/base/poco/Foundation/src/Exception.cpp:134: Poco::NullPointerException::NullPointerException(String const&, int) @ 0x2ebd3da7 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    5. /src/ch/clickhouse/base/poco/Foundation/src/Bugcheck.cpp:42: Poco::Bugcheck::nullPointer(char const*, char const*, int) @ 0x2ebc5851 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    6. /src/ch/clickhouse/base/poco/Util/include/Poco/Util/Application.h:446: Poco::Util::Application::instance() @ 0x1ee3e6cd in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    7. /src/ch/clickhouse/src/Interpreters/Context.cpp:358: DB::ContextSharedPart::ContextSharedPart() @ 0x263b0cca in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    8. /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::ContextSharedPart>::__unique_single std::make_unique[abi:v15000]<DB::ContextSharedPart>() @ 0x26387b6e in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    9. /src/ch/clickhouse/src/Interpreters/Context.cpp:652: DB::Context::createShared() @ 0x26361a3c in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    10. /src/ch/clickhouse/programs/obfuscator/Obfuscator.cpp:1293: mainEntryClickHouseObfuscator(int, char**) @ 0x1ef6eb26 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    11. /src/ch/clickhouse/programs/main.cpp:481: main @ 0x155e569a in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    12. ? @ 0x7ffff7dd2790 in ?
    13. __libc_start_main @ 0x7ffff7dd284a in ?
    14. _start @ 0x155e51ae in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
     (version 23.3.1.2537)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Ignore deprecated values max_remote_{read,write}_network_bandwidth_for_server

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Use MAKE_DEPRECATED_BY_SERVER_CONFIG() for deprecated throttle settings

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Do not throttle S3-S3 backups if native copy is possible

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Remove knowledge about throttling from IBackupWriter::supportNativeCopy()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Remove throttler for IDisk::copyFile()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* AST Fuzzer: Fix assertion in TopK serialization

Problem:
  https://s3.amazonaws.com/clickhouse-test-reports/0/fa5b2bd4a5b02336bca8837c473a7124f8ecedf2/fuzzer_astfuzzerasan/report.html

The new assertion in the Varint code was introduced with (*). It rejects
values whose serialization cannot be deserialized (and this behavior
cannot be changed due to historical reasons). Such values should be
exceptionally rare in practice but AST fuzzer managers to trigger them.

The fix is similar to (**): Bypass the check by limiting the value to
the maximum allowed value.

(if AST fuzzer triggers finds more violations of the assertion, we might
consider throwing an exception instead)

(*) ClickHouse#48154
(**) https://github.com/ClickHouse/ClickHouse/pull/48154/files#diff-653c0a18dfdaa86262c78dc6b25550add0487f165b4ad053e86f530388f6203a

* Fix 02352_lightweight_delete flakiness (due to index granularity randomization)

CI: https://s3.amazonaws.com/clickhouse-test-reports/48242/c8da660d96bea153fa0c1dc8fbdb4a560e139a56/stateless_tests__debug__[4/5].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Set index granularity settings for other two lightweight delete tests

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix build

* Add max_ prefix for backup_bandwidth settings

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix flaky test_keeper_snapshots

* Do not take lock for shared context in setTempDataOnDisk

* Better

* Update docs/en/interfaces/formats.md

Co-authored-by: Yakov Olkhovskiy <99031427+yakov-olkhovskiy@users.noreply.github.com>

* Update temp_data_on_disk in shared context

* Move suffix generation code into one function

* Rename old_*_columns to input_*_columns

* avoid race, do not set thread_group.counters.parent from query

* Fix build, pt. II

* Fix tests

* Always communicate with local replica through loopback interface for parallel replicas (ClickHouse#48246)

* Fix special build

* Fix darwin build

* fix

* Add unbound flag to system.filesystem_cache

* Rename Context::getTemporaryVolume to getGlobalTemporaryVolume

* Reduce scope storage_policies_mutex in setTemporaryStoragePolicy

* Don't check dependencies when renaming system tables automatically

* add docs for UNDROP TABLE

* lost a word

* Fix test

* Update 02703_jit_external_aggregation.sh

* Better

* Don't replicate KeeperMap DELETE and TRUNCATE

* PR comments

* Update only affected rows in KV storage

* Fix special build

* Update replication.md

* transfer counters to the query directly

* Override user and password environment variables with client parameters

* Incorporate more review feedback

* Fix stuff

* Yet another fix

* fix test

* Better exception messages from Keeper client

* Add why clickhouse-local overview

* Small documentation follow-up to ClickHouse#47246

* Update 00002_log_and_exception_messages_formatting.sql

* Update 00002_log_and_exception_messages_formatting.reference

* Update CachedOnDiskReadBufferFromFile.cpp

* Add reading step for system zookeeper. Analyze path from filter DAG.

* Fix name

* Fix another test

* Fix deadlock due to debug tracking of memory allocations

Since memory tracker should be blocked globally (by default it is blocked only
for VariableContext::User).

<details>

<summary>stacktrace</summary>

    * thread 11, name = 'Collector', stop reason = trace
        frame 1: 0x00007ffff7e37ea2 libc.so.6`pthread_mutex_lock + 274
        frame 6: 0x000000001ee10298 clickhouse`toStringCached(pointers=0x00007fffedfef068, offset=0, size=45) + 88 at StackTrace.cpp:425
        frame 7: 0x000000001ee1128e clickhouse`StackTrace::toString(this=0x00007fffedfef058) const + 46 at StackTrace.cpp:444
        frame 8: 0x000000001ede58ff clickhouse`(anonymous namespace)::debugLogBigAllocationWithoutCheck(size=25187648) + 479 at MemoryTracker.cpp:98
        frame 9: 0x000000001ede5481 clickhouse`MemoryTracker::allocImpl(this=0x0000000030729e68, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x00007ffff3c22440) + 3937 at MemoryTracker.cpp:352
        frame 10: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007ffff3c1e530, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x00007ffff3c22440) + 607 at MemoryTracker.cpp:198
        frame 11: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007ffff3c22440, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 607 at MemoryTracker.cpp:198
        frame 12: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007fffedff3bf0, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 607 at MemoryTracker.cpp:198
        frame 13: 0x000000001ed6c4c9 clickhouse`CurrentMemoryTracker::allocImpl(size=25165824, throw_if_memory_exceeded=false) + 265 at CurrentMemoryTracker.cpp:58
        frame 14: 0x000000001ed6c6a1 clickhouse`CurrentMemoryTracker::allocNoThrow(size=25165824) + 33 at CurrentMemoryTracker.cpp:91
        frame 15: 0x000000001ed63079 clickhouse`operator new(unsigned long) [inlined] void Memory::trackMemory<>(size=25165824) + 89 at memory.h:177
        frame 16: 0x000000001ed63034 clickhouse`operator new(size=25165824) + 20 at new_delete.cpp:74
        ...
        frame 27: 0x000000001ee15895 clickhouse`DB::(anonymous namespace)::collectSymbols(info=0x00007ffff7062000, (null)=<unavailable>, data_ptr=0x00007fffec2b7000) + 53 at SymbolIndex.cpp:483
        frame 28: 0x000000002e1a5d57 clickhouse`::dl_iterate_phdr(callback=(clickhouse`DB::(anonymous namespace)::collectSymbols(dl_phdr_info*, unsigned long, void*) at SymbolIndex.cpp:480), data=0x00007fffec2b7000)(dl_phdr_info *, size_t, void *), void *) + 183 at phdr_cache.cpp:76
        frame 29: 0x000000001ee15549 clickhouse`DB::SymbolIndex::update(this=0x00007fffec2b7000) + 41 at SymbolIndex.cpp:515
        frame 30: 0x000000001ee1be35 clickhouse`DB::SymbolIndex::SymbolIndex(this=<unavailable>) + 53 at SymbolIndex.h:23
        frame 31: 0x000000001ee17a6c clickhouse`DB::SymbolIndex::instanceImpl() + 108 at SymbolIndex.cpp:555
        frame 32: 0x000000001ee17b4a clickhouse`DB::SymbolIndex::instance() + 10 at SymbolIndex.cpp:561
        frame 33: 0x000000001ee1070f clickhouse`void toStringEveryLineImpl<toStringCached(std::__1::array<void*, 45ul> const&, unsigned long, unsigned long)::$_0>(fatal=false, stack_trace=0x00007fffedff1ef8, callback=0x00007fffedff1ee0)::$_0&&) + 207 at StackTrace.cpp:349
        frame 34: 0x000000001ee1040b clickhouse`toStringCached(pointers=0x00007fffedff2928, offset=0, size=43) + 459 at StackTrace.cpp:436
        frame 35: 0x000000001ee1128e clickhouse`StackTrace::toString(this=0x00007fffedff2918) const + 46 at StackTrace.cpp:444
        frame 36: 0x000000001ede58ff clickhouse`(anonymous namespace)::debugLogBigAllocationWithoutCheck(size=25165824) + 479 at MemoryTracker.cpp:98
        frame 37: 0x000000001ede5481 clickhouse`MemoryTracker::allocImpl(this=0x00007fffedff3bf0, size=25165824, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 3937 at MemoryTracker.cpp:352
        frame 38: 0x000000001ed6c4c9 clickhouse`CurrentMemoryTracker::allocImpl(size=25165824, throw_if_memory_exceeded=false) + 265 at CurrentMemoryTracker.cpp:58
        frame 39: 0x000000001ed6c6a1 clickhouse`CurrentMemoryTracker::allocNoThrow(size=25165824) + 33 at CurrentMemoryTracker.cpp:91
        frame 40: 0x000000001ed63079 clickhouse`operator new(unsigned long) [inlined] void Memory::trackMemory<>(size=25165824) + 89 at memory.h:177
        frame 41: 0x000000001ed63034 clickhouse`operator new(size=25165824) + 20 at new_delete.cpp:74
        ...
        frame 55: 0x0000000025badace clickhouse`DB::WriteBufferFromS3::nextImpl(this=0x00007ffff3c58f00) + 366 at WriteBufferFromS3.cpp:110

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Update 00002_log_and_exception_messages_formatting.sql

* Update 00002_log_and_exception_messages_formatting.reference

* fix query cache with sparse columns

* Update docs/en/sql-reference/statements/undrop.md

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* Update docs/en/sql-reference/statements/undrop.md

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* Update docs/en/sql-reference/statements/undrop.md

* add note about MV

* Fix bytesSize() of zk SetRequest

* Remove dead code and unused dependencies

* Use std::string_view instead of strlen

* Use std::string::starts_with instead of a roll your own variant

* Reduce memory usage for multiple alter delete mutations.

* Fix equals.

* Update ClickHouse logo

* Mark a function inline

* Change ClickHouse logo to dark mode

* Parallel reading in FROM file()

* Fix header

* Fix test

* Sync replicas after inserts

* Fix nullptr to memcpy

* Use forward declaration of ThreadPool

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix 01548_parallel_parsing_max_memory.sh

* Fix: do not resize pipeline when there is no files to process (globs expands to empty set)

* fix test test_multiple_disks

* Fix bracketed-paste mode messing up password input in client reconnect

* fix a bug in sync replica

* fix REPLACE_RANGE

* Perf test

* suppress a bug in filesystem cache

* Fix 02477_projection_materialize_and_zero_copy flakienss (due to index granularity randomization)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Better exception message for ZSTD

* Fix typo in word acquired, in comment for lockForAlter method

* Fix 02293_formats_json_columns

* Fix test which can become flaky due to file() parallelization

* Fix 02286_mysql_dump_input_format

* A try to prevent possible tests flakyness due to file() parallelization

* Remove misleading comment

It is no longer "significant" after `NOEXCEPT_SCOPE` macro rework ClickHouse#39229

* Fix tests: truncate output file on insert

* Add support for {server_uuid} macro

* Add a test

* Update test

* Update 02207_allow_plaintext_and_no_password.sh

* Update formats.md

* bugfix: compare Bits and sizeof(Arithmetic) * 8

* Update tests/queries/0_stateless/02711_server_uuid_macro.sql

Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>

* Adopt tests to run in parallel or mark as no-parallel (for flaky check)

* Fix uncaught exception in case of parallel loader for hashed dictionaries

Since ThreadPool::wait() rethrows the first exception (if any):

<details>

<summary>stacktrace</summary>

    2023.04.09 12:53:33.629333 [ 22361 ] {} <Fatal> BaseDaemon: (version 22.13.1.1, build id: 5FB01DCAAFFF19F0A9A61E253567F90685989D2F) (from thread 23032) Terminate called for uncaught exception:
    2023.04.09 12:53:33.630179 [ 23645 ] {} <Fatal> BaseDaemon:
    2023.04.09 12:53:33.630213 [ 23645 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f68b00baccc 0x7f68b006bef2 0x7f68b0056472 0x112a42fe 0x1c17f2a3 0x1c17f238 0xbf4bc3b 0x13961c6d 0x138ee529 0x138ed6bc 0x138dd2f0 0x138dd9c6 0x1571d0dd 0x16197c1f 0x161a231e 0x1619fc93 0x161a51b9 0x11151759 0x1115454e 0x7f68b00b8fd4 0x7f68b013966c
    2023.04.09 12:53:33.630247 [ 23645 ] {} <Fatal> BaseDaemon: 3. ? @ 0x7f68b00baccc in ?
    2023.04.09 12:53:33.630263 [ 23645 ] {} <Fatal> BaseDaemon: 4. gsignal @ 0x7f68b006bef2 in ?
    2023.04.09 12:53:33.630273 [ 23645 ] {} <Fatal> BaseDaemon: 5. abort @ 0x7f68b0056472 in ?
    2023.04.09 12:53:33.648815 [ 23645 ] {} <Fatal> BaseDaemon: 6. ./.build/./src/Daemon/BaseDaemon.cpp:456: terminate_handler() @ 0x112a42fe in /usr/lib/debug/usr/bin/clickhouse.debug
    2023.04.09 12:53:33.651484 [ 23645 ] {} <Fatal> BaseDaemon: 7. ./.build/./contrib/llvm-project/libcxxabi/src/cxa_handlers.cpp:61: std::__terminate(void (*)()) @ 0x1c17f2a3 in /usr/lib/debug/usr/bin/clickhouse.debug
    2023.04.09 12:53:33.654080 [ 23645 ] {} <Fatal> BaseDaemon: 8. ./.build/./contrib/llvm-project/libcxxabi/src/cxa_handlers.cpp:79: std::terminate() @ 0x1c17f238 in /usr/lib/debug/usr/bin/clickhouse.debug
    2023.04.09 12:53:35.025565 [ 23645 ] {} <Fatal> BaseDaemon: 9. ? @ 0xbf4bc3b in /usr/lib/debug/usr/bin/clickhouse.debug
    2023.04.09 12:53:36.495557 [ 23645 ] {} <Fatal> BaseDaemon: 10. DB::ParallelDictionaryLoader<(DB::DictionaryKeyType)0, true, true>::~ParallelDictionaryLoader() @ 0x13961c6d in /usr/lib/debug/usr/bin/clickhouse.debug
    2023.04.09 12:53:37.833142 [ 23645 ] {} <Fatal> BaseDaemon: 11. DB::HashedDictionary<(DB::DictionaryKeyType)0, true, true>::loadData() @ 0x138ee529 in /usr/lib/debug/usr/bin/clickhouse.debug
    2023.04.09 12:53:39.124989 [ 23645 ] {} <Fatal> BaseDaemon: 12. DB::HashedDictionary<(DB::DictionaryKeyType)0, true, true>::HashedDictionary(DB::StorageID const&, DB::DictionaryStructure const&, std::__1::shared_ptr<DB::IDictionarySource>, DB::HashedDictionaryStorageConfiguration const&, std::__1::shared_ptr<DB::Block>) @ 0x138ed6bc in /usr/lib/debug/usr/bin/clickhouse.debug

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Simpler way to resize pipeline

* Update CCTZ

* Update deltasumtimestamp.md

* Update deltasumtimestamp.md

* Update deltasumtimestamp.md

* Update deltasumtimestamp.md

* Remove slow test from debug builds

* Fix tests for flaky check

* Add a test

* Better exception messages for unparsed Bool

* Fixup

* Don't use type conversion with String query parameters

* Fix

* Slightly optimize Install

* Fix TSan report in Kerberos

* Update src/Interpreters/ReplaceQueryParameterVisitor.cpp

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>

* Fixing build

* Docs: Add SETTINGS to executable table function

* Remove superfluous includes of logger_userful.h from headers

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Update formats.md

Including an Avro example with DESCRIBE url in S3

---------

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Co-authored-by: Igor Nikonov <igor@clickhouse.com>
Co-authored-by: Smita Kulkarni <Smita.Kulkarni@clickhouse.com>
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
Co-authored-by: vdimir <vdimir@clickhouse.com>
Co-authored-by: Vitaly Baranov <vitlibar@yandex.ru>
Co-authored-by: Dmitry Novik <n0vik@clickhouse.com>
Co-authored-by: robot-ch-test-poll <45356221+robot-ch-test-poll@users.noreply.github.com>
Co-authored-by: Robert Schulze <robert@clickhouse.com>
Co-authored-by: Anton Popov <anton@clickhouse.com>
Co-authored-by: Boris Kuschel <Boris.Kuschel@ibm.com>
Co-authored-by: FFFFFFFHHHHHHH <75292180+FFFFFFFHHHHHHH@users.noreply.github.com>
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
Co-authored-by: Rich Raposa <richraposa@gmail.com>
Co-authored-by: robot-clickhouse-ci-1 <118761991+robot-clickhouse-ci-1@users.noreply.github.com>
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
Co-authored-by: Antonio Andelic <antonio@clickhouse.com>
Co-authored-by: avogar <avogar@clickhouse.com>
Co-authored-by: Kruglov Pavel <48961922+Avogar@users.noreply.github.com>
Co-authored-by: Yakov Olkhovskiy <99031427+yakov-olkhovskiy@users.noreply.github.com>
Co-authored-by: Sema Checherinda <Sema.Checherinda@clickhouse.com>
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
Co-authored-by: Raúl Marín <git@rmr.ninja>
Co-authored-by: DanRoscigno <dan@roscigno.com>
Co-authored-by: kssenii <sumarokovakseniia@mail.ru>
Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
Co-authored-by: Nikita Mikhaylov <nikitamikhaylov@clickhouse.com>
Co-authored-by: Nikolai Kochetov <KochetovNicolai@users.noreply.github.com>
Co-authored-by: robot-clickhouse <41385210+robot-clickhouse@users.noreply.github.com>
Co-authored-by: Alexander Gololobov <440544+davenger@users.noreply.github.com>
Co-authored-by: Sema Checherinda <104093494+CheSema@users.noreply.github.com>
Co-authored-by: Nikolay Degterinsky <evillique@gmail.com>
Co-authored-by: robot-ch-test-poll2 <63349887+robot-ch-test-poll2@users.noreply.github.com>
Co-authored-by: Yakov Olkhovskiy <yakov@clickhouse.com>
Co-authored-by: robot-ch-test-poll1 <47390204+robot-ch-test-poll1@users.noreply.github.com>
Co-authored-by: Mikhail f. Shiryaev <felixoid@clickhouse.com>
Co-authored-by: robot-ch-test-poll3 <69307580+robot-ch-test-poll3@users.noreply.github.com>
Co-authored-by: Justin de Guzman <justin@justindeguzman.net>
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
Co-authored-by: Nikolai Kochetov <nik-kochetov@yandex-team.ru>
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
Co-authored-by: ltrk2 <107155950+ltrk2@users.noreply.github.com>
Co-authored-by: robot-clickhouse-ci-2 <118811860+robot-clickhouse-ci-2@users.noreply.github.com>
Co-authored-by: alesapin <alesapin@clickhouse.com>
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
Co-authored-by: lzydmxy <13126752315@163.com>
Co-authored-by: Michael Kolupaev <michael.kolupaev@clickhouse.com>
Co-authored-by: Artem Pershin <artem.pershin@genestack.com>
Co-authored-by: caipengxiang <291458254@qq.com>
Co-authored-by: Denny Crane <denis.zhuravlov@gmail.com>
Co-authored-by: Thom O'Connor <thom@clickhouse.com>
Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>
Co-authored-by: robot-ch-test-poll4 <69306974+robot-ch-test-poll4@users.noreply.github.com>
rschu1ze added a commit that referenced this pull request Apr 11, 2023
@rschu1ze
Better handling of values too large for VarInt encoding
bbbb55c 
PR #48154 introduced a sanity check in the form of a debug assertion
that the input values for VarInt encoding are not too big. Such values
should be exceptionally rare in practice but the AST fuzzer managed to
trigger the assertion regardless. The strategy to deal with such values
until now was to bypass the check by limiting the value to the maximum
allowed value (see #48412). Because a new AST Fuzzer failure appeared
(#48497) and there may be more failures in future, this PR changes the
sanity check from an assert to an exception.

Fixes: #48497
@rschu1ze rschu1ze mentioned this pull request Apr 11, 2023
Merged
@nikitamikhaylov nikitamikhaylov mentioned this pull request Jul 1, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rschu1ze rschu1ze rschu1ze approved these changes

Assignees

@rschu1ze rschu1ze

Labels
pr-not-for-changelog This PR should not be mentioned in the changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@azat @rschu1ze @robot-ch-test-poll2