Backport #12256 to 20.5: throw exception on redirect limit in S3 request #12369
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* asiana21-DOCSUP-925-max_parser_depth (#132) * docs(max_parser_depth): added the setting description * docs(max_parser_depth): some changes * Update docs/en/operations/settings/settings.md Co-authored-by: BayoNet <da-daos@yandex.ru> * Update docs/en/operations/settings/settings.md Co-authored-by: BayoNet <da-daos@yandex.ru> * Update docs/en/operations/settings/settings.md Co-authored-by: BayoNet <da-daos@yandex.ru> * docs(max_parser_depth): added ru translation * docs(max_parser_depth): removed quotation marks Co-authored-by: asiana21 <asiana21@yandex-team.ru> Co-authored-by: BayoNet <da-daos@yandex.ru> * CLICKHOUSEDOCS-522: Fixed the link. Co-authored-by: AsiaKorushkina <43650329+AsiaKorushkina@users.noreply.github.com> Co-authored-by: asiana21 <asiana21@yandex-team.ru> Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
- add support of unbundled curl - add CURL::libcurl libraries - avoid explicit linkage of daemon with curl (added with sentry) - set CACHE variables for non-direct users: - mariadb-connector-c - aws-s3-cmake - sentry-native Cc: @alexey-milovidov Cc: @alesapin (requires docker image update) Cc: @abyss7 Refs: #11300 Refs: #8011 Refs: #8905 v2: replace cmake/find/curl.cmake with proper contrib/curl-cmake (as pointed by @abyss7, cmake/find/*.cmake is deprecated)
After #11599 it is possible that messages of the ReadBufferFromKafkaConsumer will be cleaned-up right in read_kafka_message callback (from KafkaBlockInputStream) if the stop flag isset (i.e. DROP TABLE is waiting the consumer), and if read_kafka_message already processed some rows it will not return 0 and the loop after will try to get current topic from the buffer, which uses messages in the underlying and this will got SIGSEGV: 12:14:56.173262 [ 55421 ] {f7930856-d478-4e41-af56-24ce7b693e95} <Debug> executeQuery: (from 0.0.0.0:0, user: ) DROP TABLE IF EXISTS data.queue 12:14:56.173285 [ 55421 ] {f7930856-d478-4e41-af56-24ce7b693e95} <Trace> StorageKafka (newly_queue): Waiting for cleanup 12:14:56.180016 [ 55390 ] {} <Trace> BaseDaemon: Received signal 11 12:14:56.180267 [ 4914 ] {} <Fatal> BaseDaemon: ######################################## 12:14:56.181879 [ 4914 ] {} <Fatal> BaseDaemon: (version 20.6.1.1, build id: 4CE0298F08583658) (from thread 55468) (no query) Received signal Segmentation fault (11) 12:14:56.181900 [ 4914 ] {} <Fatal> BaseDaemon: Address: 0x8 Access: read. Address not mapped to object. 12:14:56.181909 [ 4914 ] {} <Fatal> BaseDaemon: Stack trace: 12:14:56.184676 [ 4914 ] {} <Fatal> BaseDaemon: 3. /ch/contrib/cppkafka/include/cppkafka/message.h:111: DB::KafkaBlockInputStream::readImpl() @ 0xe343f1c in /usr/lib/debug/usr/bin/clickhouse 12:14:56.185553 [ 4914 ] {} <Fatal> BaseDaemon: 4. /ch/contrib/libcxx/include/vector:1003: DB::IBlockInputStream::read() @ 0xd9d95bd in /usr/lib/debug/usr/bin/clickhouse 12:14:56.188238 [ 4914 ] {} <Fatal> BaseDaemon: 5. /ch/src/DataStreams/copyData.cpp:26: DB::copyData() @ 0xd9f712a in /usr/lib/debug/usr/bin/clickhouse 12:14:56.188780 [ 4914 ] {} <Fatal> BaseDaemon: 6. /ch/contrib/libcxx/include/vector:1532: DB::StorageKafka::streamToViews() @ 0xe335e73 in /usr/lib/debug/usr/bin/clickhouse 12:14:56.189331 [ 4914 ] {} <Fatal> BaseDaemon: 7. /ch/src/Storages/Kafka/StorageKafka.cpp:491: DB::StorageKafka::threadFunc() @ 0xe336738 in /usr/lib/debug/usr/bin/clickhouse 55421 thread (shows that it still waiting for deactivation): 5 std::__1::lock_guard<>::lock_guard () at ../contrib/libcxx/include/__mutex_base:90 6 DB::BackgroundSchedulePoolTaskInfo::deactivate (this=0x7fc7e4465f20) at ../src/Core/BackgroundSchedulePool.cpp:59 7 DB::StorageKafka::shutdown (this=0x7fc7e45e4600) at ../contrib/libcxx/include/memory:3821 And just in case thread where read_kafka_message is called: 0 DB::ReadBufferFromKafkaConsumer::nextImpl (this=0x7fd4901d4118) at ../contrib/libcxx/include/atomic:1491 1 DB::ReadBuffer::next (this=0x7fd4901d4118) at ../src/IO/ReadBuffer.h:59 2 DB::ReadBuffer::eof (this=0x7fd4901d4118) at ../src/IO/ReadBuffer.h:81 3 DB::skipWhitespaceIfAny (buf=...) at ../src/IO/ReadHelpers.h:945 4 DB::JSONEachRowRowInputFormat::readRow (ext=..., columns=..., this=0x7fd499a7a020) at ../src/Processors/Formats/Impl/JSONEachRowRowInputFormat.cpp:222 5 DB::JSONEachRowRowInputFormat::readRow (this=0x7fd499a7a020, columns=..., ext=...) at ../src/Processors/Formats/Impl/JSONEachRowRowInputFormat.cpp:218 6 DB::IRowInputFormat::generate (this=0x7fd499a7a020) at ../src/Processors/Formats/IRowInputFormat.cpp:64 7 DB::ISource::work (this=0x7fd499a7a020) at ../src/Processors/ISource.cpp:48 8 DB::KafkaBlockInputStream::<lambda()>::operator()(void) const () at ../contrib/libcxx/include/memory:3826 9 DB::KafkaBlockInputStream::readImpl (this=0x7fd46e718820) at ../contrib/libcxx/include/new:340 Cc: @filimonov
This will help with tracking possible issues, when you need to know was buffer released or not.
Before this patch isStalled() was checked before polledDataUnusable(), and after DROP TABLE isStalled() == true (although this looks tricky).
As stated by @filimonov it is not relevant (after #11599)
Fix limiting the number of threads for VIEW.
* [docs] add intrdocution for statements page * Update index.md
Fix transform query for external databases in presense of aliases
Adding a simple example of using TestFlows.
…lized-uint8 Fixed logical functions for UInt8 values when they are not equal to 0 or 1.
Exclude partition key columns from SummingMergeTree
Initial explain
Fix minor issues in ternary logic
Added performance test to resemble "questionable" benchmark
throw exception on redirect limit in S3 request
…41488c1d85a55c6db1e8f163f5029 Cherry pick #12256 to 20.5: throw exception on redirect limit in S3 request
robot-clickhouse
added
the
pr-backport
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Original pull-request #12256
Cherry-pick pull-request #12357
This pull-request is a last step of an automated backporting.
Treat it as a standard pull-request: look at the checks and resolve conflicts.
Merge it only if you intend to backport changes to the target branch, otherwise just close it.