[BugFix] Fix crash during decoding min/max when type mismatch in parquet file #8849
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
imay
reviewed
Jul 18, 2022
|
@dirtysalt I think you should explain the reason why this PR will fix the crash. |
dirtysalt
commented
Jul 18, 2022
DorianZheng
reviewed
Jul 18, 2022
|
LGTM |
|
I get a very weird ASN stacktrace only when pipeline mode enabled . It looks like |
|
It's because release order of I find a quite subtle problem in Suppose we have following expr
So the elements in pool are ( GE, ctx, a, 2). And when we call
The fix is easy, to put |
5 tasks
|
LGTM |
imay
approved these changes
Jul 18, 2022
imay
approved these changes
Jul 18, 2022
caneGuy
approved these changes
Jul 19, 2022
|
run starrocks_admit_test |
gengjun-git
approved these changes
Jul 19, 2022
|
run starrocks_admit_test |
1 similar comment
|
run starrocks_admit_test |
|
pr regression 580 passed |
hiliuxg
pushed a commit
to hiliuxg/starrocks
that referenced
this pull request
Jul 21, 2022
[Refactor] rename work group to resource group (StarRocks#8527) Update release-2.0.md (StarRocks#8809) [BugFix] fix bug in runtime filter plan (StarRocks#8728) [Starrocks On ES] support explain for ES table (StarRocks#8202) [Refactor] Remove CreateTableStmt in old parser (StarRocks#8824) [SR SQL Planner Summer Camp] Support modify partition in new Parser and Analyzer (StarRocks#8485) Optimize 'DataUsedCapacity' display (StarRocks#8823) [SR SQL Planner Summer Camp] Support show partitions in new Parser and Analyzer (StarRocks#8489) [BugFix] BrokerLoad cann't handle kerberos login with multiple keytab (StarRocks#8820) UserGroupInformation.loginUserFromKeytab(principal, keytab) will modify the loginUserInfo globally. If two users concurrently start broker load, the later one will override previous one's loginUserInfo. Remove unused test case (StarRocks#8832) [Actions] add documentation label for pr (StarRocks#8841) * update label trigger * add pr documentation label Co-authored-by: dulong <dulong@starrocks.com> [Refactor] Remove cluster in ConnectContext (StarRocks#8826) Update release-2.3.md (StarRocks#8833) Fix be crash in ASAN mode (StarRocks#8722) If element of ArrayColumn is empty, we don't need to read data from data_column of ArrayColumn because data is empty. So we don't add an empty read_range into element_read_ranges, which will cause inconsistency between the begin of element_read_range and element_ordinal of element_iterator. That's why be crash in ASAN mode. [BugFix] potential dead lock in stream_load_pipe (StarRocks#8647) _push_front is only called when no_block_read is timeout, we already acquire lock at that time(condition variable will immediately acquire lock after wait for). so there is no need to acquire the lock again in _push_front [Others] Add config for librdkafka debug (StarRocks#8783) [Enhancement] Assign tablet ranges for one-phase aggregate without partition and colocate (StarRocks#8771) Fix two unstable FE UT (StarRocks#8758) Use real GlobalStateMgr, not mock GlobalStateMgr, ensure there is only one GlobalStateMgr in UT, then won't Missing invocations issue. [Enhancement] Adjust downgrade strategy of chunks_partitioner (StarRocks#8696) remove dead code mutable_index_* (StarRocks#8857) [BugFix] set bdb timeout to 1 minutes for UT (StarRocks#8855) [Enhancement] Optimize logic of materialized view when base table is dropped (StarRocks#8613) Optimize logic of materialized view when base table is dropped Fix unstable RollupJobV2Test (StarRocks#8876) mv task definition view (StarRocks#8856) select * from information_schema.task_runs; select * from information_schema.tasks; definition add insert overwrite [Refactor] Add getOriginName in Database class (StarRocks#8846) Add getOriginName and change some Log and Exception to expose origin name other than full name. Update release-2.3.md (StarRocks#8874) [Refactor] ShowDynamicPartitionStmt to new Parser and Analyzer (StarRocks#8511) [Feature] New Parser and Analyzer support ShowProcesslistStmt (StarRocks#8671) New Parser and Analyzer support ShowProcesslistStmt Support recover partition in new Parser and Analyzer (StarRocks#8493) docs add navigation (StarRocks#8864) [Refactor] Fix UT (StarRocks#8878) [Refactor] Remove used code (StarRocks#8880) [Refactor] Support BDBJEJournal with prefix (StarRocks#8672) [DOC] add validity period during which users can restore a deleted table (StarRocks#8788) * add validity period during which users can restore a deleted table * Update DROP TABLE.md * Update DROP TABLE.md [SR SQL Planner Summer Camp][Feature] alter table statement support of add list partition (StarRocks#8805) [BugFix] Fix crash during decoding min/max when type mismatch in parquet file (StarRocks#8849) [Refactor] Force reorder columns of chunk (StarRocks#8597) [Refactor] Refactor hdfs scanner expression usage (StarRocks#8884) [BugFix] Fix runtime filter delivery (StarRocks#8840) This PR fixes: 1. different wait timeout on scan node(`_scan_wait_timeout_ms`) scan and other nodes(`_wait_timeout_ms`) 2. put global runtime filter into cache(`runtime_filter_cache`) if fragment instance is not allocated yet. 3. And during wait time, nodes on fragment instance will find runtime filter from cache(`runtime_filter_cache`) [Enhancement] introduce query debug trace for BE (StarRocks#7649) display data_type in information_schema.columns for (StarRocks#8895) hll/bitmap/percentile [Bug Fix]show partitions add absence column for filter columns check (StarRocks#8894) [Enhance] take the fragment prepare time into account of vruntime (StarRocks#8865) delete two sql statement (StarRocks#8901) [Refactor] remove radix_sort (StarRocks#8891) [BugFix] Fix OlapTableSink close accelerate release resource (StarRocks#8893) [Bugfix] fix be crash when call DictMappingExpr::evaluate (StarRocks#8858) Remove dead function of primitive type size (StarRocks#8912) Revert "[SR SQL Planner Summer Camp][Feature] alter table statement support of add list partition (StarRocks#8805)" (StarRocks#8917) This reverts commit fb28ebe. [BugFix] Fix consistency problem (StarRocks#8896) [Refactor] [Step-1]Change the name from master to leader (StarRocks#8787) Change the master related code to leader. Split the task into two steps: 1. change the name from master to leader in the internal code paths. 2. change the name from master to leader in the interface. The advantage of this is that we only need to rollback the step 2 if If there are compatibility problems. Remove some old Analyzer codes 4 (StarRocks#8872) [BugFix] Fix potential dangling pointer of TabletSchema in Segment (StarRocks#8913) For lake tablet, the raw pointer of TabletSchema in Segment may point to a memory location that has been deleted, if the cached TabletSchema has been evicted from the in-memory cache of lake::TabletManager. Use a std::shared_ptr to manage the ownership of TabletSchema in Segment to fix the issue. [Lake] Disable primary key lake table (StarRocks#8924) [Refactor] Remove admin user related code (StarRocks#8921) [Refactor] Renaming constant variable in persistent index (StarRocks#8862) [bug fix]fix TaskManager initialDelay not correct (StarRocks#8633) when timeunit is not second,delay will not correct. [Enhancement] ThreadPoolToken releasing the tasks outside of lock (StarRocks#8920) the `threadpool` is from kudu: https://github.com/apache/kudu/blob/master/src/kudu/util/threadpool.cc ``` // Clear the queue under the lock, but defer the releasing of the tasks // outside the lock, in case there are concurrent threads wanting to access // the ThreadPool. The task's destructors may acquire locks, etc, so this // also prevents lock inversions. ``` ``` F0720 10:02:59.244752 123894 threadpool.cpp:183] Check failed: _entries.empty() ``` if we declare `destructor = default`，compiler will disable the compiler-generated move constructor and move assignment operator. [BugFix] Support shuffle broker load for non duplicate keys table (StarRocks#8714) Before we unify the ingestion interface, we solved the multi-replica inconsistency problem of non duplicate key table by adding shuffle service. [Refactor] move state listener out of GlobalStateMgr, and rename it to state change executor (StarRocks#8747) [Feature] Support truncate lake table (StarRocks#8886) use constant to indicate length and add a getter for kv_pair_size. (StarRocks#8885) [Feature] introduce jemalloc into thirdparty (StarRocks#6238) [others] Fix unstable RefreshTableStmtTest (StarRocks#8951) [Enhance] make re2 driver-local to avoid contention (StarRocks#8904) [BugFix] fix runtime filter cache use: hold shared pointer to avoid invalid pointer. (StarRocks#8944) Got following exception ``` *** Aborted at 1658293232 (unix time) try "date -d @1658293232" if you are using GNU date *** PC: @ 0x5cc1f60 __dynamic_cast *** SIGSEGV (@0xfffffffffffffff0) received by PID 32857 (TID 0x7f1df458d700) from PID 18446744073709551600; stack trace: *** @ 0x41c0692 google::(anonymous namespace)::FailureSignalHandler() @ 0x7f1e2fe6be9b os::Linux::chained_handler() @ 0x7f1e2fe7090c JVM_handle_linux_signal @ 0x7f1e2fe63858 signalHandler() @ 0x7f1e2f34b630 (unknown) @ 0x5cc1f60 __dynamic_cast @ 0x29a8031 starrocks::vectorized::OrcChunkReader::_add_runtime_filter() @ 0x29ba656 starrocks::vectorized::OrcChunkReader::set_conjuncts_and_runtime_filters() @ 0x1b1e41a starrocks::vectorized::HdfsOrcScanner::do_open() @ 0x1b17259 starrocks::vectorized::HdfsScanner::open() @ 0x173ceae starrocks::connector::HiveDataSource::_init_scanner() @ 0x173f6e0 starrocks::connector::HiveDataSource::open() @ 0x1c5ef30 starrocks::vectorized::ConnectorScanNode::_scanner_thread() @ 0x314c348 starrocks::PriorityThreadPool::work_thread() @ 0x414ccc7 thread_proxy @ 0x7f1e2f343ea5 start_thread @ 0x7f1e2e95eb0d __clone @ 0x0 (unknown) ``` It's because we just use runtime filter raw pointer. But runtime filter could be released by runtime filte cache, but we probably are still using it. The robust way is to hold share ptr of runtime filter. ---- the reason why we don't need following code is because, if there are multiple fragment instances executed, some of them finish early and close queyr(clean runtime filter cache), following fragment instance won't get runtime filter from cache. There is a `clean_thread` in runtime filter cache, and that will clean runtime filter has not retrived used for a long time ``` _exec_env->runtime_filter_cache()->remove(_query_id); ``` New Parser and Analyzer support User Defined Function-statements (StarRocks#8528) [BugFix] Disable one stage aggregate with one distinct function (StarRocks#8918) [Refactor] [Step 1/2] Change the name from master to leader (StarRocks#8953) Change the master related code to leader. Split the task into two steps: change the name from master to leader in the internal code paths. change the name from master to leader in the interface. The advantage of this is that we only need to rollback the step 2 if If there are compatibility problems. This is PR is still the step 1 of the task. The previous work is done by StarRocks#8787 Add move constructor and move assignment for Schema (StarRocks#8937) Otherwise the `Schema` is not movable because we define the `constructor`; [Feature] histogram statistics support collect top-n data (StarRocks#8916) [BugFix] When the status of HeartBeat is not ok, it also need to be synced to follower (StarRocks#8265) When master the received heartbeat status is not ok, the heartbeat info also need to be synced to follower. Otherwise, the failed heartbeat information will not be synchronized to the follower. Since the failed heartbeat info also modifies fe's memory, this.heartbeatRetryTimes++; if it is not synchronized to the follower, this will cause the master and follower's metadata to be inconsistent. [Refactor] Remove some unused code in old parser for admin stmt (StarRocks#8963) 1. Some code in old parser is unused, it can be removed. 2. Fix the bug for SetReplicaStatus in the new parser. [BugFix] Persist `LoadStatistic` in `EtlStatus` (StarRocks#8689) The main reason is that LoadJob did not persist LoadStatistic in the process of either replaying or snapshot. Unfortunately, there is no obvious way to add new metadata since we persist the LoadJob class in a hard-coded way. After due consideration, we serialized LoadStatics in JSON for the convenience of adding/deleting fields in the future and hide this JSON in a deprecated map in EtlStatus. [Bugfix] Fix wrong result when process 'is null' in condition expr in dictionary optimization (StarRocks#8869) [Feat] Support COM_FILED_LIST in external catalog database (StarRocks#8929) Optimize materialized view refresh task by passing properties to task (StarRocks#8860) Optimize materialized view refresh task by passing properties to task through hints [Lake] Create shard with cache property (StarRocks#8977) fix explain row count overflow (StarRocks#8980) [BugFix] Fix error of THRIFT_RPC_ERROR with CN (StarRocks#8773) [SR SQL Planner Summer Camp]New Parser and Analyzer support AlterUserStmt (StarRocks#8503) [Enhancement] Support rewrite window function sort partition by in lowcardinality optimization (StarRocks#8957) relation with StarRocks#6119 [BugFix] Fix LocalTabletsChannel head-use-after-free (StarRocks#8978) Fix StarRocks#8906 [Enhancement] Remove redundant management on open db (StarRocks#8335) Remove redundant management on open db in BDBEnvironment. Close database initiatively in BDBJournal & BDBJournalCursor In every main loop, Replayer will update names of local databases by calling refresh(), then keep reading increasing log id until it doesn't exist. [Refactor] Extract common logics to success_once (StarRocks#8952)
7 tasks
dirtysalt
added a commit
that referenced
this pull request
Aug 17, 2022
…hen type mismatch in parquet(#8849)
9 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this：
Which issues of this PR fixes ：
Fixes ##8848
Problem Summary(Required) ：
It's because we ignore return status right here.
Status::NotSupportedcan be taken as success ofdecode_min_max_columnor not. Because right down there, following code could also returnStaus::NotSupported, but this error is fatal.