[fix](cloud) provide a conf to enable/disable streamload commit on be (pick #37848) #37849
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ion tests on azure cloud (apache#36739) The file url's format on azure Blob is not S3-compatible. This pr returns the correct url when running on Azure Cloud.
support cast complex type to json which we can use array_agg/map_agg result into json_object
## Proposed changes introduce by apache#32759 when upgrade 2.1 to master, error happened: ``` java.io.EOFException: null at java.io.DataInputStream.readInt(DataInputStream.java:398) ~[?:?] at org.apache.doris.common.io.Text.readString(Text.java:413) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT] at org.apache.doris.load.loadv2.LoadJobFinalOperation.readFields(LoadJobFinalOperation.java:160) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.transaction.TxnCommitAttachment.read(TxnCommitAttachment.java:77) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.journal.JournalEntity.readFields(JournalEntity.java:569) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.journal.bdbje.BDBJournalCursor.next(BDBJournalCursor.java:115) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.replayJournal(Env.java:2863) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.transferToMaster(Env.java:1503) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.access$1300(Env.java:332) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env$5.runOneCycle(Env.java:2764) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT] ``` Using a comparison of the FE meta version to solve this problem. 2.1 fe meta version is 129, use the fe version 131 used by [pr](apache#32759) at that time as the comparison value.
## Proposed changes apache#36235 introduced, index page is not freed in query thread, so saved tracker ptr when alloc page, so that page only needs to be freed before query ends. ``` F20240624 18:44:51.880192 162249 mem_tracker_limiter.cpp:125] mem tracker label: Query#Id=f0511107829e465e-a87a985234018b77, consumption: 94, peak consumption: 636540, mem tracker not equal to 0 when mem tracker destruct, this usually means that memory tracking is inaccurate and SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not used correctly. 1. For query and load, memory leaks may have occurred, it is expected that the query mem tracker will be bound to the thread context using SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER before all memory alloc and free. 2. If a memory alloc is recorded by this tracker, it is expected that be recorded in this tracker when memory is freed. 3. Merge the remaining memory tracking value by this tracker into Orphan, if you observe that Orphan is not equal to 0 in the mem tracker web or log, this indicates that there may be a memory leak. 4. If you need to transfer memory tracking value between two trackers, can use transfer_to..[Address Sanitizer]: 0x605000160b30, size 46, strack trace: 0# Allocator<false, false, false>::alloc_impl(unsigned long, unsigned long) 1# doris::PageBase<Allocator<false, false, false> >::PageBase(unsigned long, bool, doris::segment_v2::PageTypePB) 2# doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) 3# doris::segment_v2::IndexedColumnReader::read_page(doris::segment_v2::PagePointer const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*, doris::segment_v2::PageTypePB, doris::BlockCompressionCodec*, bool) const 4# doris::segment_v2::IndexedColumnReader::load_index_page(doris::segment_v2::PagePointerPB const&, doris::segment_v2::PageHandle*, doris::segment_v2::IndexPageReader*) 5# doris::segment_v2::IndexedColumnReader::load(bool, bool) 6# doris::segment_v2::BitmapIndexReader::_load(bool, bool, std::unique_ptr<doris::segment_v2::BitmapIndexPB, std::default_delete<doris::segment_v2::BitmapIndexPB> >) 7# doris::segment_v2::BitmapIndexReader::load(bool, bool) 8# doris::segment_v2::ColumnReader::_load_bitmap_index(bool, bool) 9# doris::segment_v2::ColumnReader::new_bitmap_index_iterator(doris::segment_v2::BitmapIndexIterator**) 10# doris::segment_v2::Segment::new_bitmap_index_iterator(doris::TabletColumn const&, std::unique_ptr<doris::segment_v2::BitmapIndexIterator, std::default_delete<doris::segment_v2::BitmapIndexIterator> >*) 11# doris::segment_v2::SegmentIterator::_init_bitmap_index_iterators() 12# doris::segment_v2::SegmentIterator::init_iterators() 13# doris::segment_v2::SegmentIterator::_init_impl(doris::StorageReadOptions const&) 14# doris::segment_v2::SegmentIterator::init(doris::StorageReadOptions const&) 15# doris::segment_v2::Segment::new_iterator(std::shared_ptr<doris::Schema const>, doris::StorageReadOptions const&, std::unique_ptr<doris::RowwiseIterator, std::default_delete<doris::RowwiseIterator> >*) 16# doris::segment_v2::LazyInitSegmentIterator::init(doris::StorageReadOptions const&) 17# doris::BetaRowsetReader::_init_iterator() 18# doris::BetaRowsetReader::_init_iterator_once() 19# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) 20# doris::vectorized::VCollectIterator::_topn_next(doris::vectorized::Block*) 21# doris::vectorized::VCollectIterator::next(doris::vectorized::Block*) 22# doris::vectorized::BlockReader::_direct_next_block(doris::vectorized::Block*, bool*) 23# doris::vectorized::BlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) 24# doris::vectorized::NewOlapScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) 25# doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) 26# doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) 27# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) 28# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) 29# doris::ThreadPool::dispatch_thread() 30# doris::Thread::supervise_thread(void*) ```
Switch meta serialization to gson, including the following classes：
Database
catalog.Table
loadv2.LoadJob
loadv2.LoadJobFinalOperation
RestoreJob
BackuJob
CatalogRecyclebin
SmallFileMgr$SmallFile
Co-authored-by: iszhangpch
## Proposed changes Issue Number: close #xxx Fix cloud partial update UB error
…6735) ## Proposed changes fix wrong returned rows on parallel result sink
…pache#36758) Partition stats support truncate single partition. When truncate single partition, drop the stats of this partition and trigger auto analyze in next round.
…sce & ifnull (apache#36724) cherry-pick from branch-2.0 apache#36688, apache#36640, apache#36583 How to Reproduce: Issue1: use datetimev1 and datev2 as parameters in function coalesce admin set frontend config ("enable_date_conversion"="false"); admin set frontend config ("disable_datev1"="false"); drop table test_cls; CREATE TABLE `test_cls` ( `id` int(11) NOT NULL COMMENT '', `name` varchar(32) NOT NULL COMMENT '', `dt` datetime NOT NULL ) ENGINE=OLAP UNIQUE KEY(`id`) DISTRIBUTED BY HASH(`id`) BUCKETS 2 PROPERTIES( "replication_allocation" = "tag.location.default: 1" ); SET enable_nereids_planner=false; select dt from test_cls where coalesce (dt, str_to_date(concat('202306', '01'), '%Y%m%d')) >= '2023-06-01'; return no data, which is wrong. when see the excution plan SET enable_nereids_planner=false; explain select dt from test_cls where coalesce (dt, str_to_date(concat('202306', '01'), '%Y%m%d')) >= '2023-06-01'; you can see error predicate: CAST(coalesce(CAST(`dt` AS BIGINT), 20230601) AS DOUBLE) >= NULL after fix: CAST(coalesce(`dt`, '2023-06-01') AS DATETIMEV2(0)) >= '2023-06-01 00:00:00' Issue2: use datetimev2 and datev2 as parameters in function coalesce admin set frontend config ("enable_date_conversion"="true"); admin set frontend config ("disable_datev1"="true"); drop table test_cls; CREATE TABLE `test_cls` ( `id` int(11) NOT NULL COMMENT '', `name` varchar(32) NOT NULL COMMENT '', `dt` datetime NOT NULL ) ENGINE=OLAP UNIQUE KEY(`id`) DISTRIBUTED BY HASH(`id`) BUCKETS 2 PROPERTIES( "replication_allocation" = "tag.location.default: 1" ); SET enable_nereids_planner=false; select dt from test_cls where coalesce (dt, str_to_date(concat('202306', '01'), '%Y%m%d')) >= '2023-06-01'; return no data, which is wrong when see the excution plan SET enable_nereids_planner=false; explain select dt from test_cls where coalesce (dt, str_to_date(concat('202306', '01'), '%Y%m%d')) >= '2023-06-01'; you can see error predicate: CAST(coalesce(CAST(`dt` AS BIGINT), 20230601) AS DOUBLE) >= NULL after fix: coalesce(`dt`, '2023-06-01') >= '2023-06-01 00:00:00' Issue3: extend it to multiple parameters. Steps to reproduce: admin set frontend config ("enable_date_conversion"="false"); admin set frontend config ("disable_datev1"="false"); drop table test_cls; CREATE TABLE `test_cls` ( `id` int(11) NOT NULL COMMENT '', `name` varchar(32) NOT NULL COMMENT '', `dt` datetime NOT NULL ) ENGINE=OLAP UNIQUE KEY(`id`) DISTRIBUTED BY HASH(`id`) BUCKETS 2 PROPERTIES( "replication_allocation" = "tag.location.default: 1" ); SET enable_nereids_planner=false; error sql: select dt from test_cls where coalesce (dt, dt, str_to_date(concat('202306', '01'), '%Y%m%d')) >= '2023-06-01'; explain it and find wrong predicate: CAST(coalesce(CAST(`dt` AS BIGINT), CAST(`dt` AS BIGINT), 20230601) AS DOUBLE) >= NULL after fix: explain it coalesce(CAST(`dt` AS DATETIMEV2(0)), CAST(`dt` AS DATETIMEV2(0)), '2023-06-01 00:00:00') >= '2023-06-01 00:00:00' Issue3: nereids has the same issue with legacy planner
…2 to p0 (apache#36750) Move test `test_hive_to_date` from `external_table_p2` to `external_table_p0`, so that developer can run it locally with docker env. Guidance for how to migrate suites from hive p2 to hive p0: 1. Create suite dir under `docker/thirdparties/docker-compose/hive/scripts/suites/` eg: `docker/thirdparties/docker-compose/hive/scripts/suites/test_hive_to_date/` 2. Prepare the data and hql - Copy the table's data to `docker/thirdparties/docker-compose/hive/scripts/suites/test_hive_to_date/data/` and compress it. - If data is larger then 1MB, better upload to oss and download at runtime, see `docker/thirdparties/docker-compose/hive/scripts/suites/test_hive_compress_type/run.sh` - Prepare create table stmt like: `docker/thirdparties/docker-compose/hive/scripts/suites/test_hive_to_date/create_table.hql` 3. Create `run.sh` eg: `docker/thirdparties/docker-compose/hive/scripts/suites/test_hive_to_date/run.sh` This scripts will be called when the hive docker starts. It will: 1. put local data to hdfs in hive container. 2. create hive table in hive container.
…che#36541) ## Proposed changes before ``` mysql [(none)]>select sum(length(cast(number as string) )) from numbers("number" = "100000000"); +-----------------------------------+ | sum(length(cast(number as TEXT))) | +-----------------------------------+ | 788888890 | +-----------------------------------+ 1 row in set (7.24 sec) ``` now ``` mysql [(none)]>select sum(length(cast(number as string) )) from numbers("number" = "100000000"); +-----------------------------------+ | sum(length(cast(number as TEXT))) | +-----------------------------------+ | 788888890 | +-----------------------------------+ 1 row in set (1.40 sec) mysql [test]>select sum(length(cast(123 as string) )) from numbers("number" = "100000000"); +--------------------------------+ | sum(length(cast(123 as TEXT))) | +--------------------------------+ | 300000000 | +--------------------------------+ 1 row in set (0.09 sec) ``` More batch functions will be implemented in the future.
This pr has added support for generated columns. Values of a generated
column are computed from an expression included in the column
definition. Here is an example:
mysql> CREATE TABLE products (
product_id INT,
price DECIMAL(10,2),
quantity INT,
total_value DECIMAL(10,2) GENERATED ALWAYS AS (price * quantity)
) DISTRIBUTED BY HASH(product_id) PROPERTIES ("replication_num" = "1");
mysql> insert into products values(1, 10.00, 10, default);
mysql> insert into products(product_id, price, quantity) values(1, 20.00, 10);
mysql> select * from products;
+------------+-------+----------+-------------+
| product_id | price | quantity | total_value |
+------------+-------+----------+-------------+
| 1 | 10.00 | 10 | 100.00 |
| 1 | 20.00 | 10 | 200.00 |
+------------+-------+----------+-------------+
related docs in apache/doris-website#715
---------
Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
…#36763) The previous impl. depends on the order of views which may not be true when enable_advanced_id is true in fe.conf.
…he#36772) ## Proposed changes When the concurrency of streamload exceeds the number of threads in the remote scanner, streamload may get stuck. The reason is that the libevent thread blocks and waits for streamload to complete, and when there is no intersection between the tasks handled by the scanner thread and the libevent thread, it gets stuck. The solution is to convert the synchronous waiting tasks of libevent into asynchronous execution by using callbacks in the streamload executor thread.
…stability (apache#36770) When union rewrite by materialized view, the final plan chosen by CBO is instability. So the regression test only check mv is rewritten successful or not, doesn't check is chosen by CBO or not. Optimize to make sure chosen by CBO would be anther pr to fix this thoroughly。
We add one optional S3 load property `provider`, this pr adapts one case to test whether the FE works around without assigning one provider property.
…e#36732) ## Proposed changes In some cases, users want to set the start time to a time in the past. **For periodic scheduling tasks only, the start time of a one-time task must be set to the future or present time** **If the start time is the past time, then the next execution time should be the start time + time interval.**
…n, because low level mv aggregate roll up (apache#36567) Query is aggregate, the query group by expression is less than materialzied view group by expression. when the more dimensions than queries in materialzied view can be eliminated with functional dependencies. it can be rewritten with out roll up aggregate. For example as following: mv def is CREATE MATERIALIZED VIEW mv BUILD IMMEDIATE REFRESH AUTO ON MANUAL DISTRIBUTED BY RANDOM BUCKETS 2 PROPERTIES ('replication_num' = '1') AS select l_orderkey, l_partkey, l_suppkey, o_orderkey, o_custkey, ps_partkey, cast( sum( IFNULL(ps_suppkey, 0) * IFNULL(ps_partkey, 0) ) as decimal(28, 8) ) as agg2 from lineitem_1 inner join orders_1 on lineitem_1.l_orderkey = orders_1.o_orderkey inner join partsupp_1 on l_partkey = partsupp_1.ps_partkey and l_suppkey = partsupp_1.ps_suppkey where partsupp_1.ps_suppkey > 1 group by l_orderkey, l_partkey, l_suppkey, o_orderkey, o_custkey, ps_partkey; query is as following: select l_orderkey, l_partkey, l_suppkey, o_orderkey, o_custkey, cast( sum( IFNULL(ps_suppkey, 0) * IFNULL(ps_partkey, 0) ) as decimal(28, 8) ) as agg2 from lineitem_1 inner join orders_1 on lineitem_1.l_orderkey = orders_1.o_orderkey inner join partsupp_1 on l_partkey = partsupp_1.ps_partkey and l_suppkey = partsupp_1.ps_suppkey where partsupp_1.ps_suppkey > 1 group by l_orderkey, l_partkey, l_suppkey, o_orderkey, o_custkey; we can see that query doesn't use `ps_partkey` which is in mv group by expression. Normally will add roll up aggragate on materialized view if the gorup by dimension in mv is mucher than query group by dimension. And, in this scane we can get the function dependency on `l_suppkey = ps_suppkey `. and we doesn't need to add roll up aggregate on materialized view in rewritten plan. this improve performance and is beneficial for nest materialized view rewrite.
Fix two bugs that cause show create table index comment error 1. comment has extra single quotes. 2. quota does not escape in show create table result. ``` COMMENT ''test index" comment'' to COMMENT 'test index\" comment' ``` Refs PR: apache#6560 apache#28346
…s in CSV files are read as NULL values. (apache#36668) Previously, empty rows in CSV files were ignored by Doris. Now, add a session variable to control whether empty rows in CSV files are read as NULL values.
…he#36792) print table name when partition not exists in bind relation
apache#37490) Previously in fe, for files like `s3://qa-build/regression/tpcds/sf100_split/catalog_sales.dat.*.gz` it can not work.
…apache#37539) FIX: When dropping dynamic partition, PR apache#35778 will use math.max(start, -history_partition_num) as the first partition, but it may delete users' partitions if they specify both start and history_partition_num inappropriately. For safety reason, revert this behavious changed, only use start as the first partition when dropping partitions. For those who had specified a very small start value, drop partitions will catch an exception , and stop dropping this table's partition and then record this error in dynamic info. Users can use command `SHOW DYNAMIC PARTITION TABLES FROM DBXXX` to know this error. From this error, it will give user hint to modify start if they really specify a error start. --------- Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
## Proposed changes When paimon uses the `hms` type catalog and reads data in jni mode, it needs to use the `org.apache.commons.lang.StringUtils` class. (This problem is not tested in the pipeline environment because the pipeline environment automatically generates the `java-udf-case-jar-with-dependencies.jar` for testing, which contains the `lang` package.)
…pache#37342) When executing two SQL statements simultaneously,such as ``` DROP ROW POLICY IF EXISTS test_row_policy_3 on zd.user1;CREATE ROW POLICY test_row_policy_3 ON zd.user1 AS RESTRICTIVE TO role role1 USING (k1 in (1) or k2 in (2)); ``` fe will can not restart ``` Caused by: java.lang.ClassCastException: org.apache.doris.analysis.DropPolicyStmt cannot be cast to org.apache.doris.analysis.CreatePolicyStmt at org.apache.doris.policy.RowPolicy.gsonPostProcess(RowPolicy.java:169) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.persist.gson.GsonUtils$PostProcessTypeAdapterFactory$1.read(GsonUtils.java:640) ~[doris-fe.jar:1.2-SNAPSHOT] at com.google.gson.TypeAdapter.fromJsonTree(TypeAdapter.java:299) ~[gson-2.10.1.jar:?] ... 14 more ``` fix: - setOrigStmt use split sql instead of full sql - store index to cope with segmentation failure
…erator closed (apache#37297) Some kinds of source operators has a 1-1 relationship with a sink operator (such as AnalyticOperator). We must ensure AnalyticSinkOperator will not be blocked if AnalyticSourceOperator already closed.
… round (apache#37520) We have some problem in other rep. The problem can not be reproduced on master, so refine the test case to trace problem.
…e#37419) ## Proposed changes pick from master apache#37336
…phase if compaction on other BE finished during this load (apache#37670) ## Proposed changes Due to apache#35838, when executing load job, BE will not `sync_rowsets()` in publish phase if a compaction job is finished on another BE on the same tablet between the commit phase and the publish phase of the current load job. This PR let the meta service return the tablet compaction stats along with the getDeleteBitmapUpdateLockResponse to FE and FE will send them to BE to let the BE know whether it should `sync_rowsets()` due to compaction on other BEs.
## Proposed changes Pick from apache#37745
## Proposed changes rollback opt in apache#34134
## Proposed changes pick apache#37762 Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
…apache#37789) ## Proposed changes pick from apache#37788 Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
Query following createting table would throw table not exist error. For example. t1: client issue create table to master fe t2: client issue query sql to observer fe, the query would fail due to not exist table in plan phase. t3: observer fe receive editlog creating the table from the master fe After the pr: query at t2 would wait until latest edit log is received from master fe in the observer fe.
## Proposed changes 1. ignore rf when rf-mgr released 2. move acquire rf controller to after acquire query_ctx on send_filter_size 3. enlarge timeout limit on sync_filter_size/apply_filterv2 4. logout rf's debug string when rpc meet error
…e dropped apache#37809 (apache#37832) ## Proposed changes pick apache#37809 Issue Number: close #xxx <!--Describe your changes.-->
When to enable commit on be? Bypassing FE during commit is good for load performance, especially when doing freqent streamloading. When to disable commit on be? The current implementation of bypass will mess up the statistics, thus the query plan optimizer. So disable it when you encoutering any troubles. Such problem will be fixed in future work. Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
freemandealer
requested review from
CalvinKirs,
dataroaring,
gavinchou,
morningman and
platoneko
as code owners
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When to enable commit on be?
Bypassing FE during commit is good for load performance, especially when doing frequent streamloading.
When to disable commit on be?
The current implementation of the bypass will mess up the statistics, thus the query plan optimizer. So disable it when you encountering any troubles. Such problem will be fixed in future work.