Skip to content

[minor] add disable swap command#26797

Closed
xiedeyantu wants to merge 2991 commits intoapache:branch-2.0from
xiedeyantu:patch-2
Closed

[minor] add disable swap command#26797
xiedeyantu wants to merge 2991 commits intoapache:branch-2.0from
xiedeyantu:patch-2

Conversation

@xiedeyantu
Copy link
Member

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

LemonLiTree and others added 30 commits October 18, 2023 06:44
…nfig (apache#25595)

remove sf100 cascades specific control for global config, which is only for q72
1. add checks and handling of sequence column in apache#21896 to insert statement in origin planner and Nereids planner.
2. disable drop sequence mapping column in schema change.
apache#25500)

Support aggregate functions in select without from clause, here are some examples as following:

SELECT 1,  
  'a',
   COUNT(),  
   SUM(1) + 1,
   AVG(2) / COUNT(),
   MAX(3),
   MIN(4),
   RANK() OVER() AS w_rank,
   DENSE_RANK() OVER() AS w_dense_rank,
   ROW_NUMBER() OVER() AS w_row_number,
   SUM(5) OVER() AS w_sum,
   AVG(6) OVER() AS w_avg,
   COUNT() OVER() AS w_count,
   MAX(7) OVER() AS w_max,
   MIN(8) OVER() AS w_min;
For comparison predicate, two arguments must be cast to datetime and push down to storage if either one is date type. This PR disables predicate push-down for this case.
…sample (apache#25593)

1. To avoid misleading of inaccurate min max stats, set the stats value to NULL while using sample to collect stats.
2. Fix NDV_SAMPLE_TEMPLATE typo, it shouldn't contain row count related contents.
…he#25489)

* Using cgroup to implement cpu hard limit

* code style
keanji-x and others added 27 commits October 27, 2023 14:38
…e#26002)

Reject the edge which has an alias when ordering edge
could not run multi group_concat distinct with more than one parameters.
This bug is not just for group_concat, but we usually use literal as
parameters in group_concat. So group_concat brought the problem to light.

In the original logic, we think only distinct aggregate function with
zero or one parameter could run in multi distinct mode. But it is wrong.
We could process all distinct aggregate function with not more than one
input slots.

Think about sql:
```sql
SELECT
  group_concat(distinct c1, ','), group_concat(distinct c2, ',')
FROM t
GROUP BY c3
```
…#26003)

The image file of our cluster reaches 2.3G. After the checkpoint, Followers synchronize the image timeout, resulting in the continuous increase of the bdb directory.

related pr: apache#25768
Add tpcds sf100 hive shapes.

Disable query64 temporarily because it is not same with emr cluster after collecting metadata by analyze table xxx.
And the root cause need to analyze, will enable in future PR.
apache#25847)

Doris is not responsible for managing snapshots, but it needs to clear all
snapshots before doing backup/restore regression testing, so a property is
added to indicate that existing snapshots need to be cleared when creating a
repo.

In addition, a regression test case for backup/restore has been added.
… later) (apache#26048)

In branch 2.0, we changed the read/write method of AnalysisManager,
and rename the image module name to AnalysisMgrV2.
So we need to make the same change in master branch, so that user
can upgrade Doris from branch-2.0 to master branch.

After this PR, user can:

- upgrade from 2.0.x(or branch-2.0) to master
… type (apache#25470)

Previously,doris's `DateTimev2` was exported to orc as a `String` type.
Now, export doris's `DateTimev2` to orc timestamp type.
…e#25727)

In previous, when using file scan node(eq, querying hive table), the max number of scanner for each scan node
will be the `doris_scanner_thread_pool_thread_num`(default is 48).
And if the query parallelism is N, the total number of scanner would be 48 * N, which is too many.

In this PR, I change the logic, the max number of scanner for each scan node
will be the `doris_scanner_thread_pool_thread_num / query parallelism`. So that the total number of scanners
will be up to `doris_scanner_thread_pool_thread_num`.

Reduce the number of scanner can significantly reduce the memory usage of query.
example:
set ignore_runtime_filter_ids="3, 1";
after this setting, RF003 and RF001 will be ignored
@xiedeyantu xiedeyantu closed this Nov 10, 2023
@github-actions github-actions bot added area/pipeline kind/docs Categorizes issue or PR as related to documentation. labels Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/pipeline kind/docs Categorizes issue or PR as related to documentation.

Projects

None yet

Development

Successfully merging this pull request may close these issues.