Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement]mv task definition view #8856

Merged
merged 1 commit into from Jul 18, 2022
Merged

[Enhancement]mv task definition view #8856

merged 1 commit into from Jul 18, 2022

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Jul 18, 2022 •
edited

What type of PR is this：

  • bug
  • feature
  • enhancement
  • refactor
  • others

Which issues of this PR fixes ：

Fixes #

Problem Summary(Required) ：

select * from information_schema.task_runs;

select * from information_schema.tasks;

definition add insert overwrite

@zddr zddr changed the title mv task definition view [Enhancement]mv task definition view Jul 18, 2022
@sonarcloud
Copy link

sonarcloud bot commented Jul 18, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@zddr
Copy link
Contributor Author

zddr commented Jul 18, 2022

run starrocks_fe_unittest

@wanpengfei-git
Copy link
Collaborator

[FE PR Coverage Check]

😍 pass : 3 / 3 (100.00%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/scheduler/MvTaskRunProcessor.java 1 1 100.00% []
🔵 com/starrocks/scheduler/TaskBuilder.java 2 2 100.00% []

ABingHuang
ABingHuang approved these changes Jul 18, 2022
View reviewed changes
Copy link
Contributor

@ABingHuang ABingHuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wanpengfei-git wanpengfei-git added the Approved Ready to merge label Jul 18, 2022
@wanpengfei-git
Copy link
Collaborator

run starrocks_admit_test

@imay imay merged commit d3e8325 into StarRocks:main Jul 18, 2022
melt-code pushed a commit to melt-code/starrocks that referenced this pull request Jul 18, 2022
@zddr @melt-code
mv task definition view (StarRocks#8856)
c869a0d 
select * from information_schema.task_runs;

select * from information_schema.tasks;

definition add insert overwrite
hiliuxg pushed a commit to hiliuxg/starrocks that referenced this pull request Jul 21, 2022
@adzfolc @hiliuxg
[Feature]add UT for create external table like DDL (StarRocks#8804)
3d9dc10 
[Refactor] rename work group to resource group (StarRocks#8527)

Update release-2.0.md (StarRocks#8809)

[BugFix] fix bug in runtime filter plan (StarRocks#8728)

[Starrocks On ES] support explain for ES table (StarRocks#8202)

[Refactor] Remove CreateTableStmt in old parser (StarRocks#8824)

[SR SQL Planner Summer Camp] Support modify partition in new Parser and Analyzer (StarRocks#8485)

Optimize 'DataUsedCapacity' display (StarRocks#8823)

[SR SQL Planner Summer Camp] Support show partitions in new Parser and Analyzer (StarRocks#8489)

[BugFix] BrokerLoad cann't handle kerberos login with multiple keytab (StarRocks#8820)

UserGroupInformation.loginUserFromKeytab(principal, keytab) will modify the loginUserInfo globally.
If two users concurrently start broker load, the later one will override previous one's loginUserInfo.

Remove unused test case (StarRocks#8832)

[Actions] add documentation label for pr  (StarRocks#8841)

* update label trigger

* add pr documentation label

Co-authored-by: dulong <dulong@starrocks.com>

[Refactor] Remove cluster in ConnectContext (StarRocks#8826)

Update release-2.3.md (StarRocks#8833)

Fix be crash in ASAN mode (StarRocks#8722)

If element of ArrayColumn is empty, we don't need to read data from data_column of ArrayColumn because data is empty.
So we don't add an empty read_range into element_read_ranges, which will cause inconsistency between the begin of element_read_range and element_ordinal of element_iterator. That's why be crash in ASAN mode.

[BugFix] potential dead lock in stream_load_pipe (StarRocks#8647)

_push_front is only called when no_block_read is timeout, we already acquire lock
at that time(condition variable will immediately acquire lock after wait for).
so there is no need to acquire the lock again in _push_front

[Others] Add config for librdkafka debug (StarRocks#8783)

[Enhancement] Assign tablet ranges for one-phase aggregate without partition and colocate (StarRocks#8771)

Fix two unstable FE UT (StarRocks#8758)

Use real GlobalStateMgr, not mock GlobalStateMgr, ensure there is only one GlobalStateMgr in UT, then won't Missing invocations issue.

[Enhancement] Adjust downgrade strategy of chunks_partitioner (StarRocks#8696)

remove dead code mutable_index_* (StarRocks#8857)

[BugFix] set bdb timeout to 1 minutes for UT (StarRocks#8855)

[Enhancement] Optimize logic of materialized view when base table is dropped (StarRocks#8613)

Optimize logic of materialized view when base table is dropped

Fix unstable RollupJobV2Test (StarRocks#8876)

mv task definition view (StarRocks#8856)

select * from information_schema.task_runs;

select * from information_schema.tasks;

definition add insert overwrite

[Refactor] Add getOriginName in Database class (StarRocks#8846)

Add getOriginName and change some Log and Exception to expose origin name other than full name.

Update release-2.3.md (StarRocks#8874)

[Refactor] ShowDynamicPartitionStmt to new Parser and Analyzer (StarRocks#8511)

[Feature] New Parser and Analyzer support ShowProcesslistStmt (StarRocks#8671)

New Parser and Analyzer support ShowProcesslistStmt

Support recover partition in new Parser and Analyzer (StarRocks#8493)

docs add navigation (StarRocks#8864)

[Refactor] Fix UT (StarRocks#8878)

[Refactor] Remove used code (StarRocks#8880)

[Refactor] Support BDBJEJournal with prefix (StarRocks#8672)

[DOC] add validity period during which users can restore a deleted table (StarRocks#8788)

* add validity period during which users can restore a deleted table

* Update DROP TABLE.md

* Update DROP TABLE.md

[SR SQL Planner Summer Camp][Feature] alter table statement support of add list partition (StarRocks#8805)

[BugFix] Fix crash during decoding min/max when type mismatch in parquet file (StarRocks#8849)

[Refactor] Force reorder columns of chunk (StarRocks#8597)

[Refactor] Refactor hdfs scanner expression usage (StarRocks#8884)

[BugFix] Fix runtime filter delivery (StarRocks#8840)

This PR fixes:
1. different wait timeout on scan node(`_scan_wait_timeout_ms`) scan and other nodes(`_wait_timeout_ms`)
2. put global runtime filter into cache(`runtime_filter_cache`) if fragment instance is not allocated yet.
3. And during wait time, nodes on fragment instance will find runtime filter from cache(`runtime_filter_cache`)

[Enhancement] introduce query debug trace for BE (StarRocks#7649)

display data_type in information_schema.columns for (StarRocks#8895)

hll/bitmap/percentile

[Bug Fix]show partitions add absence column for filter columns check (StarRocks#8894)

[Enhance] take the fragment prepare time into account of vruntime (StarRocks#8865)

delete two sql statement (StarRocks#8901)

[Refactor] remove radix_sort (StarRocks#8891)

[BugFix] Fix OlapTableSink close accelerate release resource (StarRocks#8893)

[Bugfix] fix be crash when call DictMappingExpr::evaluate (StarRocks#8858)

Remove dead function of primitive type size (StarRocks#8912)

Revert "[SR SQL Planner Summer Camp][Feature] alter table statement support of add list partition (StarRocks#8805)" (StarRocks#8917)

This reverts commit fb28ebe.

[BugFix] Fix consistency problem (StarRocks#8896)

[Refactor] [Step-1]Change the name from master to leader (StarRocks#8787)

Change the master related code to leader.
Split the task into two steps:
1. change the name from master to leader in the internal code paths.
2. change the name from master to leader in the interface.
The advantage of this is that we only need to rollback the step 2 if If there are compatibility problems.

Remove some old Analyzer codes 4 (StarRocks#8872)

[BugFix] Fix potential dangling pointer of TabletSchema in Segment (StarRocks#8913)

For lake tablet, the raw pointer of TabletSchema in Segment may point
to a memory location that has been deleted, if the cached TabletSchema
has been evicted from the in-memory cache of lake::TabletManager.

Use a std::shared_ptr to manage the ownership of TabletSchema in Segment
to fix the issue.

[Lake] Disable primary key lake table (StarRocks#8924)

[Refactor] Remove admin user related code (StarRocks#8921)

[Refactor] Renaming constant variable in persistent index (StarRocks#8862)

[bug fix]fix TaskManager initialDelay not correct (StarRocks#8633)

when timeunit is not second,delay will not correct.

[Enhancement] ThreadPoolToken releasing the tasks outside of lock (StarRocks#8920)

the `threadpool` is from kudu: https://github.com/apache/kudu/blob/master/src/kudu/util/threadpool.cc

```
    // Clear the queue under the lock, but defer the releasing of the tasks
    // outside the lock, in case there are concurrent threads wanting to access
    // the ThreadPool. The task's destructors may acquire locks, etc, so this
    // also prevents lock inversions.
```

```
F0720 10:02:59.244752 123894 threadpool.cpp:183] Check failed: _entries.empty()
```

if we declare `destructor = default`，compiler will disable the compiler-generated move constructor and move assignment operator.

[BugFix] Support shuffle broker load for non duplicate keys table (StarRocks#8714)

Before we unify the ingestion interface, we solved the multi-replica inconsistency problem of non duplicate key table by adding shuffle service.

[Refactor] move state listener out of GlobalStateMgr, and rename it to state change executor (StarRocks#8747)

[Feature] Support truncate lake table (StarRocks#8886)

use constant to indicate length and add a getter for kv_pair_size. (StarRocks#8885)

[Feature] introduce jemalloc into thirdparty (StarRocks#6238)

[others] Fix unstable RefreshTableStmtTest (StarRocks#8951)

[Enhance] make re2 driver-local to avoid contention (StarRocks#8904)

[BugFix] fix runtime filter cache use: hold shared pointer to avoid invalid pointer. (StarRocks#8944)

Got following exception

```
*** Aborted at 1658293232 (unix time) try "date -d @1658293232" if you are using GNU date ***
PC: @          0x5cc1f60 __dynamic_cast
*** SIGSEGV (@0xfffffffffffffff0) received by PID 32857 (TID 0x7f1df458d700) from PID 18446744073709551600; stack trace: ***
    @          0x41c0692 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f1e2fe6be9b os::Linux::chained_handler()
    @     0x7f1e2fe7090c JVM_handle_linux_signal
    @     0x7f1e2fe63858 signalHandler()
    @     0x7f1e2f34b630 (unknown)
    @          0x5cc1f60 __dynamic_cast
    @          0x29a8031 starrocks::vectorized::OrcChunkReader::_add_runtime_filter()
    @          0x29ba656 starrocks::vectorized::OrcChunkReader::set_conjuncts_and_runtime_filters()
    @          0x1b1e41a starrocks::vectorized::HdfsOrcScanner::do_open()
    @          0x1b17259 starrocks::vectorized::HdfsScanner::open()
    @          0x173ceae starrocks::connector::HiveDataSource::_init_scanner()
    @          0x173f6e0 starrocks::connector::HiveDataSource::open()
    @          0x1c5ef30 starrocks::vectorized::ConnectorScanNode::_scanner_thread()
    @          0x314c348 starrocks::PriorityThreadPool::work_thread()
    @          0x414ccc7 thread_proxy
    @     0x7f1e2f343ea5 start_thread
    @     0x7f1e2e95eb0d __clone
    @                0x0 (unknown)
```

It's because we just use  runtime filter raw pointer. But runtime filter could be released by runtime filte cache, but we probably are still using it. The robust way is to hold share ptr of runtime filter.

----

the reason why we don't need following code is because, if there are multiple fragment instances executed, some of them  finish early and close queyr(clean runtime filter cache), following fragment instance won't get runtime filter from cache.

There is a `clean_thread` in runtime filter cache, and that will clean runtime filter has not retrived used for a long time

```
_exec_env->runtime_filter_cache()->remove(_query_id);
```

New Parser and Analyzer support User Defined Function-statements (StarRocks#8528)

[BugFix] Disable one stage aggregate with one distinct function (StarRocks#8918)

[Refactor] [Step 1/2] Change the name from master to leader (StarRocks#8953)

Change the master related code to leader.
Split the task into two steps:

change the name from master to leader in the internal code paths.
change the name from master to leader in the interface.
The advantage of this is that we only need to rollback the step 2 if If there are compatibility problems.

This is PR is still the step 1 of the task.
The previous work is done by StarRocks#8787

Add move constructor and move assignment for Schema (StarRocks#8937)

Otherwise the `Schema` is not movable because we define the `constructor`;

[Feature] histogram statistics support collect top-n data (StarRocks#8916)

[BugFix] When the status of HeartBeat is not ok, it also need to be synced to follower (StarRocks#8265)

When master the received heartbeat status is not ok, the heartbeat info also need to be synced to follower.
Otherwise, the failed heartbeat information will not be synchronized to the follower.
Since the failed heartbeat info also modifies fe's memory, this.heartbeatRetryTimes++;
if it is not synchronized to the follower, this will cause the master and follower's metadata to be inconsistent.

[Refactor] Remove some unused code in old parser for admin stmt (StarRocks#8963)

1. Some code in old parser is unused, it can be removed.
2. Fix the bug for SetReplicaStatus  in the new parser.

[BugFix] Persist `LoadStatistic` in `EtlStatus` (StarRocks#8689)

The main reason is that LoadJob did not persist LoadStatistic in the process of either replaying or snapshot. Unfortunately, there is no obvious way to add new metadata since we persist the LoadJob class in a hard-coded way. After due consideration, we serialized LoadStatics in JSON for the convenience of adding/deleting fields in the future and hide this JSON in a deprecated map in EtlStatus.

[Bugfix] Fix wrong result when process 'is null' in condition expr in dictionary optimization (StarRocks#8869)

[Feat] Support COM_FILED_LIST in external catalog database (StarRocks#8929)

Optimize materialized view refresh task by passing properties to task (StarRocks#8860)

Optimize materialized view refresh task by passing properties to task through hints

[Lake] Create shard with cache property (StarRocks#8977)

fix explain row count overflow (StarRocks#8980)

[BugFix] Fix error of THRIFT_RPC_ERROR with CN (StarRocks#8773)

[SR SQL Planner Summer Camp]New Parser and Analyzer support AlterUserStmt (StarRocks#8503)

[Enhancement] Support rewrite window function sort partition by in lowcardinality optimization (StarRocks#8957)

relation with StarRocks#6119

[BugFix] Fix LocalTabletsChannel head-use-after-free (StarRocks#8978)

Fix StarRocks#8906

[Enhancement] Remove redundant management on open db (StarRocks#8335)

Remove redundant management on open db in BDBEnvironment.
Close database initiatively in BDBJournal & BDBJournalCursor
In every main loop, Replayer will update names of local databases by calling refresh(), then keep reading increasing log id until it doesn't exist.

[Refactor] Extract common logics to success_once (StarRocks#8952)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Astralidea Astralidea Astralidea approved these changes

@chaoyli chaoyli chaoyli approved these changes

@ABingHuang ABingHuang ABingHuang approved these changes

Assignees
No one assigned
Labels
Approved Ready to merge sig/materialized view
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@zddr @wanpengfei-git @Astralidea @chaoyli @ABingHuang @imay