feat(frontend): add InternalStateTable Catalog #3139

WindowsXp-Beta · 2022-06-10T12:13:08Z

What's changed and what's your intention?

use Table Message to represent InternalStateTable.
add Table Message in prost of Stream Agg & Join.

Next Step(will be done in next PR)

fill the schemaId, databaseId and name(it will have a encoding so we can filter internal tables from normal tables) of InternalTable in meta.
add InternalTable notify in meta.

Checklist

I have written necessary docs and comments
I have added necessary unit tests and integration tests
All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

part of #3053

skyzh

Rest generally LGTM, I'll continue review after new changes are added.

skyzh · 2022-06-13T03:41:04Z

src/frontend/src/catalog/internal_state_table_catalog.rs

+    }
+}
+
+impl From<ProstInternalStateTable> for InternalStateTableCatalog {


The catalog looks incorrect? Every agg function has its own encoding schema. Anyway, we can refactor this later when we eventually move state stable schema inference to frontend.

Can we leave all properties empty for now, so that future developers won't be confused?

skyzh · 2022-06-13T03:42:15Z

proto/stream_plan.proto

@@ -91,6 +94,8 @@ message HashJoinNode {
  uint32 left_table_id = 7;
  // Used for internal table states. Id of the right table.
  uint32 right_table_id = 8;
+  catalog.InternalStateTable left_table = 9;


Can we merge this with left_table_id?

yezizp2012

Rest LGTM.

yezizp2012 · 2022-06-13T05:35:55Z

proto/stream_plan.proto

  repeated uint32 table_ids = 3;
  bool append_only = 4;
+  catalog.InternalStateTable table = 5;


Suggested change

repeated uint32 table_ids = 3;

bool append_only = 4;

catalog.InternalStateTable table = 5;

repeated catalog.InternalStateTable internal_tables = 3;

bool append_only = 4;

yezizp2012 · 2022-06-13T05:36:34Z

proto/stream_plan.proto

  repeated uint32 table_ids = 3;
  bool append_only = 4;
+  catalog.InternalStateTable table = 5;


Suggested change

repeated uint32 table_ids = 3;

bool append_only = 4;

catalog.InternalStateTable table = 5;

repeated catalog.InternalStateTable internal_tables = 3;

bool append_only = 4;

yezizp2012 · 2022-06-13T05:42:49Z

proto/catalog.proto

+  uint32 schema_id = 2;
+  uint32 database_id = 3;


schema_id and database_id are not necessary. And I think we may still need associated_table_id field, which is necessary for permission check when query on internal state.

…te-table-catalog

codecov · 2022-06-15T03:42:15Z

Codecov Report

Merging #3139 (23dc3c9) into main (1e190dd) will increase coverage by 0.04%.
The diff coverage is 85.14%.

@@            Coverage Diff             @@
##             main    #3139      +/-   ##
==========================================
+ Coverage   73.69%   73.74%   +0.04%     
==========================================
  Files         760      760              
  Lines      103884   104108     +224     
==========================================
+ Hits        76561    76770     +209     
- Misses      27323    27338      +15

Flag	Coverage Δ
rust	`73.74% <85.14%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ontend/src/stream_fragmenter/rewrite/delta_join.rs	`0.00% <0.00%> (ø)`
src/meta/src/stream/mod.rs	`39.13% <0.00%> (-1.78%)`	⬇️
src/stream/src/from_proto/global_simple_agg.rs	`0.00% <0.00%> (ø)`
src/stream/src/from_proto/hash_agg.rs	`0.00% <0.00%> (ø)`
src/stream/src/from_proto/hash_join.rs	`0.00% <0.00%> (ø)`
src/meta/src/stream/stream_graph.rs	`79.85% <22.22%> (-0.58%)`	⬇️
...rc/frontend/src/optimizer/plan_node/logical_agg.rs	`96.48% <81.35%> (-1.26%)`	⬇️
src/common/src/catalog/mod.rs	`60.44% <100.00%> (+1.85%)`	⬆️
...rontend/src/optimizer/plan_node/stream_hash_agg.rs	`95.94% <100.00%> (+0.78%)`	⬆️
...ontend/src/optimizer/plan_node/stream_hash_join.rs	`97.77% <100.00%> (+0.63%)`	⬆️
... and 9 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

st1page · 2022-06-15T04:02:10Z

src/frontend/src/optimizer/plan_node/logical_agg.rs

+impl LogicalAgg {
+    pub fn infer_internal_table_catalog(
+        &self,
+        plan_node: PlanRef,


this parameter seems unnecessary? and why implement it in logicalAgg instead of streamHashAgg and streamSimpleAgg?

this parameter seems unnecessary?

exactly. My fault.

why implement it in logicalAgg instead of streamHashAgg and streamSimpleAgg?

because I think their implementations are same (maybe I miss something?

st1page · 2022-06-15T04:07:10Z

src/frontend/src/optimizer/plan_node/logical_agg.rs

+        for agg_call in &self.agg_calls {
+            let mut internal_pk_indices = vec![];
+            let mut columns = vec![];
+            let mut order_desc = vec![];


we need add the group by keys as the prefix of the table's pk

src/frontend/src/optimizer/plan_node/stream_hash_join.rs

…te-table-catalog

xx01cyx · 2022-06-17T06:01:50Z

src/frontend/src/optimizer/plan_node/logical_agg.rs

+impl LogicalAgg {
+    pub fn infer_internal_table_catalog(&self) -> (Vec<TableCatalog>, HashMap<usize, i32>) {
+        let mut table_catalogs = vec![];
+        let mut column_index2columb_id = HashMap::new();


Suggested change

let mut column_index2columb_id = HashMap::new();

let mut column_index2column_id = HashMap::new();

what a typo🥲

xx01cyx · 2022-06-17T06:01:56Z

src/meta/src/stream/mod.rs

-            hash_mapping_manager.set_fragment_state_table(fragment_id, node.left_table_id);
-            hash_mapping_manager.set_fragment_state_table(fragment_id, node.right_table_id);
+            hash_mapping_manager
+                .set_fragment_state_table(fragment_id, node.left_table.clone().unwrap().id);


Suggested change

.set_fragment_state_table(fragment_id, node.left_table.clone().unwrap().id);

.set_fragment_state_table(fragment_id, node.left_table.as_ref().unwrap().id);

And there are other clones that could be avoided in this way.

…te-table-catalog

st1page · 2022-06-17T07:13:52Z

proto/stream_plan.proto

+  catalog.Table left_table = 6;
+  // Used for internal table states.
+  catalog.Table right_table = 7;
  repeated uint32 dist_key_l = 8;
  repeated uint32 dist_key_r = 9;


maybe we can merge these fields. but delaying this after the executor really uses the catalog in proto is ok.

wcy-fdu · 2022-06-20T03:19:30Z

Any update for this PR?

WindowsXp-Beta · 2022-06-20T03:53:28Z

Any update for this PR?

I intended to add meta part in another PR. Maybe I can commit directly to this one.

st1page

LGTM

…te-table-catalog

commit 309ce36 Author: Bowen <36908971+BowenXiao1999@users.noreply.github.com> Date: Tue Jun 21 13:19:44 2022 +0800 refactor(agg): clean up unused fields & refactor (#3339) * refactor(agg): clean up unused fields * delete file Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit e48dced Author: Shmiwy <wyf000219@126.com> Date: Tue Jun 21 12:55:21 2022 +0800 feat(storage): support compression setting per level (#3362) Signed-off-by: Shmiwy <wyf000219@126.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 961936f Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 12:42:45 2022 +0800 feat(test): parallelize sqlsmith test (#3360) * feat(test): parallelize sqlsmith test Signed-off-by: Alex Chi <iskyzh@gmail.com> * more tests Signed-off-by: Alex Chi <iskyzh@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit fa90541 Author: TennyZhuang <zty0826@gmail.com> Date: Tue Jun 21 12:25:33 2022 +0800 chore(github): feature-request template should use the feature label (#3359) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 02405e1 Author: Bowen <36908971+BowenXiao1999@users.noreply.github.com> Date: Tue Jun 21 12:13:07 2022 +0800 style: add more comments & refactor on pg-wire (#3358) style: add more comments on pg-wire code Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 1c09432 Author: Li0k <yuli@singularity-data.com> Date: Tue Jun 21 12:00:37 2022 +0800 fix(storage): fix slow unit-test in compactor_test (#3357) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 19036a6 Author: xxchan <37948597+xxchan@users.noreply.github.com> Date: Tue Jun 21 05:48:08 2022 +0200 fix(binder): do not allow correlated subquery in join tables (#3352) * fix(binder): do not allow correlated subquery in join tables * clippy Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit c37c9c4 Author: TennyZhuang <zty0826@gmail.com> Date: Tue Jun 21 11:24:40 2022 +0800 refactor: remove unnecessary lazy_static (#3353) Signed-off-by: TennyZhuang <zty0826@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 68596ab Author: Croxx <mrcroxx@outlook.com> Date: Tue Jun 21 11:11:53 2022 +0800 feat(cache): introduce LruCacheEventListener to subscribe erasure and eviction (#3334) commit 5627f25 Author: Steven Chua <stevengkc714@protonmail.com> Date: Tue Jun 21 10:54:12 2022 +0800 feat(ctl): Display SstableIdInfo and Block Metadata in sst-dump (#3338) * feat(ctl): Add sst-dump command to risectl * feat(ctl): Fix risectl compatibility and remove VNode info * feat(ctl): Add checksum and compression algo for each block * feat(ctl): Add SstableIdInfo data to sst-dump * feat(ctl): Fix compilation errors * feat(ctl): Fix compilation errors and bugs commit 16ffd98 Author: Name1e5s <name1e5s@qq.com> Date: Tue Jun 21 10:50:33 2022 +0800 fix(expr): cast int16/int32/int64/float32 to float64 in floor/ceil/round (#3319) * fix(expr): cast int16/int32/int64/float32 to float64 in floor/ceil/round * fix plan Co-authored-by: TennyZhuang <zty0826@gmail.com> commit 964bb92 Author: TennyZhuang <zty0826@gmail.com> Date: Tue Jun 21 10:23:56 2022 +0800 ci(Mergify): configuration update (#3355) Signed-off-by: null <zty0826@gmail.com> commit dac904e Author: jon-chuang <9093549+jon-chuang@users.noreply.github.com> Date: Tue Jun 21 09:57:33 2022 +0800 feat(executor): streaming hyperloglog improvements (#3315) * minor * rename tests * minor * remove option, const eval of param, better comments, succint tests commit cd4f302 Author: TennyZhuang <zty0826@gmail.com> Date: Tue Jun 21 09:34:54 2022 +0800 ci(Mergify): configuration update (#3252) * ci(Mergify): configuration update Signed-off-by: null <zty0826@gmail.com> * Update .mergify.yml * Update .mergify.yml * Update .mergify.yml Co-authored-by: xxchan <37948597+xxchan@users.noreply.github.com> commit 2aa7e8e Author: Xinpeng Wei <windowsxp@sjtu.edu.cn> Date: Mon Jun 20 22:11:51 2022 +0800 feat(frontend): add InternalStateTable Catalog (#3139) * use TableMessage for internal table * fix risedev check * update planner test * fix unit test * fix misc check * fix ci * fix issues in PR comments * fix clippy * fix ci * update planner test commit 1e190dd Author: xxchan <37948597+xxchan@users.noreply.github.com> Date: Mon Jun 20 14:16:12 2022 +0200 fix(binder): do not allow correlated input ref in order by (#3346) commit 263d770 Author: Tao Wu <wutao@singularity-data.com> Date: Mon Jun 20 18:54:25 2022 +0800 fix: build failure caused by OptimzierContext::new (#3340) commit 9f18401 Author: Tao Wu <wutao@singularity-data.com> Date: Mon Jun 20 17:47:22 2022 +0800 feat: introduce the framework of sqlsmith (#3305) commit 096a991 Author: Alex Chi <iskyzh@gmail.com> Date: Mon Jun 20 17:40:05 2022 +0800 feat(ctl): add bench command (#3337) Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 075d596 Author: TennyZhuang <zty0826@gmail.com> Date: Mon Jun 20 17:05:50 2022 +0800 build: bump toolchain to 20220620 (#3324) * build: bump toolchain to 20220620 Signed-off-by: TennyZhuang <zty0826@gmail.com> * also update docker-compose Signed-off-by: TennyZhuang <zty0826@gmail.com> commit d864f30 Author: Wenzhuo Liu <lwzbill@foxmail.com> Date: Mon Jun 20 16:38:21 2022 +0800 feat: add output_indices to join executors (#3047) commit 13b9d58 Author: StrikeW <wangsiyuanse@gmail.com> Date: Mon Jun 20 16:29:56 2022 +0800 feat(stream): enable append-only mv plan for kafka source (#3333) commit ea386d3 Author: Liang <44948473+soundOfDestiny@users.noreply.github.com> Date: Mon Jun 20 15:50:55 2022 +0800 refactor(compaction): deprecate the HashStrategy for OverlapStrategy (#3331) commit a9fba38 Author: Liang <44948473+soundOfDestiny@users.noreply.github.com> Date: Mon Jun 20 15:34:25 2022 +0800 fix(picker): fetch info from table_id field in sstableinfo (#3332) commit 5d2bb42 Author: Bohan Zhang <tabvision@bupt.icu> Date: Mon Jun 20 14:55:59 2022 +0800 test(stream): add ci for split change mutation in source (#3039) * stage Signed-off-by: tabVersion <tabvision@bupt.icu> * stage Signed-off-by: tabVersion <tabvision@bupt.icu> * add test Signed-off-by: tabVersion <tabvision@bupt.icu> * change e2e to datagen Signed-off-by: tabVersion <tabvision@bupt.icu> * stage Signed-off-by: tabVersion <tabvision@bupt.icu> * some bug to fix Signed-off-by: tabVersion <tabvision@bupt.icu> * fix async issue Signed-off-by: tabVersion <tabvision@bupt.icu> * add assert Signed-off-by: tabVersion <tabvision@bupt.icu> commit 04fe6d6 Author: Liang <44948473+soundOfDestiny@users.noreply.github.com> Date: Mon Jun 20 14:47:56 2022 +0800 refactor(vnode bitmap): remove vnode bitmap in sst info (#3329) commit c6d1288 Author: Li0k <yuli@singularity-data.com> Date: Mon Jun 20 14:29:56 2022 +0800 feat(storage): add manual compaction picker for targeted compaction (#3288) * feat(storage): add ManualCompactionPicker * feat(storage): distinguish get_compaction_task for manual * feat(storage): meta client support more parameters for manual_compaction * chore(storage): add tracing and some notes * chore(storage): split manual_compaction_picker to independent file * feat(storage): fix target_input check pending and support manual_pick for dynamic_level_selector * fix(storage): internal_table_id include mv_id * fix(storage): fix picker check target_input_ssts pending * fix(storage): fix picker with total_file_size commit d04954f Author: Renjie Liu <liurenjie2008@gmail.com> Date: Mon Jun 20 14:27:01 2022 +0800 fix(ci): Reduce log (#3330) commit eca9239 Author: Bugen Zhao <i@bugenzhao.com> Date: Mon Jun 20 13:59:56 2022 +0800 refactor(storage): remove `Option` on pk serializer of cell-based table (#3328) * minor refactor Signed-off-by: Bugen Zhao <i@bugenzhao.com> * remove option of pk serializer Signed-off-by: Bugen Zhao <i@bugenzhao.com> * remove pk serializer in state table Signed-off-by: Bugen Zhao <i@bugenzhao.com> * extract vnode compute Signed-off-by: Bugen Zhao <i@bugenzhao.com> * remove into order types Signed-off-by: Bugen Zhao <i@bugenzhao.com> commit 9169436 Author: Bowen <36908971+BowenXiao1999@users.noreply.github.com> Date: Mon Jun 20 13:44:10 2022 +0800 feat: apply relational refactor for hash agg (max, min) (#2999) * feat: two closure can not get mut ref of same variable * use Arc::Mutex to wrap the state table * roll back string agg * add StateTable to get_output * finish basic coding (unit test failed) * finish basic coding * fix bug * show case * use empty Row for scan * tweak commit 35bb16a Author: Bugen Zhao <i@bugenzhao.com> Date: Mon Jun 20 13:43:11 2022 +0800 refactor: use packed bitmap struct for vnode bitmap (#3310) * use bitmap in streaming Signed-off-by: Bugen Zhao <i@bugenzhao.com> * use bitmap in storage Signed-off-by: Bugen Zhao <i@bugenzhao.com> * minor fix Signed-off-by: Bugen Zhao <i@bugenzhao.com> * make bitmap optional Signed-off-by: Bugen Zhao <i@bugenzhao.com> commit be50f93 Author: Liang <44948473+soundOfDestiny@users.noreply.github.com> Date: Mon Jun 20 13:27:37 2022 +0800 feat(compaction): let compactor be unaware of vnode mapping (#3321) commit 88258a6 Author: lmatz <lmatz823@gmail.com> Date: Sun Jun 19 21:31:01 2022 -0700 doc: no need to manually check in PR from forks (#3325) commit c590e18 Author: zwang28 <70626450+zwang28@users.noreply.github.com> Date: Mon Jun 20 12:13:50 2022 +0800 refactor(storage): split HummockVersion's levels by compaction group. (#3206) commit f1c3298 Author: zwang28 <70626450+zwang28@users.noreply.github.com> Date: Mon Jun 20 11:35:18 2022 +0800 feat(meta): register source to compaction group manager (#3300) commit bf08b54 Author: Zack <52342064+nhzaci@users.noreply.github.com> Date: Mon Jun 20 11:25:39 2022 +0800 feat(frontend): Add sql string into context for debugging (#3312) * feat(frontend): Add sql string into context for debugging * Remove renaming * Refactor to use str commit f784ba3 Author: zwang28 <70626450+zwang28@users.noreply.github.com> Date: Mon Jun 20 11:22:12 2022 +0800 feat(storage): shared buffer flush L0 by compaction group (#3200) commit bd48bba Author: Kexiang Wang <kx.wang@hotmail.com> Date: Sun Jun 19 07:48:45 2022 -0400 feat: modify interfaces to support specifying parallelism for each fr… (#3283) feat: modify interfaces to support specifying parallelism for each fragment commit 8f0e0b2 Author: Steven Chua <stevengkc714@protonmail.com> Date: Sun Jun 19 12:56:51 2022 +0800 feat(ctl): Support basic sst dump in risectl (#3309) * feat(ctl): Add sst-dump command to risectl * feat(ctl): Fix risectl compatibility and remove VNode info commit daf9222 Author: Alex Chi <iskyzh@gmail.com> Date: Sat Jun 18 21:41:44 2022 +0800 feat(risedev): generate risectl config (#3318) * feat(risedev): generate risectl config Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 86ff992 Author: Alex Chi <iskyzh@gmail.com> Date: Sat Jun 18 20:52:51 2022 +0800 feat(ctl): support table scan (#3317) * feat(ctl): support table scan Signed-off-by: Alex Chi <iskyzh@gmail.com> * license header Signed-off-by: Alex Chi <iskyzh@gmail.com> * add docs Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 5ac5637 Author: Yikun Chen <36026213+cykbls01@users.noreply.github.com> Date: Sat Jun 18 08:03:06 2022 -0400 feat: support interval comparison (#3222) 1. fix timestamp substract timestamp. 2. support interval comparison. From pgsql, 1 month equal to 30 days and 1 day equal to 86400000 ms. Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

commit 309ce36 Author: Bowen <36908971+BowenXiao1999@users.noreply.github.com> Date: Tue Jun 21 13:19:44 2022 +0800 refactor(agg): clean up unused fields & refactor (risingwavelabs#3339) * refactor(agg): clean up unused fields * delete file Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit e48dced Author: Shmiwy <wyf000219@126.com> Date: Tue Jun 21 12:55:21 2022 +0800 feat(storage): support compression setting per level (risingwavelabs#3362) Signed-off-by: Shmiwy <wyf000219@126.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 961936f Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 12:42:45 2022 +0800 feat(test): parallelize sqlsmith test (risingwavelabs#3360) * feat(test): parallelize sqlsmith test Signed-off-by: Alex Chi <iskyzh@gmail.com> * more tests Signed-off-by: Alex Chi <iskyzh@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit fa90541 Author: TennyZhuang <zty0826@gmail.com> Date: Tue Jun 21 12:25:33 2022 +0800 chore(github): feature-request template should use the feature label (risingwavelabs#3359) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 02405e1 Author: Bowen <36908971+BowenXiao1999@users.noreply.github.com> Date: Tue Jun 21 12:13:07 2022 +0800 style: add more comments & refactor on pg-wire (risingwavelabs#3358) style: add more comments on pg-wire code Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 1c09432 Author: Li0k <yuli@singularity-data.com> Date: Tue Jun 21 12:00:37 2022 +0800 fix(storage): fix slow unit-test in compactor_test (risingwavelabs#3357) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 19036a6 Author: xxchan <37948597+xxchan@users.noreply.github.com> Date: Tue Jun 21 05:48:08 2022 +0200 fix(binder): do not allow correlated subquery in join tables (risingwavelabs#3352) * fix(binder): do not allow correlated subquery in join tables * clippy Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit c37c9c4 Author: TennyZhuang <zty0826@gmail.com> Date: Tue Jun 21 11:24:40 2022 +0800 refactor: remove unnecessary lazy_static (risingwavelabs#3353) Signed-off-by: TennyZhuang <zty0826@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 68596ab Author: Croxx <mrcroxx@outlook.com> Date: Tue Jun 21 11:11:53 2022 +0800 feat(cache): introduce LruCacheEventListener to subscribe erasure and eviction (risingwavelabs#3334) commit 5627f25 Author: Steven Chua <stevengkc714@protonmail.com> Date: Tue Jun 21 10:54:12 2022 +0800 feat(ctl): Display SstableIdInfo and Block Metadata in sst-dump (risingwavelabs#3338) * feat(ctl): Add sst-dump command to risectl * feat(ctl): Fix risectl compatibility and remove VNode info * feat(ctl): Add checksum and compression algo for each block * feat(ctl): Add SstableIdInfo data to sst-dump * feat(ctl): Fix compilation errors * feat(ctl): Fix compilation errors and bugs commit 16ffd98 Author: Name1e5s <name1e5s@qq.com> Date: Tue Jun 21 10:50:33 2022 +0800 fix(expr): cast int16/int32/int64/float32 to float64 in floor/ceil/round (risingwavelabs#3319) * fix(expr): cast int16/int32/int64/float32 to float64 in floor/ceil/round * fix plan Co-authored-by: TennyZhuang <zty0826@gmail.com> commit 964bb92 Author: TennyZhuang <zty0826@gmail.com> Date: Tue Jun 21 10:23:56 2022 +0800 ci(Mergify): configuration update (risingwavelabs#3355) Signed-off-by: null <zty0826@gmail.com> commit dac904e Author: jon-chuang <9093549+jon-chuang@users.noreply.github.com> Date: Tue Jun 21 09:57:33 2022 +0800 feat(executor): streaming hyperloglog improvements (risingwavelabs#3315) * minor * rename tests * minor * remove option, const eval of param, better comments, succint tests commit cd4f302 Author: TennyZhuang <zty0826@gmail.com> Date: Tue Jun 21 09:34:54 2022 +0800 ci(Mergify): configuration update (risingwavelabs#3252) * ci(Mergify): configuration update Signed-off-by: null <zty0826@gmail.com> * Update .mergify.yml * Update .mergify.yml * Update .mergify.yml Co-authored-by: xxchan <37948597+xxchan@users.noreply.github.com> commit 2aa7e8e Author: Xinpeng Wei <windowsxp@sjtu.edu.cn> Date: Mon Jun 20 22:11:51 2022 +0800 feat(frontend): add InternalStateTable Catalog (risingwavelabs#3139) * use TableMessage for internal table * fix risedev check * update planner test * fix unit test * fix misc check * fix ci * fix issues in PR comments * fix clippy * fix ci * update planner test commit 1e190dd Author: xxchan <37948597+xxchan@users.noreply.github.com> Date: Mon Jun 20 14:16:12 2022 +0200 fix(binder): do not allow correlated input ref in order by (risingwavelabs#3346) commit 263d770 Author: Tao Wu <wutao@singularity-data.com> Date: Mon Jun 20 18:54:25 2022 +0800 fix: build failure caused by OptimzierContext::new (risingwavelabs#3340) commit 9f18401 Author: Tao Wu <wutao@singularity-data.com> Date: Mon Jun 20 17:47:22 2022 +0800 feat: introduce the framework of sqlsmith (risingwavelabs#3305) commit 096a991 Author: Alex Chi <iskyzh@gmail.com> Date: Mon Jun 20 17:40:05 2022 +0800 feat(ctl): add bench command (risingwavelabs#3337) Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 075d596 Author: TennyZhuang <zty0826@gmail.com> Date: Mon Jun 20 17:05:50 2022 +0800 build: bump toolchain to 20220620 (risingwavelabs#3324) * build: bump toolchain to 20220620 Signed-off-by: TennyZhuang <zty0826@gmail.com> * also update docker-compose Signed-off-by: TennyZhuang <zty0826@gmail.com> commit d864f30 Author: Wenzhuo Liu <lwzbill@foxmail.com> Date: Mon Jun 20 16:38:21 2022 +0800 feat: add output_indices to join executors (risingwavelabs#3047) commit 13b9d58 Author: StrikeW <wangsiyuanse@gmail.com> Date: Mon Jun 20 16:29:56 2022 +0800 feat(stream): enable append-only mv plan for kafka source (risingwavelabs#3333) commit ea386d3 Author: Liang <44948473+soundOfDestiny@users.noreply.github.com> Date: Mon Jun 20 15:50:55 2022 +0800 refactor(compaction): deprecate the HashStrategy for OverlapStrategy (risingwavelabs#3331) commit a9fba38 Author: Liang <44948473+soundOfDestiny@users.noreply.github.com> Date: Mon Jun 20 15:34:25 2022 +0800 fix(picker): fetch info from table_id field in sstableinfo (risingwavelabs#3332) commit 5d2bb42 Author: Bohan Zhang <tabvision@bupt.icu> Date: Mon Jun 20 14:55:59 2022 +0800 test(stream): add ci for split change mutation in source (risingwavelabs#3039) * stage Signed-off-by: tabVersion <tabvision@bupt.icu> * stage Signed-off-by: tabVersion <tabvision@bupt.icu> * add test Signed-off-by: tabVersion <tabvision@bupt.icu> * change e2e to datagen Signed-off-by: tabVersion <tabvision@bupt.icu> * stage Signed-off-by: tabVersion <tabvision@bupt.icu> * some bug to fix Signed-off-by: tabVersion <tabvision@bupt.icu> * fix async issue Signed-off-by: tabVersion <tabvision@bupt.icu> * add assert Signed-off-by: tabVersion <tabvision@bupt.icu> commit 04fe6d6 Author: Liang <44948473+soundOfDestiny@users.noreply.github.com> Date: Mon Jun 20 14:47:56 2022 +0800 refactor(vnode bitmap): remove vnode bitmap in sst info (risingwavelabs#3329) commit c6d1288 Author: Li0k <yuli@singularity-data.com> Date: Mon Jun 20 14:29:56 2022 +0800 feat(storage): add manual compaction picker for targeted compaction (risingwavelabs#3288) * feat(storage): add ManualCompactionPicker * feat(storage): distinguish get_compaction_task for manual * feat(storage): meta client support more parameters for manual_compaction * chore(storage): add tracing and some notes * chore(storage): split manual_compaction_picker to independent file * feat(storage): fix target_input check pending and support manual_pick for dynamic_level_selector * fix(storage): internal_table_id include mv_id * fix(storage): fix picker check target_input_ssts pending * fix(storage): fix picker with total_file_size commit d04954f Author: Renjie Liu <liurenjie2008@gmail.com> Date: Mon Jun 20 14:27:01 2022 +0800 fix(ci): Reduce log (risingwavelabs#3330) commit eca9239 Author: Bugen Zhao <i@bugenzhao.com> Date: Mon Jun 20 13:59:56 2022 +0800 refactor(storage): remove `Option` on pk serializer of cell-based table (risingwavelabs#3328) * minor refactor Signed-off-by: Bugen Zhao <i@bugenzhao.com> * remove option of pk serializer Signed-off-by: Bugen Zhao <i@bugenzhao.com> * remove pk serializer in state table Signed-off-by: Bugen Zhao <i@bugenzhao.com> * extract vnode compute Signed-off-by: Bugen Zhao <i@bugenzhao.com> * remove into order types Signed-off-by: Bugen Zhao <i@bugenzhao.com> commit 9169436 Author: Bowen <36908971+BowenXiao1999@users.noreply.github.com> Date: Mon Jun 20 13:44:10 2022 +0800 feat: apply relational refactor for hash agg (max, min) (risingwavelabs#2999) * feat: two closure can not get mut ref of same variable * use Arc::Mutex to wrap the state table * roll back string agg * add StateTable to get_output * finish basic coding (unit test failed) * finish basic coding * fix bug * show case * use empty Row for scan * tweak commit 35bb16a Author: Bugen Zhao <i@bugenzhao.com> Date: Mon Jun 20 13:43:11 2022 +0800 refactor: use packed bitmap struct for vnode bitmap (risingwavelabs#3310) * use bitmap in streaming Signed-off-by: Bugen Zhao <i@bugenzhao.com> * use bitmap in storage Signed-off-by: Bugen Zhao <i@bugenzhao.com> * minor fix Signed-off-by: Bugen Zhao <i@bugenzhao.com> * make bitmap optional Signed-off-by: Bugen Zhao <i@bugenzhao.com> commit be50f93 Author: Liang <44948473+soundOfDestiny@users.noreply.github.com> Date: Mon Jun 20 13:27:37 2022 +0800 feat(compaction): let compactor be unaware of vnode mapping (risingwavelabs#3321) commit 88258a6 Author: lmatz <lmatz823@gmail.com> Date: Sun Jun 19 21:31:01 2022 -0700 doc: no need to manually check in PR from forks (risingwavelabs#3325) commit c590e18 Author: zwang28 <70626450+zwang28@users.noreply.github.com> Date: Mon Jun 20 12:13:50 2022 +0800 refactor(storage): split HummockVersion's levels by compaction group. (risingwavelabs#3206) commit f1c3298 Author: zwang28 <70626450+zwang28@users.noreply.github.com> Date: Mon Jun 20 11:35:18 2022 +0800 feat(meta): register source to compaction group manager (risingwavelabs#3300) commit bf08b54 Author: Zack <52342064+nhzaci@users.noreply.github.com> Date: Mon Jun 20 11:25:39 2022 +0800 feat(frontend): Add sql string into context for debugging (risingwavelabs#3312) * feat(frontend): Add sql string into context for debugging * Remove renaming * Refactor to use str commit f784ba3 Author: zwang28 <70626450+zwang28@users.noreply.github.com> Date: Mon Jun 20 11:22:12 2022 +0800 feat(storage): shared buffer flush L0 by compaction group (risingwavelabs#3200) commit bd48bba Author: Kexiang Wang <kx.wang@hotmail.com> Date: Sun Jun 19 07:48:45 2022 -0400 feat: modify interfaces to support specifying parallelism for each fr… (risingwavelabs#3283) feat: modify interfaces to support specifying parallelism for each fragment commit 8f0e0b2 Author: Steven Chua <stevengkc714@protonmail.com> Date: Sun Jun 19 12:56:51 2022 +0800 feat(ctl): Support basic sst dump in risectl (risingwavelabs#3309) * feat(ctl): Add sst-dump command to risectl * feat(ctl): Fix risectl compatibility and remove VNode info commit daf9222 Author: Alex Chi <iskyzh@gmail.com> Date: Sat Jun 18 21:41:44 2022 +0800 feat(risedev): generate risectl config (risingwavelabs#3318) * feat(risedev): generate risectl config Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 86ff992 Author: Alex Chi <iskyzh@gmail.com> Date: Sat Jun 18 20:52:51 2022 +0800 feat(ctl): support table scan (risingwavelabs#3317) * feat(ctl): support table scan Signed-off-by: Alex Chi <iskyzh@gmail.com> * license header Signed-off-by: Alex Chi <iskyzh@gmail.com> * add docs Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 5ac5637 Author: Yikun Chen <36026213+cykbls01@users.noreply.github.com> Date: Sat Jun 18 08:03:06 2022 -0400 feat: support interval comparison (risingwavelabs#3222) 1. fix timestamp substract timestamp. 2. support interval comparison. From pgsql, 1 month equal to 30 days and 1 day equal to 86400000 ms. commit 722ff53 Author: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com> Date: Fri Jun 17 16:47:22 2022 +0800 feat(meta): inform frontend of mview data distribution (risingwavelabs#3304) * feat(meta): inform frontend of mview data distribution * fix ut * set vnode mapping for materialzied source * move ParallelUnitId into common, move vnode related contants into common/types commit 8571ff4 Author: Renjie Liu <liurenjie2008@gmail.com> Date: Fri Jun 17 16:46:17 2022 +0800 feat(batch): All tests should run in both local and distributed mode (risingwavelabs#3306) * feat(batch): All tests should run in both local and distributed mode Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

WindowsXp-Beta requested review from yezizp2012 and st1page June 10, 2022 12:13

github-actions bot added the type/feature label Jun 10, 2022

skyzh reviewed Jun 13, 2022

View reviewed changes

yezizp2012 reviewed Jun 13, 2022

View reviewed changes

WindowsXp-Beta added 3 commits June 14, 2022 21:20

use TableMessage for internal table

4607610

fix risedev check

6efee84

update planner test

434544c

WindowsXp-Beta force-pushed the WindowsXp/store-state-table-catalog branch from 94d2d74 to 434544c Compare June 14, 2022 14:20

WindowsXp-Beta added 5 commits June 15, 2022 11:05

fix unit test

f1cbf21

fix misc check

55dabd9

Merge remote-tracking branch 'upstream/main' into WindowsXp/store-sta…

5923002

…te-table-catalog

fix ci

eb466ec

Merge remote-tracking branch 'upstream/main' into WindowsXp/store-sta…

ce0089f

…te-table-catalog

st1page reviewed Jun 15, 2022

View reviewed changes

WindowsXp-Beta added 6 commits June 15, 2022 15:35

fix issues in PR comments

2e2911a

Merge remote-tracking branch 'upstream/main' into WindowsXp/store-sta…

303d1e5

…te-table-catalog

fix clippy

d4165e7

Merge remote-tracking branch 'upstream/main' into WindowsXp/store-sta…

a14b31e

…te-table-catalog

fix ci

d67565b

update planner test

e439ccb

WindowsXp-Beta enabled auto-merge (squash) June 15, 2022 08:38

xx01cyx reviewed Jun 17, 2022

View reviewed changes

xx01cyx requested a review from st1page June 17, 2022 06:03

WindowsXp-Beta mentioned this pull request Jun 17, 2022

store the catalog of state table #3053

Closed

xx01cyx requested review from yezizp2012 and skyzh June 17, 2022 06:13

replace clone with as_ref

a72f976

WindowsXp-Beta added 2 commits June 17, 2022 14:25

Merge remote-tracking branch 'upstream/main' into WindowsXp/store-sta…

2a3f220

…te-table-catalog

remove table_id in proto

a7d7b66

st1page reviewed Jun 17, 2022

View reviewed changes

xx01cyx requested a review from st1page June 20, 2022 03:54

st1page approved these changes Jun 20, 2022

View reviewed changes

WindowsXp-Beta added 2 commits June 20, 2022 20:27

Merge remote-tracking branch 'upstream/main' into WindowsXp/store-sta…

2c39ebc

…te-table-catalog

rename to column_mapping

23dc3c9

WindowsXp-Beta merged commit 2aa7e8e into main Jun 20, 2022

WindowsXp-Beta deleted the WindowsXp/store-state-table-catalog branch June 20, 2022 14:11

WindowsXp-Beta mentioned this pull request Jun 21, 2022

feat(meta): store internal table catalog in meta. #3389

Merged

3 tasks

BugenZhao mentioned this pull request Jun 24, 2022

frontend: distribution key of StreamNode::HashAgg should be group key #3456

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(frontend): add InternalStateTable Catalog #3139

feat(frontend): add InternalStateTable Catalog #3139

WindowsXp-Beta commented Jun 10, 2022 •

edited

skyzh left a comment

skyzh Jun 13, 2022

skyzh Jun 13, 2022

skyzh Jun 13, 2022

yezizp2012 left a comment

yezizp2012 Jun 13, 2022

yezizp2012 Jun 13, 2022

yezizp2012 Jun 13, 2022

codecov bot commented Jun 15, 2022 •

edited

st1page Jun 15, 2022

WindowsXp-Beta Jun 15, 2022 •

edited

st1page Jun 15, 2022

xx01cyx Jun 17, 2022

WindowsXp-Beta Jun 17, 2022

xx01cyx Jun 17, 2022

st1page Jun 17, 2022

wcy-fdu commented Jun 20, 2022

WindowsXp-Beta commented Jun 20, 2022

st1page left a comment

	let mut column_index2columb_id = HashMap::new();
	let mut column_index2column_id = HashMap::new();

	.set_fragment_state_table(fragment_id, node.left_table.clone().unwrap().id);
	.set_fragment_state_table(fragment_id, node.left_table.as_ref().unwrap().id);

feat(frontend): add InternalStateTable Catalog #3139

feat(frontend): add InternalStateTable Catalog #3139

Conversation

WindowsXp-Beta commented Jun 10, 2022 • edited

What's changed and what's your intention?

Next Step(will be done in next PR)

Checklist

Refer to a related PR or issue link (optional)

skyzh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yezizp2012 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 15, 2022 • edited

Codecov Report

Choose a reason for hiding this comment

WindowsXp-Beta Jun 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wcy-fdu commented Jun 20, 2022

WindowsXp-Beta commented Jun 20, 2022

st1page left a comment

Choose a reason for hiding this comment

WindowsXp-Beta commented Jun 10, 2022 •

edited

codecov bot commented Jun 15, 2022 •

edited

WindowsXp-Beta Jun 15, 2022 •

edited