feat(frontend): catalog add properties for ttl #3382

Li0k · 2022-06-21T07:30:46Z

What's changed and what's your intention?

catalog add properties for

Please explain IN DETAIL what the changes are in this PR and why they are needed:

add properties in catalog.proto for persisting
handler fit properties
fix some test with properties

Checklist

I have written necessary docs and comments
I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

BowenXiao1999

Rest LG

BowenXiao1999 · 2022-06-21T07:47:53Z

src/frontend/src/handler/create_mv.rs


    Ok((plan, table))
 }

+fn handle_create_mv_with_properties(options: Vec<SqlOption>) -> Result<HashMap<String, String>> {


Actually this function can not only applys to MV?

BowenXiao1999 · 2022-06-21T07:49:16Z

proto/catalog.proto

@@ -60,6 +60,7 @@ message Table {
  bool appendonly = 14;
  string owner = 15;
  common.ParallelUnitMapping mapping = 16;
+  map<string, string> properties = 17;


This is funny. I thought We already have this....

yes, it exist in Source , but not Table (

BowenXiao1999 · 2022-06-21T07:53:55Z

src/frontend/src/handler/create_mv.rs

+            context.into(),
+            query,
+            name,
+            handle_create_mv_with_properties(with_options.0)?,


Why not simply done this inside gen_create_mv_plan so we can avoid change the function signature

BowenXiao1999 · 2022-06-21T07:56:50Z

Do we previously use with_option in mv/table?

xiangjinwu

Now there are 4 ways to convert a Vec<SqlOption> to HashMap<String, String> 😂

handle_source_with_properties
handle_create_mv_with_properties
handle_create_table_with_properties
From<WithProperties> for HashMap<String, String>

codecov · 2022-06-21T09:52:10Z

Codecov Report

Merging #3382 (94f1499) into main (78861e3) will increase coverage by 0.07%.
The diff coverage is 77.27%.

@@            Coverage Diff             @@
##             main    #3382      +/-   ##
==========================================
+ Coverage   73.64%   73.71%   +0.07%     
==========================================
  Files         760      760              
  Lines      104285   104321      +36     
==========================================
+ Hits        76797    76904     +107     
+ Misses      27488    27417      -71

Flag	Coverage Δ
rust	`73.71% <77.27%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/frontend/src/handler/explain.rs	`0.00% <0.00%> (ø)`
src/sqlparser/src/ast/statement.rs	`82.20% <ø> (-0.52%)`	⬇️
src/frontend/test_runner/src/lib.rs	`74.23% <75.00%> (-0.08%)`	⬇️
src/frontend/src/handler/util.rs	`87.23% <76.47%> (-1.48%)`	⬇️
src/frontend/src/catalog/table_catalog.rs	`98.28% <100.00%> (+0.02%)`	⬆️
src/frontend/src/handler/create_mv.rs	`98.65% <100.00%> (+0.08%)`	⬆️
src/frontend/src/handler/create_source.rs	`99.21% <100.00%> (+2.15%)`	⬆️
src/frontend/src/handler/create_table.rs	`96.15% <100.00%> (-0.52%)`	⬇️
src/frontend/src/handler/mod.rs	`66.66% <100.00%> (+0.36%)`	⬆️
...rc/frontend/src/optimizer/plan_node/logical_agg.rs	`96.48% <100.00%> (+<0.01%)`	⬆️
... and 10 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

BowenXiao1999 · 2022-06-21T10:01:39Z

Now there are 4 ways to convert a Vec to HashMap<String, String> 😂
handle_source_with_properties
handle_create_mv_with_properties
handle_create_table_with_properties
From for HashMap<String, String>

Indeed, I also want to reuse them.

Li0k · 2022-06-21T11:02:09Z

Now there are 4 ways to convert a Vec to HashMap<String, String> 😂
handle_source_with_properties
handle_create_mv_with_properties
handle_create_table_with_properties
From for HashMap<String, String>

Indeed, I also want to reuse them.

i unify above 4 ways to handle_with_properties.

…ialized_source to support ttl

…WithProperties

commit aac69d2 Author: William Wen <44139337+wenym1@users.noreply.github.com> Date: Wed Jun 22 17:23:31 2022 +0800 fix(storage): fix ignoring delete record when getting from sst (#3405) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ebfbd0c Author: Alex Chi <iskyzh@gmail.com> Date: Wed Jun 22 17:01:53 2022 +0800 chore(storage): move read lock acquire out of lock (#3400) Signed-off-by: Alex Chi <iskyzh@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit c5b4fe2 Author: lmatz <lmatz823@gmail.com> Date: Wed Jun 22 01:49:23 2022 -0700 fix(expr): trim (#3402) commit 28cf5ff Author: Wallace <bupt2013211450@gmail.com> Date: Wed Jun 22 16:29:03 2022 +0800 feat(compaction): compress data with dynamic level (#3388) * do not compress data in high level Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit 7602285 Author: Lee Zong Yu <65748142+marvenlee2486@users.noreply.github.com> Date: Wed Jun 22 15:58:31 2022 +0800 fix(grafana): Edit update.sh (#3399) * Edit update.sh * Edit format error * Add payload to gitignore * Revert risedev.yml modification * Fix wrong gitignore * Fix wrong gitignore Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4e66ca3 Author: congyi <58715567+wcy-fdu@users.noreply.github.com> Date: Wed Jun 22 13:16:48 2022 +0800 feat(docs): add docs for the relational table layer (#3313) * add doc for relational layer commit 7c44a15 Author: Bugen Zhao <i@bugenzhao.com> Date: Wed Jun 22 12:51:59 2022 +0800 fix: build failure of cell based table with release profile (#3395) commit 93c62a3 Author: Li0k <yuli@singularity-data.com> Date: Wed Jun 22 11:50:18 2022 +0800 chore(frontend): fix handle_with_properties ctx of explain (#3394) commit 233ee5d Author: Li0k <yuli@singularity-data.com> Date: Wed Jun 22 11:32:05 2022 +0800 feat(frontend): catalog add properties for ttl (#3382) * feat(frontend): catlog add properties for materialized_view and materialized_source to support ttl * fix(frontend): fix test_runner * chore(frontend): unify handle_with_properties logic for handler * chore(frontend): explain introduce handle_with_properties to replace WithProperties * chore(sqlparser): remove unused logic of WithProperties commit 78861e3 Author: Bugen Zhao <i@bugenzhao.com> Date: Wed Jun 22 02:19:37 2022 +0800 feat(storage): store all column descs in cell based table (#3344) * add pk indices Signed-off-by: Bugen Zhao <i@bugenzhao.com> * extract mapping Signed-off-by: Bugen Zhao <i@bugenzhao.com> * distinguish parital table Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refactor proto Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix clippy Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix output schema Signed-off-by: Bugen Zhao <i@bugenzhao.com> * store column ids in serializer Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refine docs Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix plan test Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refine docs Signed-off-by: Bugen Zhao <i@bugenzhao.com> * write check Signed-off-by: Bugen Zhao <i@bugenzhao.com> * Apply suggestions from code review Co-authored-by: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com> * trigger ci Signed-off-by: Bugen Zhao <i@bugenzhao.com> Co-authored-by: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit bc74ff4 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 22:59:22 2022 +0800 feat(streaming): use tokio channels and better coop scheduling (#3374) * feat(streaming): use tokio channels and better coop scheduling Signed-off-by: Alex Chi <iskyzh@gmail.com> * more info when deploy Signed-off-by: Alex Chi <iskyzh@gmail.com> * remove manual coop scheduling Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 4b33d95 Author: Huangjw <1223644280@qq.com> Date: Tue Jun 21 18:32:36 2022 +0800 chore(ci): update ci image vesion and remove changelog.json (#3384) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ed1de15 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 18:19:02 2022 +0800 feat(risedev): fix meta args and support healthcheck in docker (#3378) * feat(risedev): support healthcheck in docker Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * pin version Signed-off-by: Alex Chi <iskyzh@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 5a7e115 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 18:05:56 2022 +0800 chore(ci): manual CLA check (#3386) Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 2688491 Author: ZENOTME <43447882+ZENOTME@users.noreply.github.com> Date: Tue Jun 21 18:03:02 2022 +0800 feat(pgwire): support row limit in extended query mode (#3354) * * add row_end flag in PgResponse * add values interface in PgResponse * * add result_cache * add execute interface * fix complile error of row_end * * split process_query_msg into process_query_msg_simple and process_query_response execute flow: simple mode: extended query mode: process_query_msg_simple portal.execute() | (response) (response) | process_query_response --------------------- | process_query_with_results * fix clippy * add process at Execute * add PortalSuspended msg * add row_limit process at process_query_with_results * fix small problem and add some comment Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit dc8631e Author: Tao Wu <wutao@singularity-data.com> Date: Tue Jun 21 17:35:36 2022 +0800 test: sqlsmith supports generating unary func (#3370) commit cb0d7c1 Author: Bowen <36908971+BowenXiao1999@users.noreply.github.com> Date: Tue Jun 21 17:16:10 2022 +0800 refactor: deparallel hash agg apply batch + remove Arc Mutex (#3377) * refactor: deparallel hash agg apply batch * remove Arc Mutex Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…3363) * implement cascade && restrict by add relation map * fmt * implement cascade && restrict by add relation map * fmt * correct pre unit test error * correct misc test error * fix * update proto && fix * change grantor to granted by * implement recursive privilege relations && fix * Squashed commit of the following: commit aac69d2 Author: William Wen <44139337+wenym1@users.noreply.github.com> Date: Wed Jun 22 17:23:31 2022 +0800 fix(storage): fix ignoring delete record when getting from sst (#3405) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ebfbd0c Author: Alex Chi <iskyzh@gmail.com> Date: Wed Jun 22 17:01:53 2022 +0800 chore(storage): move read lock acquire out of lock (#3400) Signed-off-by: Alex Chi <iskyzh@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit c5b4fe2 Author: lmatz <lmatz823@gmail.com> Date: Wed Jun 22 01:49:23 2022 -0700 fix(expr): trim (#3402) commit 28cf5ff Author: Wallace <bupt2013211450@gmail.com> Date: Wed Jun 22 16:29:03 2022 +0800 feat(compaction): compress data with dynamic level (#3388) * do not compress data in high level Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit 7602285 Author: Lee Zong Yu <65748142+marvenlee2486@users.noreply.github.com> Date: Wed Jun 22 15:58:31 2022 +0800 fix(grafana): Edit update.sh (#3399) * Edit update.sh * Edit format error * Add payload to gitignore * Revert risedev.yml modification * Fix wrong gitignore * Fix wrong gitignore Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4e66ca3 Author: congyi <58715567+wcy-fdu@users.noreply.github.com> Date: Wed Jun 22 13:16:48 2022 +0800 feat(docs): add docs for the relational table layer (#3313) * add doc for relational layer commit 7c44a15 Author: Bugen Zhao <i@bugenzhao.com> Date: Wed Jun 22 12:51:59 2022 +0800 fix: build failure of cell based table with release profile (#3395) commit 93c62a3 Author: Li0k <yuli@singularity-data.com> Date: Wed Jun 22 11:50:18 2022 +0800 chore(frontend): fix handle_with_properties ctx of explain (#3394) commit 233ee5d Author: Li0k <yuli@singularity-data.com> Date: Wed Jun 22 11:32:05 2022 +0800 feat(frontend): catalog add properties for ttl (#3382) * feat(frontend): catlog add properties for materialized_view and materialized_source to support ttl * fix(frontend): fix test_runner * chore(frontend): unify handle_with_properties logic for handler * chore(frontend): explain introduce handle_with_properties to replace WithProperties * chore(sqlparser): remove unused logic of WithProperties commit 78861e3 Author: Bugen Zhao <i@bugenzhao.com> Date: Wed Jun 22 02:19:37 2022 +0800 feat(storage): store all column descs in cell based table (#3344) * add pk indices Signed-off-by: Bugen Zhao <i@bugenzhao.com> * extract mapping Signed-off-by: Bugen Zhao <i@bugenzhao.com> * distinguish parital table Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refactor proto Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix clippy Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix output schema Signed-off-by: Bugen Zhao <i@bugenzhao.com> * store column ids in serializer Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refine docs Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix plan test Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refine docs Signed-off-by: Bugen Zhao <i@bugenzhao.com> * write check Signed-off-by: Bugen Zhao <i@bugenzhao.com> * Apply suggestions from code review Co-authored-by: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com> * trigger ci Signed-off-by: Bugen Zhao <i@bugenzhao.com> Co-authored-by: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit bc74ff4 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 22:59:22 2022 +0800 feat(streaming): use tokio channels and better coop scheduling (#3374) * feat(streaming): use tokio channels and better coop scheduling Signed-off-by: Alex Chi <iskyzh@gmail.com> * more info when deploy Signed-off-by: Alex Chi <iskyzh@gmail.com> * remove manual coop scheduling Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 4b33d95 Author: Huangjw <1223644280@qq.com> Date: Tue Jun 21 18:32:36 2022 +0800 chore(ci): update ci image vesion and remove changelog.json (#3384) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ed1de15 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 18:19:02 2022 +0800 feat(risedev): fix meta args and support healthcheck in docker (#3378) * feat(risedev): support healthcheck in docker Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * pin version Signed-off-by: Alex Chi <iskyzh@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 5a7e115 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 18:05:56 2022 +0800 chore(ci): manual CLA check (#3386) Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 2688491 Author: ZENOTME <43447882+ZENOTME@users.noreply.github.com> Date: Tue Jun 21 18:03:02 2022 +0800 feat(pgwire): support row limit in extended query mode (#3354) * * add row_end flag in PgResponse * add values interface in PgResponse * * add result_cache * add execute interface * fix complile error of row_end * * split process_query_msg into process_query_msg_simple and process_query_response execute flow: simple mode: extended query mode: process_query_msg_simple portal.execute() | (response) (response) | process_query_response --------------------- | process_query_with_results * fix clippy * add process at Execute * add PortalSuspended msg * add row_limit process at process_query_with_results * fix small problem and add some comment Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit dc8631e Author: Tao Wu <wutao@singularity-data.com> Date: Tue Jun 21 17:35:36 2022 +0800 test: sqlsmith supports generating unary func (#3370) commit cb0d7c1 Author: Bowen <36908971+BowenXiao1999@users.noreply.github.com> Date: Tue Jun 21 17:16:10 2022 +0800 refactor: deparallel hash agg apply batch + remove Arc Mutex (#3377) * refactor: deparallel hash agg apply batch * remove Arc Mutex Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * add unit test * fmt * correct pre test * some fix Co-authored-by: August <pin@singularity-data.com>

Li0k requested a review from xiangjinwu June 21, 2022 07:30

github-actions bot added the Invalid PR Title label Jun 21, 2022

Li0k force-pushed the li0k/feat_frontend_ttl_option branch 2 times, most recently from a396858 to 7e8e824 Compare June 21, 2022 07:38

Li0k requested a review from BowenXiao1999 June 21, 2022 07:44

BowenXiao1999 reviewed Jun 21, 2022

View reviewed changes

BowenXiao1999 approved these changes Jun 21, 2022

View reviewed changes

Li0k changed the title ~~[WIP] feat(frontend): catlog add properties for ttl~~ feat(frontend): catlog add properties for ttl Jun 21, 2022

Li0k changed the title ~~feat(frontend): catlog add properties for ttl~~ feat(frontend): catlog add properties for ttl Jun 21, 2022

github-actions bot added type/feature and removed Invalid PR Title labels Jun 21, 2022

Li0k force-pushed the li0k/feat_frontend_ttl_option branch from 7e8e824 to 904f0d9 Compare June 21, 2022 08:42

xiangjinwu reviewed Jun 21, 2022

View reviewed changes

Li0k force-pushed the li0k/feat_frontend_ttl_option branch 2 times, most recently from 9b6bfa0 to 998428d Compare June 21, 2022 09:40

Li0k added the mergify/can-merge Indicates that the PR can be added to the merge queue label Jun 21, 2022

Li0k removed the mergify/can-merge Indicates that the PR can be added to the merge queue label Jun 21, 2022

Li0k force-pushed the li0k/feat_frontend_ttl_option branch from 998428d to 8aa6770 Compare June 21, 2022 11:00

Li0k added 3 commits June 22, 2022 10:49

feat(frontend): catlog add properties for materialized_view and mater…

f9bfda5

…ialized_source to support ttl

fix(frontend): fix test_runner

978336d

chore(frontend): unify handle_with_properties logic for handler

381534e

Li0k force-pushed the li0k/feat_frontend_ttl_option branch from 8aa6770 to 381534e Compare June 22, 2022 02:49

chore(frontend): explain introduce handle_with_properties to replace …

f24a205

…WithProperties

Li0k changed the title ~~feat(frontend): catlog add properties for ttl~~ feat(frontend): catalog add properties for ttl Jun 22, 2022

chore(sqlparser): remove unused logic of WithProperties

94f1499

xiangjinwu approved these changes Jun 22, 2022

View reviewed changes

Li0k added the mergify/can-merge Indicates that the PR can be added to the merge queue label Jun 22, 2022

mergify bot merged commit 233ee5d into main Jun 22, 2022

mergify bot deleted the li0k/feat_frontend_ttl_option branch June 22, 2022 03:32

Li0k mentioned this pull request Jun 22, 2022

chore(frontend): fix handle_with_properties ctx of explain #3394

Merged

Li0k mentioned this pull request Jul 1, 2022

feature: support epoch base TTL on hummock #3298

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(frontend): catalog add properties for ttl #3382

feat(frontend): catalog add properties for ttl #3382

Li0k commented Jun 21, 2022 •

edited

BowenXiao1999 left a comment

BowenXiao1999 Jun 21, 2022

BowenXiao1999 Jun 21, 2022

Li0k Jun 21, 2022

BowenXiao1999 Jun 21, 2022

BowenXiao1999 commented Jun 21, 2022

xiangjinwu left a comment

codecov bot commented Jun 21, 2022 •

edited

BowenXiao1999 commented Jun 21, 2022 •

edited

Li0k commented Jun 21, 2022

feat(frontend): catalog add properties for ttl #3382

feat(frontend): catalog add properties for ttl #3382

Conversation

Li0k commented Jun 21, 2022 • edited

What's changed and what's your intention?

Checklist

Refer to a related PR or issue link (optional)

BowenXiao1999 left a comment

Choose a reason for hiding this comment

BowenXiao1999 Jun 21, 2022

Choose a reason for hiding this comment

BowenXiao1999 Jun 21, 2022

Choose a reason for hiding this comment

Li0k Jun 21, 2022

Choose a reason for hiding this comment

BowenXiao1999 Jun 21, 2022

Choose a reason for hiding this comment

BowenXiao1999 commented Jun 21, 2022

xiangjinwu left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 21, 2022 • edited

Codecov Report

BowenXiao1999 commented Jun 21, 2022 • edited

Li0k commented Jun 21, 2022

Li0k commented Jun 21, 2022 •

edited

codecov bot commented Jun 21, 2022 •

edited

BowenXiao1999 commented Jun 21, 2022 •

edited