feat(docs): add docs for the relational table layer #3313

wcy-fdu · 2022-06-18T06:01:07Z

What's changed and what's your intention?

Just click docs change to review this PR.
Todo

Draw some figures

Checklist

I have written necessary docs and comments
I have added necessary unit tests and integration tests
All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

close #3299

codecov · 2022-06-18T06:19:51Z

Codecov Report

Merging #3313 (6020368) into main (7c44a15) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #3313   +/-   ##
=======================================
  Coverage   73.71%   73.72%           
=======================================
  Files         760      760           
  Lines      104321   104321           
=======================================
+ Hits        76903    76907    +4     
+ Misses      27418    27414    -4

Flag	Coverage Δ
rust	`73.72% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/storage/src/hummock/local_version_manager.rs	`84.53% <0.00%> (+0.15%)`	⬆️
src/frontend/src/expr/utils.rs	`99.24% <0.00%> (+0.25%)`	⬆️
src/meta/src/barrier/mod.rs	`69.59% <0.00%> (+0.29%)`	⬆️
src/meta/src/hummock/mock_hummock_meta_client.rs	`41.50% <0.00%> (+0.94%)`	⬆️

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

docs/state-store-overview.md

…/risingwave into wcy_update_docs

skyzh

Generally LGTM

docs/state-store-overview.md

docs/storing-state-using-relational-table.md

skyzh · 2022-06-21T14:50:46Z

I think we can also explain the table's properties in this section (storage pk, stream pk, distribution key, order key, etc.), and also the vnode part? I believe relational table will have a big upcoming refactor in the near future. Do you want to include the vnode-related things in this docs?

wcy-fdu · 2022-06-21T14:53:27Z

Is vnode part better put in consistence hash doc? @xx01cyx will write a doc to summarize consistence hash soon.

xx01cyx · 2022-06-21T15:04:21Z

Is vnode part better put in consistence hash doc? @xx01cyx will write a doc to summarize consistence hash soon.

If this doc is to give the detailed encoding, just like

| mv     | table_id \| vnode \| sort key \| pk \| col_id| materialized value |

then maybe there should be a brief description on vnode in this doc. I'm sure we will have a more detailed description on how vnode works in our system in the doc for consistent hash.

…/risingwave into wcy_update_docs

wcy-fdu · 2022-06-22T03:55:25Z

After vnode is encoded in keyspace, will add vonde in this form and add link to corresponding doc.

fuyufjh

LGTM

docs/storing-state-using-relational-table.md

commit aac69d2 Author: William Wen <44139337+wenym1@users.noreply.github.com> Date: Wed Jun 22 17:23:31 2022 +0800 fix(storage): fix ignoring delete record when getting from sst (#3405) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ebfbd0c Author: Alex Chi <iskyzh@gmail.com> Date: Wed Jun 22 17:01:53 2022 +0800 chore(storage): move read lock acquire out of lock (#3400) Signed-off-by: Alex Chi <iskyzh@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit c5b4fe2 Author: lmatz <lmatz823@gmail.com> Date: Wed Jun 22 01:49:23 2022 -0700 fix(expr): trim (#3402) commit 28cf5ff Author: Wallace <bupt2013211450@gmail.com> Date: Wed Jun 22 16:29:03 2022 +0800 feat(compaction): compress data with dynamic level (#3388) * do not compress data in high level Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit 7602285 Author: Lee Zong Yu <65748142+marvenlee2486@users.noreply.github.com> Date: Wed Jun 22 15:58:31 2022 +0800 fix(grafana): Edit update.sh (#3399) * Edit update.sh * Edit format error * Add payload to gitignore * Revert risedev.yml modification * Fix wrong gitignore * Fix wrong gitignore Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4e66ca3 Author: congyi <58715567+wcy-fdu@users.noreply.github.com> Date: Wed Jun 22 13:16:48 2022 +0800 feat(docs): add docs for the relational table layer (#3313) * add doc for relational layer commit 7c44a15 Author: Bugen Zhao <i@bugenzhao.com> Date: Wed Jun 22 12:51:59 2022 +0800 fix: build failure of cell based table with release profile (#3395) commit 93c62a3 Author: Li0k <yuli@singularity-data.com> Date: Wed Jun 22 11:50:18 2022 +0800 chore(frontend): fix handle_with_properties ctx of explain (#3394) commit 233ee5d Author: Li0k <yuli@singularity-data.com> Date: Wed Jun 22 11:32:05 2022 +0800 feat(frontend): catalog add properties for ttl (#3382) * feat(frontend): catlog add properties for materialized_view and materialized_source to support ttl * fix(frontend): fix test_runner * chore(frontend): unify handle_with_properties logic for handler * chore(frontend): explain introduce handle_with_properties to replace WithProperties * chore(sqlparser): remove unused logic of WithProperties commit 78861e3 Author: Bugen Zhao <i@bugenzhao.com> Date: Wed Jun 22 02:19:37 2022 +0800 feat(storage): store all column descs in cell based table (#3344) * add pk indices Signed-off-by: Bugen Zhao <i@bugenzhao.com> * extract mapping Signed-off-by: Bugen Zhao <i@bugenzhao.com> * distinguish parital table Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refactor proto Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix clippy Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix output schema Signed-off-by: Bugen Zhao <i@bugenzhao.com> * store column ids in serializer Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refine docs Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix plan test Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refine docs Signed-off-by: Bugen Zhao <i@bugenzhao.com> * write check Signed-off-by: Bugen Zhao <i@bugenzhao.com> * Apply suggestions from code review Co-authored-by: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com> * trigger ci Signed-off-by: Bugen Zhao <i@bugenzhao.com> Co-authored-by: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit bc74ff4 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 22:59:22 2022 +0800 feat(streaming): use tokio channels and better coop scheduling (#3374) * feat(streaming): use tokio channels and better coop scheduling Signed-off-by: Alex Chi <iskyzh@gmail.com> * more info when deploy Signed-off-by: Alex Chi <iskyzh@gmail.com> * remove manual coop scheduling Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 4b33d95 Author: Huangjw <1223644280@qq.com> Date: Tue Jun 21 18:32:36 2022 +0800 chore(ci): update ci image vesion and remove changelog.json (#3384) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ed1de15 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 18:19:02 2022 +0800 feat(risedev): fix meta args and support healthcheck in docker (#3378) * feat(risedev): support healthcheck in docker Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * pin version Signed-off-by: Alex Chi <iskyzh@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 5a7e115 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 18:05:56 2022 +0800 chore(ci): manual CLA check (#3386) Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 2688491 Author: ZENOTME <43447882+ZENOTME@users.noreply.github.com> Date: Tue Jun 21 18:03:02 2022 +0800 feat(pgwire): support row limit in extended query mode (#3354) * * add row_end flag in PgResponse * add values interface in PgResponse * * add result_cache * add execute interface * fix complile error of row_end * * split process_query_msg into process_query_msg_simple and process_query_response execute flow: simple mode: extended query mode: process_query_msg_simple portal.execute() | (response) (response) | process_query_response --------------------- | process_query_with_results * fix clippy * add process at Execute * add PortalSuspended msg * add row_limit process at process_query_with_results * fix small problem and add some comment Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit dc8631e Author: Tao Wu <wutao@singularity-data.com> Date: Tue Jun 21 17:35:36 2022 +0800 test: sqlsmith supports generating unary func (#3370) commit cb0d7c1 Author: Bowen <36908971+BowenXiao1999@users.noreply.github.com> Date: Tue Jun 21 17:16:10 2022 +0800 refactor: deparallel hash agg apply batch + remove Arc Mutex (#3377) * refactor: deparallel hash agg apply batch * remove Arc Mutex Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…3363) * implement cascade && restrict by add relation map * fmt * implement cascade && restrict by add relation map * fmt * correct pre unit test error * correct misc test error * fix * update proto && fix * change grantor to granted by * implement recursive privilege relations && fix * Squashed commit of the following: commit aac69d2 Author: William Wen <44139337+wenym1@users.noreply.github.com> Date: Wed Jun 22 17:23:31 2022 +0800 fix(storage): fix ignoring delete record when getting from sst (#3405) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ebfbd0c Author: Alex Chi <iskyzh@gmail.com> Date: Wed Jun 22 17:01:53 2022 +0800 chore(storage): move read lock acquire out of lock (#3400) Signed-off-by: Alex Chi <iskyzh@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit c5b4fe2 Author: lmatz <lmatz823@gmail.com> Date: Wed Jun 22 01:49:23 2022 -0700 fix(expr): trim (#3402) commit 28cf5ff Author: Wallace <bupt2013211450@gmail.com> Date: Wed Jun 22 16:29:03 2022 +0800 feat(compaction): compress data with dynamic level (#3388) * do not compress data in high level Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> commit 7602285 Author: Lee Zong Yu <65748142+marvenlee2486@users.noreply.github.com> Date: Wed Jun 22 15:58:31 2022 +0800 fix(grafana): Edit update.sh (#3399) * Edit update.sh * Edit format error * Add payload to gitignore * Revert risedev.yml modification * Fix wrong gitignore * Fix wrong gitignore Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4e66ca3 Author: congyi <58715567+wcy-fdu@users.noreply.github.com> Date: Wed Jun 22 13:16:48 2022 +0800 feat(docs): add docs for the relational table layer (#3313) * add doc for relational layer commit 7c44a15 Author: Bugen Zhao <i@bugenzhao.com> Date: Wed Jun 22 12:51:59 2022 +0800 fix: build failure of cell based table with release profile (#3395) commit 93c62a3 Author: Li0k <yuli@singularity-data.com> Date: Wed Jun 22 11:50:18 2022 +0800 chore(frontend): fix handle_with_properties ctx of explain (#3394) commit 233ee5d Author: Li0k <yuli@singularity-data.com> Date: Wed Jun 22 11:32:05 2022 +0800 feat(frontend): catalog add properties for ttl (#3382) * feat(frontend): catlog add properties for materialized_view and materialized_source to support ttl * fix(frontend): fix test_runner * chore(frontend): unify handle_with_properties logic for handler * chore(frontend): explain introduce handle_with_properties to replace WithProperties * chore(sqlparser): remove unused logic of WithProperties commit 78861e3 Author: Bugen Zhao <i@bugenzhao.com> Date: Wed Jun 22 02:19:37 2022 +0800 feat(storage): store all column descs in cell based table (#3344) * add pk indices Signed-off-by: Bugen Zhao <i@bugenzhao.com> * extract mapping Signed-off-by: Bugen Zhao <i@bugenzhao.com> * distinguish parital table Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refactor proto Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix clippy Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix output schema Signed-off-by: Bugen Zhao <i@bugenzhao.com> * store column ids in serializer Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refine docs Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix plan test Signed-off-by: Bugen Zhao <i@bugenzhao.com> * refine docs Signed-off-by: Bugen Zhao <i@bugenzhao.com> * write check Signed-off-by: Bugen Zhao <i@bugenzhao.com> * Apply suggestions from code review Co-authored-by: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com> * trigger ci Signed-off-by: Bugen Zhao <i@bugenzhao.com> Co-authored-by: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit bc74ff4 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 22:59:22 2022 +0800 feat(streaming): use tokio channels and better coop scheduling (#3374) * feat(streaming): use tokio channels and better coop scheduling Signed-off-by: Alex Chi <iskyzh@gmail.com> * more info when deploy Signed-off-by: Alex Chi <iskyzh@gmail.com> * remove manual coop scheduling Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 4b33d95 Author: Huangjw <1223644280@qq.com> Date: Tue Jun 21 18:32:36 2022 +0800 chore(ci): update ci image vesion and remove changelog.json (#3384) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ed1de15 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 18:19:02 2022 +0800 feat(risedev): fix meta args and support healthcheck in docker (#3378) * feat(risedev): support healthcheck in docker Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * fix Signed-off-by: Alex Chi <iskyzh@gmail.com> * pin version Signed-off-by: Alex Chi <iskyzh@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 5a7e115 Author: Alex Chi <iskyzh@gmail.com> Date: Tue Jun 21 18:05:56 2022 +0800 chore(ci): manual CLA check (#3386) Signed-off-by: Alex Chi <iskyzh@gmail.com> commit 2688491 Author: ZENOTME <43447882+ZENOTME@users.noreply.github.com> Date: Tue Jun 21 18:03:02 2022 +0800 feat(pgwire): support row limit in extended query mode (#3354) * * add row_end flag in PgResponse * add values interface in PgResponse * * add result_cache * add execute interface * fix complile error of row_end * * split process_query_msg into process_query_msg_simple and process_query_response execute flow: simple mode: extended query mode: process_query_msg_simple portal.execute() | (response) (response) | process_query_response --------------------- | process_query_with_results * fix clippy * add process at Execute * add PortalSuspended msg * add row_limit process at process_query_with_results * fix small problem and add some comment Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit dc8631e Author: Tao Wu <wutao@singularity-data.com> Date: Tue Jun 21 17:35:36 2022 +0800 test: sqlsmith supports generating unary func (#3370) commit cb0d7c1 Author: Bowen <36908971+BowenXiao1999@users.noreply.github.com> Date: Tue Jun 21 17:16:10 2022 +0800 refactor: deparallel hash agg apply batch + remove Arc Mutex (#3377) * refactor: deparallel hash agg apply batch * remove Arc Mutex Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * add unit test * fmt * correct pre test * some fix Co-authored-by: August <pin@singularity-data.com>

wcy-fdu and others added 16 commits June 16, 2022 16:33

add doc

2e756a9

Added Untitled Diagram.drawio.svg

536ede2

relational-table.svg

2296b0b

relational-table.svg

7fb3e3b

add doc

16f7eb6

add doc

2a75236

update

73b2edd

update

85ba604

更新relational-table-01.svg

e648b6a

update docs

4c72e3e

update fig

35d374c

update fig

d10b7a5

update doc

41ac4c2

update doc

f7e9a52

update doc

6bf077c

update doc

d827cc2

wcy-fdu marked this pull request as draft June 18, 2022 06:01

github-actions bot added the type/feature label Jun 18, 2022

wcy-fdu and others added 3 commits June 18, 2022 14:02

remove .svg

adb4659

update

908a311

Merge branch 'main' into wcy_update_docs

7ccc832

wcy-fdu marked this pull request as ready for review June 18, 2022 06:18

wcy-fdu requested review from hzxa21 and Sunt-ing June 20, 2022 02:40

wcy-fdu added 2 commits June 20, 2022 10:42

update

bee4650

update

4c32de2

wcy-fdu requested a review from skyzh June 20, 2022 02:44

update

333638b

Sunt-ing reviewed Jun 20, 2022

View reviewed changes

docs/state-store-overview.md Outdated Show resolved Hide resolved

wcy-fdu added 7 commits June 21, 2022 13:51

update

6b170f8

Merge branch 'wcy_update_docs' of https://github.com/singularity-data…

4893156

…/risingwave into wcy_update_docs

update

1b7ce4e

update

be9ed68

update fig

52f40c4

add more example

006102b

add more example

2564f25

wcy-fdu requested review from skyzh, fuyufjh and st1page June 21, 2022 07:27

risingwavelabs deleted a comment from Sunt-ing Jun 21, 2022

skyzh reviewed Jun 21, 2022

View reviewed changes

wcy-fdu and others added 4 commits June 21, 2022 16:32

fix

7f66898

fix

dde36ad

x

95c36a0

Merge branch 'main' into wcy_update_docs

c83a06c

wcy-fdu added 2 commits June 22, 2022 11:49

add todo

f168ef9

Merge branch 'wcy_update_docs' of https://github.com/singularity-data…

67f8075

…/risingwave into wcy_update_docs

fuyufjh approved these changes Jun 22, 2022

View reviewed changes

docs/storing-state-using-relational-table.md Outdated Show resolved Hide resolved

wcy-fdu and others added 2 commits June 22, 2022 12:42

fix

a1e2a3d

Merge branch 'main' into wcy_update_docs

6020368

skyzh approved these changes Jun 22, 2022

View reviewed changes

wcy-fdu merged commit 4e66ca3 into main Jun 22, 2022

wcy-fdu deleted the wcy_update_docs branch June 22, 2022 05:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(docs): add docs for the relational table layer #3313

feat(docs): add docs for the relational table layer #3313

wcy-fdu commented Jun 18, 2022 •

edited

codecov bot commented Jun 18, 2022 •

edited

skyzh left a comment

skyzh commented Jun 21, 2022 •

edited

wcy-fdu commented Jun 21, 2022 •

edited

xx01cyx commented Jun 21, 2022

wcy-fdu commented Jun 22, 2022

fuyufjh left a comment

feat(docs): add docs for the relational table layer #3313

feat(docs): add docs for the relational table layer #3313

Conversation

wcy-fdu commented Jun 18, 2022 • edited

What's changed and what's your intention?

Checklist

Refer to a related PR or issue link (optional)

codecov bot commented Jun 18, 2022 • edited

Codecov Report

skyzh left a comment

Choose a reason for hiding this comment

skyzh commented Jun 21, 2022 • edited

wcy-fdu commented Jun 21, 2022 • edited

xx01cyx commented Jun 21, 2022

wcy-fdu commented Jun 22, 2022

fuyufjh left a comment

Choose a reason for hiding this comment

wcy-fdu commented Jun 18, 2022 •

edited

codecov bot commented Jun 18, 2022 •

edited

skyzh commented Jun 21, 2022 •

edited

wcy-fdu commented Jun 21, 2022 •

edited