Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: introduce system tables about table fragment info #10712

Merged
merged 3 commits into from
Jul 4, 2023

Conversation

yezizp2012
Copy link
Contributor

@yezizp2012 yezizp2012 commented Jul 3, 2023

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Resolve #8803 , introduced system tables rw_table_fragments, rw_fragments and rw_actors to expose status, type and distribution information about streaming jobs.

rw_table_fragments

Field Type Description
table_id Int32 Unique identifier of the table fragment
status Varchar Status of the table fragment

rw_fragments

Field Type Description
fragment_id Int32 Unique identifier of the fragment
table_id Int32 Identifier of the associated table
distribution_type Varchar Type of distribution for the fragment
state_table_ids List (Int32) List of state table identifiers associated with the fragment
upstream_fragment_ids List (Int32) List of upstream fragment identifiers

rw_actors

Field Type Description
actor_id Int32 Unique identifier of the actor
fragment_id Int32 Identifier of the associated fragment
parallel_unit_id Int32 Identifier of the parallel unit the actor belongs to
status Varchar Status of the actor

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

  • My PR contains user-facing changes.
Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

  • SQL commands, functions, and operators

Release note

Introduce system tables rw_table_fragments, rw_fragments and rw_actors.

@github-actions github-actions bot added type/feature user-facing-changes Contains changes that are visible to users labels Jul 3, 2023
@codecov
Copy link

codecov bot commented Jul 3, 2023

Codecov Report

Merging #10712 (6c62de0) into main (1ac1726) will decrease coverage by 0.06%.
The diff coverage is 7.93%.

@@            Coverage Diff             @@
##             main   #10712      +/-   ##
==========================================
- Coverage   70.25%   70.20%   -0.06%     
==========================================
  Files        1291     1292       +1     
  Lines      220677   220852     +175     
==========================================
+ Hits       155038   155049      +11     
- Misses      65639    65803     +164     
Flag Coverage Δ
rust 70.20% <7.93%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ntend/src/catalog/system_catalog/pg_catalog/mod.rs 0.00% <ø> (ø)
...ntend/src/catalog/system_catalog/rw_catalog/mod.rs 0.00% <0.00%> (ø)
src/frontend/src/meta_client.rs 0.00% <0.00%> (ø)
src/frontend/src/test_utils.rs 77.77% <0.00%> (-1.13%) ⬇️
src/meta/src/model/stream.rs 69.59% <0.00%> (-2.49%) ⬇️
src/meta/src/rpc/service/stream_service.rs 0.00% <0.00%> (ø)
src/rpc_client/src/meta_client.rs 4.22% <0.00%> (-0.08%) ⬇️
src/frontend/src/catalog/system_catalog/mod.rs 46.66% <100.00%> (+1.08%) ⬆️
.../catalog/system_catalog/rw_catalog/rw_fragments.rs 100.00% <100.00%> (ø)

... and 6 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@yezizp2012
Copy link
Contributor Author

Examples in a local run:

dev=> select * from rw_actors;
 actor_id | fragment_id | parallel_unit_id | status
----------+-------------+------------------+---------
     2001 |        2001 |            11000 | RUNNING
     2002 |        2001 |            11001 | RUNNING
     2003 |        2001 |            11002 | RUNNING
     2004 |        2001 |            11003 | RUNNING
     2005 |        2002 |            11000 | RUNNING
     2006 |        2002 |            11001 | RUNNING
     2007 |        2002 |            11002 | RUNNING
     2008 |        2002 |            11003 | RUNNING
(8 rows)

dev=> select * from rw_fragments;
 fragment_id | table_id | distribution_type | state_table_ids | upstream_fragment_ids
-------------+----------+-------------------+-----------------+-----------------------
        2001 |     3001 | HASH              | {3001}          | {2002}
        2002 |     3001 | HASH              | {}              | {}
(2 rows)

dev=> select * from rw_table_fragments;
 table_id | status
----------+---------
     3001 | CREATED
(1 row)

@@ -595,4 +601,82 @@ impl SysCatalogReaderImpl {
})
.collect_vec())
}

/// FIXME: we need to introduce revision snapshot read on meta to avoid any inconsistency when
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened an issue for it: #10732 .

Copy link
Member

@BugenZhao BugenZhao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@yezizp2012 yezizp2012 added this pull request to the merge queue Jul 4, 2023
Merged via the queue into main with commit 048e71d Jul 4, 2023
36 checks passed
@yezizp2012 yezizp2012 deleted the zp/rw-fragments branch July 4, 2023 11:27
@emile-00 emile-00 added the 📖✓ Covered or will be covered in the user docs. label Jul 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/feature user-facing-changes Contains changes that are visible to users 📖✓ Covered or will be covered in the user docs.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

streaming: expose stream graph with system tables
3 participants