Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(meta): store internal table catalog in meta. #3389

Merged
merged 14 commits into from
Jun 24, 2022

Conversation

WindowsXp-Beta
Copy link
Contributor

@WindowsXp-Beta WindowsXp-Beta commented Jun 21, 2022

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

following #3139, this is the meta part, including:

  1. construct the name of internal table.
  2. fill the table id, schema id and database id and vnode mapping of internal tables.
  3. notify frontend.
  4. filter internal tables using the pattern of its name in SHOW MATERIALIZED VIEW.

update:

  1. use an increasing id suffix to avoid duplicate column name.

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

close #3053

@xx01cyx
Copy link
Contributor

xx01cyx commented Jun 21, 2022

We still have to take vnode_mapping into consideration now (for this will not be removed in the foreseeable future 🥺). The vnode_mapping of a internal state table should be set as its hash mapping in HashMappingManager. The hash mapping could be retrieved using TableId of internal state table.

@codecov
Copy link

codecov bot commented Jun 23, 2022

Codecov Report

Merging #3389 (20e0dc1) into main (8cec189) will decrease coverage by 0.07%.
The diff coverage is 28.20%.

❗ Current head 20e0dc1 differs from pull request most recent head 80a9681. Consider uploading reports for the commit 80a9681 to get more accurate results

@@            Coverage Diff             @@
##             main    #3389      +/-   ##
==========================================
- Coverage   73.83%   73.75%   -0.08%     
==========================================
  Files         765      766       +1     
  Lines      105450   105457       +7     
==========================================
- Hits        77856    77783      -73     
- Misses      27594    27674      +80     
Flag Coverage Δ
rust 73.75% <28.20%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/common/src/catalog/mod.rs 60.44% <ø> (ø)
src/frontend/src/catalog/schema_catalog.rs 81.90% <0.00%> (-3.25%) ⬇️
src/meta/src/manager/catalog.rs 19.80% <0.00%> (-0.63%) ⬇️
src/meta/src/rpc/service/ddl_service.rs 0.00% <0.00%> (ø)
src/meta/src/stream/mod.rs 29.03% <0.00%> (-10.10%) ⬇️
src/common/src/catalog/internal_table.rs 33.33% <33.33%> (ø)
src/meta/src/stream/stream_graph.rs 78.85% <65.78%> (-1.00%) ⬇️
src/frontend/src/catalog/table_catalog.rs 97.81% <66.66%> (-0.47%) ⬇️
...rc/frontend/src/optimizer/plan_node/logical_agg.rs 96.27% <80.00%> (-0.22%) ⬇️
src/meta/src/stream/stream_manager.rs 76.05% <100.00%> (-0.06%) ⬇️
... and 14 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@@ -90,9 +92,17 @@ impl SchemaCatalog {

/// Iterate all materialized views, excluding the indexs.
pub fn iter_mv(&self) -> impl Iterator<Item = &TableCatalog> {
lazy_static! {
static ref INTERNAL_TABLE_NAME: Regex = Regex::new(r"__INTERNAL_.*_\d+").unwrap();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about moving internal table naming logic and verify into common crate? Besides, we should also provide a valid_table_name(&str) -> bool function, to validate normal created table.

@@ -530,13 +538,18 @@ impl StreamGraphBuilder {
stream_node: &StreamNode,
actor_id: LocalActorId,
upstream_actor_id: &mut HashMap<u64, OrderedActorLink>,
) -> Result<StreamNode> {
) -> Result<(StreamNode, Vec<Table>)> {
fn generate_intertable_name(mview_name: &String, table_id: u32) -> String {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Copy link
Contributor

@yezizp2012 yezizp2012 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@WindowsXp-Beta WindowsXp-Beta enabled auto-merge (squash) June 24, 2022 02:09
@WindowsXp-Beta WindowsXp-Beta merged commit 68be229 into main Jun 24, 2022
@WindowsXp-Beta WindowsXp-Beta deleted the WindowsXp/meta-internal-state-table branch June 24, 2022 03:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

store the catalog of state table
3 participants