Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(rw_catalog): make rw_worker_nodes list all nodes and resource #13487

Merged
merged 1 commit into from
Nov 20, 2023

Conversation

zwang28
Copy link
Contributor

@zwang28 zwang28 commented Nov 17, 2023

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This PR extends rw_catalog.rw_worker_nodes by

  • including all types of nodes, i.e. meta node/compactor node/compute node/frontend node. Before it only includes compute nodes.
  • adding more columns:
    • rw_version/total memory bytes/total cpu cores, which is reported by worker node when it starts.
    • started_at, which is populated by meta node when a worker node starts.
    • These info are not persisted in meta store. They'll be updated whenever worker node restarts.
dev=> select * from rw_worker_nodes;
 id |   host    | port |           type           |  state  | parallelism | is_streaming | is_serving | is_unschedulable | rw_version  | total_memory_bytes | total_cpu_cores |        started_at         
----+-----------+------+--------------------------+---------+-------------+--------------+------------+------------------+-------------+--------------------+-----------------+---------------------------
  0 | 127.0.0.1 | 5690 | WORKER_TYPE_META         | RUNNING |           0 |              |            |                  | 1.3.0-alpha |        34359738368 |              10 | 2023-11-17 04:05:28+00:00
  1 | 127.0.0.1 | 5688 | WORKER_TYPE_COMPUTE_NODE | RUNNING |           4 | t            | t          | f                | 1.3.0-alpha |        34359738368 |              10 | 2023-11-17 04:05:32+00:00
  2 | 127.0.0.1 | 4566 | WORKER_TYPE_FRONTEND     | RUNNING |           0 |              |            |                  | 1.3.0-alpha |        34359738368 |              10 | 2023-11-17 04:06:27+00:00
(3 rows)

part of #12826

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added test labels as necessary. See details.
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

System catalog

  • Add columns rw_version, total_memory_bytes, total_cpu_cores and started_at in system table rw_worker_nodes. Also include nodes of all kinds in it, e.g. meta node/compute node/frontend node/compactor node.

@zwang28 zwang28 force-pushed the wangzheng/more_worker_node_info branch from c55e0eb to 6682a0d Compare November 17, 2023 04:25
@@ -304,7 +305,7 @@ message UpdateWorkerNodeSchedulabilityResponse {
}

message ListAllNodesRequest {
common.WorkerType worker_type = 1;
optional common.WorkerType worker_type = 1;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is warned as a breaking change. But this RPC is only used by risectl and this PR, so it's fine.

Copy link

codecov bot commented Nov 17, 2023

Codecov Report

Attention: 70 lines in your changes are missing coverage. Please review.

Comparison is base (094a071) 67.62% compared to head (6682a0d) 67.67%.
Report is 29 commits behind head on main.

Files Patch % Lines
...talog/system_catalog/rw_catalog/rw_worker_nodes.rs 0.00% 17 Missing ⚠️
src/meta/src/controller/cluster.rs 16.66% 10 Missing ⚠️
src/rpc_client/src/meta_client.rs 0.00% 10 Missing ⚠️
src/meta/src/manager/cluster.rs 91.42% 6 Missing ⚠️
src/meta/src/dashboard/mod.rs 0.00% 5 Missing ⚠️
src/ctl/src/cmd_impl/await_tree.rs 0.00% 4 Missing ⚠️
src/frontend/src/meta_client.rs 0.00% 3 Missing ⚠️
src/frontend/src/test_utils.rs 0.00% 3 Missing ⚠️
src/meta/service/src/cluster_service.rs 0.00% 3 Missing ⚠️
src/meta/src/hummock/manager/gc.rs 25.00% 3 Missing ⚠️
... and 5 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #13487      +/-   ##
==========================================
+ Coverage   67.62%   67.67%   +0.05%     
==========================================
  Files        1527     1527              
  Lines      260464   260542      +78     
==========================================
+ Hits       176132   176319     +187     
+ Misses      84332    84223     -109     
Flag Coverage Δ
rust 67.67% <56.25%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@yezizp2012 yezizp2012 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Please make it user-facing and add some brief description.

@zwang28 zwang28 added the user-facing-changes Contains changes that are visible to users label Nov 20, 2023
@zwang28 zwang28 added this pull request to the merge queue Nov 20, 2023
Merged via the queue into main with commit 0c35dc7 Nov 20, 2023
33 of 35 checks passed
@zwang28 zwang28 deleted the wangzheng/more_worker_node_info branch November 20, 2023 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/refactor user-facing-changes Contains changes that are visible to users
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants