Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

not cache the output of ss -tln command when checking wether components are started/stopped #2298

Merged
merged 2 commits into from
Nov 1, 2023

Conversation

KanShiori
Copy link
Collaborator

@KanShiori KanShiori commented Nov 1, 2023

What problem does this PR solve?

Close #2267

What is changed and how it works?

Add a function UnwarpCheckPointExecutor to get a real executor without checkpoints.

Use UnwarpCheckPointExecutor to executor ss -tln when checking wether components are started/stopped.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  1. Deploy a new cluster and check audit logs.

    The ss -tln command isn't cached when starting components.

// ...
2023-11-01T02:34:40.179Z        INFO    Starting component blackbox_exporter
2023-11-01T02:34:40.179Z        INFO            Starting instance 127.0.0.1
2023-11-01T02:34:41.152Z        INFO    LocalCommand    {"cmd": "export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H -u root bash -c \"cd; systemctl daemon-reload && systemctl start blackbox_exporter-9115.service\"", "stdout": "", "stderr": ""}
2023-11-01T02:34:41.153Z        INFO    CheckPoint      {"host": "127.0.0.1", "port": 22, "user": "tidb", "sudo": true, "cmd": "systemctl daemon-reload && systemctl start blackbox_exporter-9115.service", "stdout": "", "stderr": "", "__hash__": "7634a3f7", "__func__": "github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute", "hit": false}
2023-11-01T02:34:41.189Z        INFO    LocalCommand    {"cmd": "export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H -u tidb bash -c \"cd; ss -ltn\"", "stdout": "State  Recv-Q Send-Q      Local Address:Port  Peer Address:PortProcess\nLISTEN 0      70              127.0.0.1:33060      0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:45347      0.0.0.0:*          \nLISTEN 0      200             127.0.0.1:8234       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:9090       0.0.0.0:*          \nLISTEN 0      511             127.0.0.1:42251      0.0.0.0:*          \nLISTEN 0      4096           127.0.0.54:53         0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:2380       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:2379       0.0.0.0:*          \nLISTEN 0      4096        127.0.0.53%lo:53         0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:3000       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:44301      0.0.0.0:*          \nLISTEN 0      151             127.0.0.1:3306       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:44037      0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:4000       0.0.0.0:*          \nLISTEN 0      128             127.0.0.1:20292      0.0.0.0:*          \nLISTEN 0      128             127.0.0.1:20180      0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:12020      0.0.0.0:*          \nLISTEN 0      4096                    *:9100             *:*          \nLISTEN 0      4096                    *:22               *:*          \nLISTEN 0      4096                    *:8300             *:*          \nLISTEN 0      4096                    *:10080            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:3930             *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:3930             *:*          \nLISTEN 0      4096                    *:9527             *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160
*:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \n", "stderr": ""}
2023-11-01T02:34:42.229Z        INFO    LocalCommand    {"cmd": "export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H -u tidb bash -c \"cd; ss -ltn\"", "stdout": "State  Recv-Q Send-Q      Local Address:Port  Peer Address:PortProcess\nLISTEN 0      70              127.0.0.1:33060      0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:45347      0.0.0.0:*          \nLISTEN 0      200             127.0.0.1:8234       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:9090       0.0.0.0:*          \nLISTEN 0      511             127.0.0.1:42251      0.0.0.0:*          \nLISTEN 0      4096           127.0.0.54:53         0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:2380       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:2379       0.0.0.0:*          \nLISTEN 0      4096        127.0.0.53%lo:53         0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:3000       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:44301      0.0.0.0:*          \nLISTEN 0      151             127.0.0.1:3306       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:44037      0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:4000       0.0.0.0:*          \nLISTEN 0      128             127.0.0.1:20292      0.0.0.0:*          \nLISTEN 0      128             127.0.0.1:20180      0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:12020      0.0.0.0:*          \nLISTEN 0      4096                    *:9100             *:*          \nLISTEN 0      4096                    *:9115             *:*          \nLISTEN 0      4096                    *:22               *:*          \nLISTEN 0      4096                    *:8300             *:*          \nLISTEN 0      4096                    *:10080            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:3930             *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:3930             *:*          \nLISTEN 0      4096                    *:9527
*:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \n", "stderr": ""}
2023-11-01T02:34:42.229Z        INFO            Start 127.0.0.1 success
// ...
  1. Upgrade the cluster and check audit logs.

    The ss -tln command isn't cached when stopping components.

2023-11-01T02:36:29.240Z        INFO    Stopping component node_exporter
2023-11-01T02:36:29.246Z        INFO            Stopping instance 127.0.0.1
2023-11-01T02:36:30.044Z        INFO    LocalCommand    {"cmd": "export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H -u root bash -c \"cd; systemctl daemon-reload && systemctl stop node_exporter-9100.service\"", "stdout": "", "stderr": ""}
2023-11-01T02:36:30.044Z        INFO    CheckPoint      {"host": "127.0.0.1", "port": 22, "user": "tidb", "sudo": true, "cmd": "systemctl daemon-reload && systemctl stop node_exporter-9100.service", "stdout": "", "stderr": "", "__hash__": "7634a3f7", "__func__": "github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute", "hit": false}
2023-11-01T02:36:30.072Z        INFO    LocalCommand    {"cmd": "export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H -u tidb bash -c \"cd; ss -ltn\"", "stdout": "State  Recv-Q Send-Q      Local Address:Port  Peer Address:PortProcess\nLISTEN 0      70              127.0.0.1:33060      0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:45347      0.0.0.0:*          \nLISTEN 0      200             127.0.0.1:8234       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:9090       0.0.0.0:*          \nLISTEN 0      511             127.0.0.1:42251      0.0.0.0:*          \nLISTEN 0      4096           127.0.0.54:53         0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:2380       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:2379       0.0.0.0:*          \nLISTEN 0      4096        127.0.0.53%lo:53         0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:3000       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:44301      0.0.0.0:*          \nLISTEN 0      151             127.0.0.1:3306       0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:44037      0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:4000       0.0.0.0:*          \nLISTEN 0      128             127.0.0.1:20292      0.0.0.0:*          \nLISTEN 0      128             127.0.0.1:20180      0.0.0.0:*          \nLISTEN 0      4096            127.0.0.1:12020      0.0.0.0:*          \nLISTEN 0      4096                    *:9115             *:*          \nLISTEN 0      4096                    *:22               *:*          \nLISTEN 0      4096                    *:8300             *:*          \nLISTEN 0      4096                    *:10080            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:3930             *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:3930             *:*          \nLISTEN 0      4096                    *:9527             *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160
*:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20160            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \nLISTEN 0      4096   [::ffff:127.0.0.1]:20170            *:*          \n", "stderr": ""}
2023-11-01T02:36:30.072Z        INFO            Stop 127.0.0.1 success
  • No code

Code changes

  • Has exported function/method change
  • Has exported variable/fields change
  • Has interface methods change
  • Has persistent data change

Side effects

  • Possible performance regression
  • Increased code complexity
  • Breaking backward compatibility

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation

Release notes:

NONE

Copy link
Contributor

ti-chi-bot bot commented Nov 1, 2023

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign srstack for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 1, 2023
@codecov-commenter
Copy link

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (1477739) 55.01% compared to head (9897aa7) 55.33%.

❗ Current head 9897aa7 differs from pull request most recent head 3c878fb. Consider uploading reports for the commit 3c878fb to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2298      +/-   ##
==========================================
+ Coverage   55.01%   55.33%   +0.32%     
==========================================
  Files         326      326              
  Lines       34473    34479       +6     
==========================================
+ Hits        18964    19076     +112     
+ Misses      13237    13117     -120     
- Partials     2272     2286      +14     
Flag Coverage Δ
cluster 44.47% <71.43%> (+0.39%) ⬆️
dm 25.47% <71.43%> (+0.01%) ⬆️
playground 15.43% <0.00%> (-<0.01%) ⬇️
tiup 15.69% <0.00%> (+0.02%) ⬆️
unittest 22.43% <0.00%> (-<0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
pkg/cluster/api/tidbapi.go 0.00% <ø> (ø)
pkg/cluster/module/wait_for.go 80.95% <100.00%> (ø)
pkg/cluster/executor/executor.go 64.00% <66.67%> (+0.17%) ⬆️

... and 10 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ti-chi-bot ti-chi-bot bot added the lgtm label Nov 1, 2023
Copy link
Contributor

ti-chi-bot bot commented Nov 1, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-11-01 06:38:08.855438762 +0000 UTC m=+3021486.442548892: ☑️ agreed by kaaaaaaang.

@KanShiori KanShiori merged commit 721b065 into pingcap:master Nov 1, 2023
17 of 18 checks passed
@KanShiori KanShiori deleted the shiori/fix-checkpoint branch November 1, 2023 07:57
@kaaaaaaang kaaaaaaang added this to the v1.14.0 milestone Nov 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lgtm size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

tiup cluster replay keeps using cached checkpoint after stopping service
3 participants