Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resourcemanager: stop scheduler in the old linux kernal #41766

Merged
merged 4 commits into from Mar 2, 2023
Merged

resourcemanager: stop scheduler in the old linux kernal #41766

merged 4 commits into from Mar 2, 2023

Conversation

hawkingrei
Copy link
Member

@hawkingrei hawkingrei commented Feb 27, 2023 •
edited

What problem does this PR solve?

Issue Number: close #41808

Problem Summary:

If you run the TiDB in the container with a kernel version lower than 4.7, you can not get the CPU usage of this container from groups. We get the CPU usage from the usage_user and usage_sys of cgroups. But those values are created on Linux 4.7.

In summary, We prevent unnecessary trouble. If we find the missing path of the cgroups path, we will stop the scheduler of the resource manager so that it is still able to work but it can not tune the concurrency.

the main cause in linux is here.

https://elixir.bootlin.com/linux/v4.7/source/kernel/sched/cpuacct.c
https://elixir.bootlin.com/linux/v4.6.7/source/kernel/sched/cpuacct.c

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

run on the old Linux kernel. you will read the error message and then not start the scheduler.

  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@hawkingrei
resourcemanager: stop scheduler in the old linux kernal
31ff410 
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Feb 27, 2023 •
edited

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • lance6716
  • zimulala

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note-none size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 27, 2023
hawkingrei
hawkingrei commented Mar 1, 2023
View reviewed changes
util/cpu/cpu.go
@@ -56,6 +59,12 @@ func NewCPUObserver() *Observer {

// Start starts the cpu observer.
func (c *Observer) Start() {
_, err := cgroup.GetCgroupCPU()
Copy link
Member Author

@hawkingrei hawkingrei Mar 1, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the kernel version is too old, here will return an error and not start to scheduler the pool.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 1, 2023
zimulala
zimulala approved these changes Mar 1, 2023
View reviewed changes
Copy link
Contributor

@zimulala zimulala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 1, 2023
@hawkingrei hawkingrei requested a review from tangenta March 2, 2023 06:51
tangenta
tangenta reviewed Mar 2, 2023
View reviewed changes
util/cpu/cpu_test.go Outdated Show resolved Hide resolved
@hawkingrei
refactor test
78b80aa 
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
@ti-chi-bot ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 2, 2023
@hawkingrei hawkingrei requested a review from tangenta March 2, 2023 07:54
@hawkingrei
Copy link
Member Author

PTAL @tangenta

@hawkingrei
Copy link
Member Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 78b80aa

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 2, 2023
@ti-chi-bot ti-chi-bot merged commit b18d0eb into pingcap:master Mar 2, 2023
@hawkingrei hawkingrei deleted the fix_docker_in_old_kernal branch March 2, 2023 09:06
@Benjamin2037
Copy link
Collaborator

Why only manual testing?

Benjamin2037
Benjamin2037 reviewed Mar 4, 2023
View reviewed changes
util/cpu/cpu_test.go
require.Less(t, GetCPUUsage(), 1.0)
for n := 0; n < 10; n++ {
time.Sleep(1 * time.Second)
value, unsupported := GetCPUUsage()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please mock unsupported scenarios and make sure we will stop tune.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Benjamin2037 Benjamin2037 Benjamin2037 left review comments

@lance6716 lance6716 lance6716 approved these changes

@zimulala zimulala zimulala approved these changes

@tangenta tangenta tangenta approved these changes

@wjhuang2016 wjhuang2016 Awaiting requested review from wjhuang2016

Assignees
No one assigned
Labels
release-note-none size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

disable the scheduler when run under too old cgroups
6 participants
@hawkingrei @ti-chi-bot @Benjamin2037 @lance6716 @zimulala @tangenta