Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit iops per container #70980

Closed

Conversation

@Avanpourm
Copy link

Avanpourm commented Nov 13, 2018

What type of PR is this?
/kind feature

What this PR does / why we need it:
I need to limit the read and write iops of rootfs
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #70364

Special notes for your reviewer:
Thankyou !!!

Does this PR introduce a user-facing change?:

Use the annotation of kubernetes to pass the parameters of the cgroup to control the container blkio
Specifying the device path as "rootfs" indicates that you specify limiting rootfs
For example:
io.kubernetes.container.blkio: '{"weight":200,"weight_device":[{"device":"rootfs","value":"200k"}],"device_read_bps":[{"device":"/dev/sda1","value":"20m"}],"device_write_bps":[{"device":"rootfs","value":"20m"}],"device_read_iops":[{"device":"rootfs","value":"200"}]"device_write_iops":[{"device":"rootfs","value":"300"}]}'

Parameter

parameter type default value means parameter range example corresponds to Cgoup
weight Uint16 0 Weight value for all devices 10-1000 1
weight_device Array nil Weight value for all devices -
weight_device.[*].device String The absolute path of the device /dev/sda1
weight_device.[*].value String Weight value of the device 10-1000
device_read_bps Array nil A device reads the maximum number of bytes per second
device_read_bps.[*].device String The absolute path of the device. You can use "rootfs" for the specified root directory
device_read_bps.[*].value String Value ,like: 1[ K M G ]. Limit the minimum to prevent the process from entering a D state >1k 1M
device_write_bps Array nil Maximum number of bytes per second allowed to write to the device
device_write_bps.[*].device String The absolute path of the device. You can use "rootfs" for the specified root directory
device_write_bps.[*].value String Value ,like: 1[ K M G ]. Limit the minimum to prevent the process from entering a D state >1k
device_read_iops Array nil Max IOPS
device_read_iops.[*].device String The absolute path of the device. You can use "rootfs" for the specified root directory
device_read_iops.[*].value String Value ,like: 1[ K M G ]. Limit the minimum to prevent the process from entering a D state >5
device_write_iops Array nil Max IOPS
device_write_iops.[*].device String The absolute path of the device. You can use "rootfs" for the specified root directory
device_write_iops.[*].value String Value ,like: 1[ K M G ]. Limit the minimum to prevent the process from entering a D state >5

Deployment Example

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: iops-app
  name: iops-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: iops-app
  template:
    metadata:
      labels:
        app: iops-app
      annotations:
        io.kubernetes.container.blkio: '{"weight":200,"weight_device":[{"device":"rootfs","value":"200k"}],"device_read_bps":[{"device":"/dev/sad1","value":"20m"}],"device_write_bps":[{"device":"rootfs","value":"20m"}],"device_read_iops":[{"device":"rootfs","value":"200"}],"device_write_iops":[{"device":"rootfs","value":"300"}]}'
    spec:
      nodeSelector:
        kubernetes.io/hostname: szdc-k8sm-2-2.meitu-inc.com
      containers:
      - image: docker.io/centos:7
        command: ["/bin/sh","-c"]
        args: ["while(true);do /bin/sleep 10;done"]
        name: iops-app
        resources:
          requests:
            memory: "200Mi"
            cpu: 1
          limits:
            memory: "200Mi"
            cpu: 1
        volumeMounts:
        - name: host-time
          mountPath: /etc/localtime
      volumes:
      - name: host-time
        hostPath:
          path: /etc/localtime
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 13, 2018

@Avanpourm: This PR is not for the master branch but does not have the cherry-pick-approved label. Adding the do-not-merge/cherry-pick-not-approved label.

To approve the cherry-pick, please assign the patch release manager for the release branch by writing /assign @username in a comment when ready.

The list of patch release managers for each release can be found here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 13, 2018

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 13, 2018

Hi @Avanpourm. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 13, 2018

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Avanpourm
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: derekwaynecarr

If they are not already assigned, you can assign the PR to them by writing /assign @derekwaynecarr in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Avanpourm added 3 commits Oct 29, 2018
We hope that kubernetes support limiting blkio resources.We use the annotation of kubernetes to pass
the parameters of the cgroup to control the container blkio. The current features apply only to the
Linux devmapper environment.In my production environment, multi-tenancy shares a storage device, we
must isolate the io of each tenant to ensure that the tenants do not interfere with each other. CRI
does not provide an interface for IO resource isolation.
@Avanpourm Avanpourm force-pushed the Avanpourm:limit-iops-per-container branch from 487f903 to 1c99dcb Nov 13, 2018
@Avanpourm

This comment has been minimized.

Copy link
Author

Avanpourm commented Nov 13, 2018

/assign @derekwaynecarr

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 15, 2018

@Avanpourm: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@lovejoy

This comment has been minimized.

Copy link
Contributor

lovejoy commented Dec 11, 2018

@Avanpourm Would you please request this pr to master branch ?

@Avanpourm

This comment has been minimized.

Copy link
Author

Avanpourm commented Dec 12, 2018

@feiskyer

This comment has been minimized.

Copy link
Member

feiskyer commented Jan 4, 2019

@Avanpourm Thanks for contributing. For new features, please send PRs to master branch instead.

@feiskyer

This comment has been minimized.

Copy link
Member

feiskyer commented Feb 21, 2019

Please create a new PR targeted at master branch. stable branches are only for bug fixes, not for new features.

/close

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Feb 21, 2019

@feiskyer: Closed this PR.

In response to this:

Please create a new PR targeted at master branch. stable branches are only for bug fixes, not for new features.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@cuericlee

This comment has been minimized.

Copy link
Contributor

cuericlee commented Sep 6, 2019

@Avanpourm did you get chance to request to merge this PR to master branch?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.