Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

limiting bandwidth and iops per container. #27000

Closed
timothysc opened this issue Jun 7, 2016 · 24 comments

Comments

@timothysc
Copy link
Member

@timothysc timothysc commented Jun 7, 2016

We've seen several issues where multiple concurrent openshift builds running on a node can denial of service attack the nodes control plane such that the controller thinks the node is unresponsive.

/cc @kubernetes/sig-node @kubernetes/sig-network @jeremyeder

xref: #21331

@vishh

This comment has been minimized.

Copy link
Member

@vishh vishh commented Jun 7, 2016

It was just a matter of time before this issue surfaced 😉
Excepting logs, I think we can isolate all other modes of disk access. Logs flow through the docker daemon as of now.
To begin with, we should expose disk IO stats so that the user has enough context to react to node outages.
Next step will be to explore how to isolate disk IO. Given that we will have cgroups at the QoS level soon, we can let system daemons and Guaranteed pods to have higher proportional access to disk over Burstable or Besteffort pods. That works only if CFQ is enabled though.

@derekwaynecarr

This comment has been minimized.

Copy link
Member

@derekwaynecarr derekwaynecarr commented Jun 7, 2016

I am not sure giving proportional access will be enough within a given QoS. If everything is Burstable, it would be good to vary iops within that tier on a per pod basis, but agree with the requisite steps outlined by @vishh.

@jeremyeder

This comment has been minimized.

Copy link
Member

@jeremyeder jeremyeder commented Jun 8, 2016

We can take a piece-meal approach to nailing down the reasons we can swamp the control plane, but I thought taking the inverse approach would be better and create "floors" for the all the processes in the system-reserved cgroups...no? "shield" them no matter what the apps do?

BTW RHEL defaults to deadline, not cfq, but that's an aside for now...

Further...specifically about builds... I may not even want those co-located with my prod apps. I want to shove them off in a dark corner where they can't affect anything in the production zone. This is like compiling an app on your prod boxes and wondering why your page load times suddenly suck. Poor design IMHO.

@timothysc

This comment has been minimized.

Copy link
Member Author

@timothysc timothysc commented Jun 9, 2016

We can take a piece-meal approach to nailing down the reasons we can swamp the control plane, but I thought taking the inverse approach would be better and create "floors" for the all the processes in the system-reserved cgroups...no? "shield" them no matter what the apps do?

Yes, but we're still going to have the noisy neighbors problem so doesn't hurt to start the conversation. If it's not builds it will be something else.

@pinkencryption

This comment has been minimized.

Copy link

@pinkencryption pinkencryption commented Jun 23, 2017

I would like to request to prioritize this. It is important to isolate IO to run pods properly to include database storage etc.

@pinkencryption

This comment has been minimized.

Copy link

@pinkencryption pinkencryption commented Jun 23, 2017

This might also help run a system like hadoop without affecting the node health

@vishh

This comment has been minimized.

Copy link
Member

@vishh vishh commented Jun 26, 2017

FYI: The general guideline thus far is to increase network bandwidth and user storage over the network for most applications.

@pinkencryption

This comment has been minimized.

Copy link

@pinkencryption pinkencryption commented Jun 26, 2017

@vishh : Can you elaborate ?

@astropuffin

This comment has been minimized.

Copy link

@astropuffin astropuffin commented Oct 19, 2017

@vishh "increase network bandwidth" isn't an option for iops centered workloads (like image processing). Docker now offers very useful flags to handle this:

      --device-read-bps value       Limit read rate (bytes per second) from a device (default [])
      --device-read-iops value      Limit read rate (IO per second) from a device (default [])
      --device-write-bps value      Limit write rate (bytes per second) to a device (default [])
      --device-write-iops value     Limit write rate (IO per second) to a device (default [])

Not sure about rkt support, but I don't think the scope needs to be that large. Assuming docker as the runtime, iops and bps could be added as another optional container level resource like cpu and memory. Storage limits is handled much better with persistent volumes.

@liqlin2015

This comment has been minimized.

Copy link

@liqlin2015 liqlin2015 commented Nov 10, 2017

Any update on this issue? Will pod support iops and bps option as docker?

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Feb 8, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@mkleint

This comment has been minimized.

Copy link

@mkleint mkleint commented Feb 25, 2018

Any update on this issue? Will pod support iops and bps option as docker?

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Mar 27, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@mkleint

This comment has been minimized.

Copy link

@mkleint mkleint commented Mar 27, 2018

Any update on this issue? Will pod support iops and bps option as docker?

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Apr 26, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@mkleint

This comment has been minimized.

Copy link

@mkleint mkleint commented Apr 27, 2018

closed just in time to remind me to corner someone about this at Kubecon!

@pinkencryption

This comment has been minimized.

Copy link

@pinkencryption pinkencryption commented Apr 27, 2018

This needs to happen, how can we request this ?

@max-lobur

This comment has been minimized.

Copy link

@max-lobur max-lobur commented Oct 25, 2018

Having same issue with own k8s distro here. Would like to see this feature

@lovejoy

This comment has been minimized.

Copy link
Contributor

@lovejoy lovejoy commented Oct 26, 2018

We made some modification to k8s to add storage limit option to pod's resource ,and it use the blkio cgroup to control device read-write bandwidth. Although we know that the blkio can't control all the io bandwidth ,It works well now.

@max-lobur

This comment has been minimized.

Copy link

@max-lobur max-lobur commented Oct 26, 2018

@lovejoy what k8s release? can you share doc link pls, if it's published already
Thank You

@vishh

This comment has been minimized.

Copy link
Member

@vishh vishh commented Oct 29, 2018

@pinkencryption why does this feature need to happen? Have you thought of how this feature (IOPS isolation) will function in a coherent way across diverse environments (spinning/solid/network storage media)?
Is using local (NVMe) SSDs an option whenever IO is a constraint, especially for workloads like ML? Can databases be dedicated physical spinning disks especially when I/O is a constraint?

@lovejoy

This comment has been minimized.

Copy link
Contributor

@lovejoy lovejoy commented Nov 2, 2018

@max-lobur We made some changes like this #70573

@xiegang1

This comment has been minimized.

Copy link

@xiegang1 xiegang1 commented Feb 25, 2019

Do we have any update about this feature? Can we isolated the iops now in k8s? We are planning to migrate our hadoop apps to k8s, and think iops could be a problem and iops isolation is required.

@Pandoraemon

This comment has been minimized.

Copy link

@Pandoraemon Pandoraemon commented Jul 4, 2019

Any update on this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.