Quotas for Ephemeral Storage

[RobertKrawitz]

Enhancement Description

• Use filesystem quotas to monitor local ephemeral storage utilization
• Kubernetes Enhancement Proposal: 1029-ephemeral-storage-quotas
• Primary contact (assignee): @RobertKrawitz @pacoxu
• Responsible SIGs: SIG-Node
• Reviewer(s) - @dashpole @derekwaynecarr
• Approver: #derekwaynecarr @dchen1107
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.15
  • Beta release target (x.y): 1.29?
    • promoted in v1.25.0 and then reverted in v1.25.1
  • Stable release target (x.y): TBD
• Alpha
  • KEP (k/enhancements) update PR(s): Updates from SIG-Node meeting 20181204 #646
  • Code (k/k) update PR(s): Ephemeral storage monitoring via filesystem quotas kubernetes#66928
  • Docs (k/website) update(s): Add support for quotas for ephemeral storage monitoring. website#14268
• Reverted Beta (first-try: promoted in v1.25.0 and then reverted in v1.25.1)
  • KEP (k/enhancements) update PR(s): promote ephemeral-storage-quotas to beta in 1.25 #2697
  • Code (k/k) update PR(s):
    • promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta kubernetes#107329
    • add VolumeStatCalDuration metrics for fsquato monitoring benchmark kubernetes#107201
    • assignQuota checks if the underlying medium supports quotas and if so setting it kubernetes#107302
    • add warning log if volume calculation took too long than 1 second kubernetes#107490
  • Docs(k/website) updates: Revert: promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta website#36401
  • Revert PR: Revert "promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta" kubernetes#112076, Changelog-1.25: add 1.25.0 known issues section kubernetes#112106, LocalStorageCapacityIsolationFSQuotaMonitoring revert to alpha for regression #3485
• Beta (second-try)
  • KEP (k/enhancements) update PR(s): [v1.29] ephemeral-storage-quotas: repromote to beta #3821
  • Code (k/k) update PR(s):
    • Add CI for quota monitoring test-infra#28614
    • 1.27 fsquota: only generate pod uuid is nil kubernetes#112624
    • 1.27 kubelet: Fix fs quota monitoring on volumes kubernetes#115314
    • promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta kubernetes#112626
  • Docs (k/website) update(s): [WIP] Update LocalStorageCapacityIsolationFSQuotaMonitoring to beta website#43455

[RobertKrawitz]

/sig node

[derekwaynecarr]

per sig-node discussion, this is good.

the kep will merge here: #646

/milestone v1.15

[kacole2]

@RobertKrawitz This is OK to track for 1.15. However, please update the KEP with established graduation criteria for Beta and Alpha when possible.

/stage alpha

[makoscafee]

Hey, @RobertKrawitz @derekwaynecarr 👋 I'm the v1.15 docs Lead.
Does this enhancement require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[RobertKrawitz]

Yes, this will require new documentation.

I will open the website PR today.

[kacole2]

Hi @RobertKrawitz @derekwaynecarr . Code Freeze is Thursday, May 30th 2019 @ EOD PST. All enhancements going into the release must be code-complete, including tests, and have docs PRs open.

Please list all current k/k PRs so they can be tracked going into freeze. If the PRs aren't merged by freeze, this feature will slip for the 1.15 release cycle. Only release-blocking issues and PRs will be allowed in the milestone.

If you know this will slip, please reply back and let us know. Thanks!

[makoscafee]

Hey, @RobertKrawitz @derekwaynecarr .Just a friendly reminder we're looking for at least a draft/placeholder PR against k/website (branch dev-1.15) due by Thursday, May 30th 2019 @ EOD PST.

[RobertKrawitz]

@makoscafee The k/website PR is kubernetes/website#14268

[kacole2]

Hi @RobertKrawitz @derekwaynecarr , today is code freeze. I do not see a reply for any k/k PRs to track for this merge. It's now being marked as At Risk in the 1.15 Enhancement Tracking Sheet. If there is no response, or you respond with PRs to track and they are not merged by EOD PST, this will be dropped from the 1.15 Milestone. After this point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[RobertKrawitz]

The PR for this is kubernetes/kubernetes#66928

[RobertKrawitz]

/hold cancel

[kacole2]

Hi @RobertKrawitz , I'm the 1.16 Enhancement Lead. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.6 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[kcmartin]

Hello @RobertKrawitz -- 1.17 Enhancement Shadow here! 🙂

I wanted to reach out to see if this enhancement will be graduating to alpha/beta/stable in 1.17?


Please let me know so that this enhancement can be added to 1.17 tracking sheet.

Thank you!

🔔Friendly Reminder

The current release schedule is

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pacoxu]

/remove-lifecycle stale
A related issue: kubernetes/kubernetes#114506

[pacoxu]

If kubernetes/kubernetes#112624 can fix the bug we meet in v1.25.0, can we re-promote this feature to beta?

[pacoxu]

There are some discussions about this feature in the last sig node meetings.

• Sig-Node Weekly meeting: Jan 17th, 2023 Recording: https://youtu.be/wirWRKSqY10

Why do we have this feature?: we don’t have an efficient way to track disk usage

I prefer to promote this feature if the blocking bugs(kubernetes/kubernetes#112081) are fixed. The reason is that du in some scenarios will hang or take too much time. It happens in our customers' clusters and we want to alter it to a stable quick way for that. This is one of the solutions.

Of course, there may be some other ways.

@rphillips: user can change the project id of a directory which becomes a security problem.

This is one of the reasons that @rphillips asked for deprecating this feature as it is not promoted to beta for a long time. Should we open an issue to track this security problem?

[pacoxu]

https://testgrid.k8s.io/sig-node-kubelet#kubelet-gce-e2e-fsquota-ubuntu is added for this feature to enable it in tests.

[SergeyKanzhelev]

/milestone v1.28

adding tracking of this feature since it is being worked on

[pacoxu]

#3821

• I still prefer to promote it to beta again since the significant bug is fixed.

[SergeyKanzhelev]

/label lead-opted-in

[npolshakova]

Hello @pacoxu 👋, 1.29 Enhancements team here!

Just checking in as we approach enhancements freeze on 01:00 UTC, Friday, 6th October, 2023.

This enhancement is targeting for stage beta for 1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.29. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

It looks like https://github.com/kubernetes/enhancements/pull/3821/files will address most of these issues.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[npolshakova]

Hi @pacoxu, just checking in once more as we approach the 1.29 enhancement freeze deadline this week on 01:00 UTC, Friday, 6th October, 2023. The status of this enhancement is marked as at risk for enhancement freeze.

It looks like when #3821 merges in it will address most of the requirements. Let me know if I missed anything. Thanks!

[npolshakova]

With #3821 merged this is now tracked for enhancement freeze for 1.29. 🚀

[drewhagen]

Hello @RobertKrawitz @pacoxu 👋, v1.29 Docs Shadow here.
Does this enhancement work planned for v1.29 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023.
Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[pacoxu]

kubernetes/website#43455 is open for website.

[a-mccarthy]

Hi @pacoxu, 👋 from the v1.29 Release Team-Communications! We would like to check if you have any plans to publish a blog for this KEP regarding new features, removals, and deprecations for this release.

If so, you need to open a PR placeholder in the website repository.
The deadline will be on Tuesday 14th November 2023 (after the Docs deadline PR ready for review)

Here is the 1.29 calendar

[npolshakova]

Hey again @pacoxu 👋 1.29 Enhancements team here,

Just checking in as we approach code freeze at 01:00 UTC Wednesday 1st November 2023: .

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs have been merged:

• Add CI for quota monitoring test-infra#28614
• fsquota: only generate pod uuid is nil kubernetes#112624
• kubelet: Fix fs quota monitoring on volumes kubernetes#115314

The following PRs are open and need to be merged before code freeze (and we need to update the Issue description to include all the related PRs of this KEP):

• promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta kubernetes#112626

The status of this KEP is currently at risk for code freeze.

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.
As always, we are here to help if any questions come up. Thanks!

[pacoxu]

• promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta kubernetes#112626

The pr needs review/approve.
@rphillips @SergeyKanzhelev @derekwaynecarr PTAL

[npolshakova]

Hello @pacoxu 👋 1.29 Enhancements lead here,

Unfortunately, the implementation (code related) PR(s) associated with this enhancement is not in the merge-ready state by code-freeze and hence this enhancement is now removed from the 1.29 milestone.

If you still wish to progress this enhancement in 1.29, please file an exception request. Thanks!

/milestone clear