Skip to content

PodresourceAPI reports about resources of pods in terminal phase #119423

Closed
#132028
Closed
PodresourceAPI reports about resources of pods in terminal phase#119423
#132028
Labels
kind/bugCategorizes issue or PR as related to a bug.priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.sig/nodeCategorizes an issue or PR as relevant to SIG Node.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

Description

@Tal-or
Tal-or
opened on Jul 19, 2023

What happened?

PodResourcesAPI provide a List() endpoint which reports about all the resources that consumed by pods and containers on the node.

The problem is that pods which are in terminal phase (i.e. are in Failed or Succeeded status) are reported as well. about The internal managers reassign resources assigned to pods in terminal phase, so PodResources should ignore them, because they can still be in used.

What did you expect to happen?

PodResources should ignore and not reports about resources which are in used by pods which are in terminal phase.

How can we reproduce it (as minimally and precisely as possible)?

Provided a new test-case that demonstrates the exact problem and can be used as a reproducer: #119402

Anything else we need to know?

The docs that describes PodResourceAPI are not refer explicitly to whether List() should ignore terminal pod's resources or not, meaning it might not be a bug at all.

OTOH, internal managers reassign resources assigned to pods in terminal phase so it make sense to not account them.

Kubernetes version

Kubernetes v1.26.2

Cloud provider

Details

OS version

Details 
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Details

Container runtime (CRI) and version (if applicable)

Details

Related plugins (CNI, CSI, ...) and versions (if applicable)

Details

Activity

  1. Tal-or
    added
    kind/bugCategorizes issue or PR as related to a bug.
    on Jul 19, 2023
  2. k8s-ci-robot
    added
    needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
    on Jul 19, 2023

  3. ffromani commented on Jul 19, 2023

    @ffromani
    ffromani
    on Jul 19, 2023
    Contributor

    /sig node

  4. k8s-ci-robot
    added
    sig/nodeCategorizes an issue or PR as relevant to SIG Node.
    and removed
    needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
    on Jul 19, 2023

  5. SergeyKanzhelev commented on Jul 19, 2023

    @SergeyKanzhelev
    SergeyKanzhelev
    on Jul 19, 2023
    Member

    /triage accepted

  6. k8s-ci-robot
    added
    triage/acceptedIndicates an issue or PR is ready to be actively worked on.
    and removed
    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
    on Jul 19, 2023

  7. carlory commented on Jul 21, 2023

    @carlory
    carlory
    on Jul 21, 2023
    Member

    /assign

  8. k8s-ci-robot
    assigned
    carlory
    on Jul 21, 2023

  9. ffromani commented on Jul 27, 2023

    @ffromani
    ffromani
    on Jul 27, 2023
    Contributor

    I was thinking about this issue for a bit, and the root cause is probably that there is not a (good enough) formal spec for the API itself so the implementation is pretty much the reference.
    We should probably improve here but not sure how (e.g. where to describe more formally the API, perhaps just in the docs?)

    That said, my thoughts about the ideal semantics are:

    1. Considering the docs as the most accurate, possibly the only, spec "The List endpoint provides information on resources of running pods, with details such as the id of exclusively allocated CPUs, device id as it was reported by device plugins and id of the NUMA node where these devices are allocated. Also, for NUMA-based machines, it contains the information about memory and hugepages reserved for a container." This alone should be sufficient to exclude pods in terminal phase from the running set.
    2. I need to doublecheck the actual behavior of the resource managers. If resources assigned to containers are NOT released until the containers are actually cleared by the system, then the API is returning the correct data because these resources can't be re-allocated to newly admitted containers, and the accounting must reflect that
    3. But this brings us to the question: is the allocation behavior of the resource manager documented? should it?

  10. SergeyKanzhelev commented on Jul 28, 2023

    @SergeyKanzhelev
    SergeyKanzhelev
    on Jul 28, 2023
    Member

    I wonder if pod resources API may report the same set of resources used by two pods - one succeeded and one newly scheduled in a single response?

    @ffromani should we try to summarize shortcomings at the page you listed as a starting point? We can start with the list of edge cases like:

    • "You cannot rely on data completeness while kubelet is starting"
    • "Succeeded or Failed pods may be listed"
    • "When container has failed, can resources still be allocated?"
    • "Is pod information guaranteed visible when NRI plugins are being called for this pod?"
    • "what about init containers resources? When it will be returned and when not?"

    I like the idea that podresources API is driven by the fact whether resources are actually in use, not by the Pod status" Thus the resource manager behavior ideally needs to be documented.

  11. ffromani commented on Jul 28, 2023

    @ffromani
    ffromani
    on Jul 28, 2023
    Contributor

    @SergeyKanzhelev I totally agree that the behavior should be documented in a more precse way. I'm just not sure that page is the best place for reference docs. I'll check with SIG-docs and get advice.

  12. ffromani commented on Jul 28, 2023

    @ffromani
    ffromani
    on Jul 28, 2023
    Contributor

    I wonder if pod resources API may report the same set of resources used by two pods - one succeeded and one newly scheduled in a single response?

    From top of my head, I'm not sure we have a code path which prevents this. It should be extremely unlikely (at very least you will need a very limited amount of resources, bypass the scheduler entirely and hit a race within the kubelet) but possible. Should also be transient - the next query should return the expected consistent result. But still, I'm not sure there's prevention for this flow. Also depends on the specifics of resource managers.

    The gist is that the podresources API is a thin layer which reformats and exposes the internal data of resource managers. This makes the code very simple, but leaks pretty badly the internal implementation of the resource managers allocation process.

    This in turns makes the API not really formally specified and with perhaps poor stability guarantees.

  13. 60 remaining items

  14. haircommander
    added 2 commits that reference this issue on Aug 21, 2025

    UPSTREAM: 132028: podresources: list: use active pods in list

    4795ccd

    UPSTREAM: 132028: podresources: list: use active pods in list

    22692b9
  15. haircommander
    added a commit that references this issue on Sep 9, 2025

    UPSTREAM: 132028: podresources: list: use active pods in list

    63ee843
  16. swatisehgal
    added 3 commits that reference this issue on Oct 21, 2025

    node: e2e: update podresources check post fix of kubernetes#119423

    164eac7

    node: e2e: update podresources check post fix of kubernetes#119423

    2769026

    node: e2e: update podresources check post fix of kubernetes#119423

    1e3a6e1
  17. k8s-ci-robot
    added a commit that references this issue on Oct 28, 2025

    Merge pull request #134816 from swatisehgal/device-plugin-e2e-flakine…

    7be689f
  18. LogicalShark
    added a commit that references this issue on Oct 29, 2025

    node: e2e: update podresources check post fix of kubernetes#119423

    fc91340
  19. darshansreenivas
    added a commit that references this issue on Nov 2, 2025

    node: e2e: update podresources check post fix of kubernetes#119423

    ff3c541
  20. yt2985
    added a commit that references this issue on Nov 4, 2025

    node: e2e: update podresources check post fix of kubernetes#119423

    1f57069
  21. mhan8796
    added 4 commits that reference this issue on Jun 27, 2026

    podresources: list: use active pods in list

    ca66ff7

    e2e: podresources: disable memory manager integration

    a777690

    node: e2e: update podresources check post fix of kubernetes#119423

    bdce26b

    Merge pull request kubernetes#134816 from swatisehgal/device-plugin-e…

    3875f83
  22. github-actions
    added a commit that references this issue on Sep 12, 2026

    nrt: cache: add support for dedicated informer

    99daf8b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.sig/nodeCategorizes an issue or PR as relevant to SIG Node.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions