Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PSI Cluster Quota resported as full without any resource consuming it. #2493

Closed
harshad16 opened this issue Apr 4, 2022 · 3 comments
Closed

PSI Cluster Quota resported as full without any resource consuming it. #2493

harshad16 opened this issue Apr 4, 2022 · 3 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@harshad16
Copy link
Member

harshad16 commented Apr 4, 2022 •
edited
Loading

Describe the bug

Noticing that pods are not getting terminated after their action has been completed.

https://console-openshift-console.apps.ocp4.prod.psi.redhat.com/k8s/ns/thoth-frontend-stage/pods
https://console-openshift-console.apps.ocp4.prod.psi.redhat.com/k8s/ns/thoth-frontend-stage/cronjobs/graph-refresh/jobs

thoth-application/graph-refresh/base/cronjob.yaml

Line 13 in 251d057

successfulJobsHistoryLimit: 2

Another set of examples:
The job was terminated due to successful execution, however, the pods are still present
https://console-openshift-console.apps.ocp4.prod.psi.redhat.com/k8s/ns/thoth-frontend-stage/pods/graph-refresh-27483880-7c6fb
https://console-openshift-console.apps.ocp4.prod.psi.redhat.com/k8s/ns/thoth-frontend-stage/jobs/graph-refresh-27483880

The various namespaces are reporting resource quota full. ex:
frontend

Seems like once the quota is set and used by pods it is not getting reset.

To Reproduce
Steps to reproduce the behavior:

  1. Follow the links
  2. See the errors

Expected behavior
Quota is freed up for other resources to run in a timing fashion

Additional context

This affects all the stage deployments.
cluster: ocp4-stage and ocp4-test
Thread link: https://chat.google.com/room/AAAAVjnVXFk/NAw7nZ1IKdw
@harshad16 harshad16 added the kind/bug Categorizes issue or PR as related to a bug. label Apr 4, 2022
@sesheta sesheta added the needs-triage Indicates an issue or PR lacks a `triage/...` label and requires one. label Apr 4, 2022
@harshad16
Copy link
Member Author

/triage accepted
/priority important-soon

@sesheta sesheta added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-triage Indicates an issue or PR lacks a `triage/...` label and requires one. labels Apr 4, 2022
@harshad16
Copy link
Member Author

PSI ticket: https://redhat.service-now.com/surl.do?n=RITM1143044

@harshad16 harshad16 mentioned this issue Apr 4, 2022
Closed
@harshad16
Copy link
Member Author

This is fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@harshad16 @sesheta