AUTOSCALE-692: remove default debug log level from karpenter container by maxcao13 · Pull Request #8561 · openshift/hypershift

maxcao13 · 2026-05-20T23:16:49Z

What this PR does / why we need it:

Debug was a leftover from early development. It floods CloudWatch with unnecessary volume and increases customer costs.

Assisted-by: Claude Opus 4 (via Cursor)

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/AUTOSCALE-692

Special notes for your reviewer:

Checklist:

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

Summary by CodeRabbit

Chores
- Reduced Karpenter controller logging verbosity from debug to info to lower log noise while retaining important operational messages.
Tests
- Simplified end-to-end Karpenter test flow by removing log-based version checks; test now focuses on CRD validation and core plumbing checks.

openshift-merge-bot · 2026-05-20T23:16:52Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci-robot · 2026-05-20T23:16:54Z

@maxcao13: This pull request references AUTOSCALE-692 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Debug was a leftover from early development. It floods CloudWatch with unnecessary volume and increases customer costs.

Assisted-by: Claude Opus 4 (via Cursor)

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/AUTOSCALE-692

Special notes for your reviewer:

Checklist:

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-05-20T23:17:04Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 8140a720-cdd5-426c-8288-9aa860c162f3

📥 Commits

Reviewing files that changed from the base of the PR and between d62f07d and 56123a8.

📒 Files selected for processing (1)

test/e2e/karpenter_test.go

💤 Files with no reviewable changes (1)

test/e2e/karpenter_test.go

📝 Walkthrough

Walkthrough

This PR lowers Karpenter controller logging by changing the Deployment container arg --log-level from debug to info, and it removes Karpenter pod log-scanning from the E2E test: the test no longer creates a kube client to list pods or scan logs for a version string, and the checkKarpenterVersionInLogs helper and its related imports were deleted.

🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Test Structure And Quality	⚠️ Warning	testKarpenterPlumbing violates single responsibility by testing 8+ unrelated behaviors (metrics, vCPUs, CRDs, NodeClass validation, deletion/modification protection, conditions) in one test block.	Split testKarpenterPlumbing into focused single-responsibility tests, e.g., testKarpenterMetrics, testKarpenterCRDs, testNodeClassProtection, testAutoNodeCondition, following patterns established in testARM64Provisioning.

✅ Passed checks (11 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the main change: switching Karpenter's log level from debug to info, which aligns with both the deployment YAML modification and the test update to remove version log verification.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PR modifies test/e2e/karpenter_test.go which uses Go stdlib testing (t.Run), not Ginkgo. The check for stable Ginkgo test names is not applicable here.
Microshift Test Compatibility	✅ Passed	PR removes existing tests rather than adding new ones. Revert deletes `checkKarpenterVersionInLogs` function and 67 lines of test code. Check only applies when new tests are added.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PR does not add new Ginkgo e2e tests. test/e2e/karpenter_test.go uses standard Go testing with t.Run(), not Ginkgo patterns. Custom check applies only to new Ginkgo tests.
Topology-Aware Scheduling Compatibility	✅ Passed	No topology-aware scheduling constraints introduced. Changes only adjust Karpenter log levels and update tests. Deployment has no affinity, toleration, nodeSelector, or topology constraints.
Ote Binary Stdout Contract	✅ Passed	Removed code was pod log scanning inside test cases, not process-level code. Test suite uses zap logging configured to stderr with no problematic stdout writes in TestMain.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PR removes test code and changes a YAML deployment file; no new Ginkgo e2e tests are added. Check only applies when new Ginkgo tests (It/Describe/Context/When) are added.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

codecov · 2026-05-20T23:25:53Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 40.40%. Comparing base (d86f3d4) to head (56123a8).
⚠️ Report is 17 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8561      +/-   ##
==========================================
+ Coverage   40.34%   40.40%   +0.06%     
==========================================
  Files         755      755              
  Lines       93167    93235      +68     
==========================================
+ Hits        37587    37675      +88     
+ Misses      52877    52858      -19     
+ Partials     2703     2702       -1

see 3 files with indirect coverage changes

Flag	Coverage Δ
cmd-support	`34.44% <ø> (+0.13%)`	⬆️
cpo-hostedcontrolplane	`41.76% <ø> (ø)`
cpo-other	`40.31% <ø> (+0.17%)`	⬆️
hypershift-operator	`50.72% <ø> (ø)`
other	`31.54% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

joshbranham · 2026-05-20T23:34:44Z

          image: aws-karpenter-provider-aws # replaced by payload
          args:
-            - "--log-level=debug"
+            - "--log-level=info"


Should we just omit the flag and let Karpenter use its default?

Fine with me, 👍

The deployment asset hardcoded --log-level=debug, a leftover from early development. This floods CloudWatch and increases customer costs. Remove the flag entirely and let Karpenter use its default (info). Signed-off-by: Max Cao <macao@redhat.com> Assisted-by: Claude Opus 4 (via Cursor) Co-authored-by: Cursor <cursoragent@cursor.com>

joshbranham · 2026-05-21T00:38:12Z

Can we check that it looks good in a prow run (ie logs flow as info)?

/lgtm

openshift-merge-bot · 2026-05-21T00:38:26Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

cwbotbot · 2026-05-21T01:00:02Z

Test Results

e2e-aws

Status: ✅ PASS
Started: 2026-05-21T23:36:25Z
View Job
View Job History

e2e-aks

Status: ✅ PASS
Started: 2026-05-21T15:46:26Z
View Job
View Job History

hypershift-jira-solve-ci · 2026-05-21T02:27:27Z

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2057259763279859712 | Cost: $2.60012325 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

maxcao13 · 2026-05-21T03:12:14Z

Seems like it's working: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_hypershift/8561/pull-ci-openshift-hypershift-main-e2e-aws/2057259763279859712/artifacts/e2e-aws/hypershift-aws-run-e2e-nested/artifacts/TestKarpenter/namespaces/e2e-clusters-rcqtn-karpenter-gjg6q/core/pods/logs/karpenter-7d57544986-2gt8q-karpenter.log

But it also seems like it removes the version log. Not sure how we feel about that. Is that okay from SRE standpoint?

So either we ignore it, and remove the test, or we need to change our karpenter fork code and backport.

joshbranham · 2026-05-21T03:55:34Z

Seems like it's working: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_hypershift/8561/pull-ci-openshift-hypershift-main-e2e-aws/2057259763279859712/artifacts/e2e-aws/hypershift-aws-run-e2e-nested/artifacts/TestKarpenter/namespaces/e2e-clusters-rcqtn-karpenter-gjg6q/core/pods/logs/karpenter-7d57544986-2gt8q-karpenter.log

But it also seems like it removes the version log. Not sure how we feel about that. Is that okay from SRE standpoint?

So either we ignore it, and remove the test, or we need to change our karpenter fork code and backport.

I'm not sure why we would care about the karpenter version in the logs, or why we have a test for it, so all good from my perspective.

enxebre · 2026-05-21T07:08:55Z

this might be legit https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_hypershift/8561/pull-ci-openshift-hypershift-main-e2e-aws/2057259763279859712
fwiw I'm fine dropping that e2e entirely

/approve

openshift-ci · 2026-05-21T07:09:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enxebre, maxcao13

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Reverts openshift@a58ab5e. We need to do this since we switched default log level of karpenter to INFO, which removes version log. We will move this verification to openshift/autoscale-tests. Signed-off-by: Max Cao <macao@redhat.com>

maxcao13 · 2026-05-21T15:40:33Z

/test e2e-aws-autonode

joshbranham · 2026-05-21T15:45:28Z

/lgtm

openshift-merge-bot · 2026-05-21T15:45:45Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

hypershift-jira-solve-ci · 2026-05-21T19:00:54Z

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2057488157519122432 | Cost: $3.0523737500000006 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

maxcao13 · 2026-05-21T19:44:32Z

/retest-required

hypershift-jira-solve-ci · 2026-05-21T19:54:28Z

That background download also completed successfully (the e2e nested step build log for the e2e-aws-4-22 job). I had already gathered all the necessary evidence from the build logs and JUnit XML files to complete the analysis. The full report above covers both job failures comprehensively.

maxcao13 · 2026-05-21T23:36:14Z

/test e2e-aws

maxcao13 · 2026-05-22T03:55:27Z

/verified by e2e-aws

openshift-ci-robot · 2026-05-22T03:55:39Z

@maxcao13: This PR has been marked as verified by e2e-aws.

Details

In response to this:

/verified by e2e-aws

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-05-22T05:10:45Z

@maxcao13: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 20, 2026

openshift-ci Bot added the do-not-merge/needs-area label May 20, 2026

openshift-ci Bot requested review from Nirshal and jparrill May 20, 2026 23:17

openshift-ci Bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release and removed do-not-merge/needs-area labels May 20, 2026

joshbranham reviewed May 20, 2026

View reviewed changes

maxcao13 force-pushed the AUTOSCALE-692/karpenter-log-level-info branch from bcd2b12 to d62f07d Compare May 21, 2026 00:29

maxcao13 changed the title ~~AUTOSCALE-692: change default log level from debug to info~~ AUTOSCALE-692: remove default log level from karpenter container May 21, 2026

maxcao13 changed the title ~~AUTOSCALE-692: remove default log level from karpenter container~~ AUTOSCALE-692: remove default debug log level from karpenter container May 21, 2026

openshift-ci Bot assigned joshbranham May 21, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label May 21, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 21, 2026

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label May 21, 2026

openshift-ci Bot added the area/testing Indicates the PR includes changes for e2e testing label May 21, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label May 21, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label May 22, 2026

openshift-merge-bot Bot merged commit d24af10 into openshift:main May 22, 2026
44 checks passed

Conversation

maxcao13 commented May 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Summary by CodeRabbit

Uh oh!

openshift-merge-bot Bot commented May 20, 2026

Uh oh!

openshift-ci-robot commented May 20, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

coderabbitai Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

❌ Failed checks (1 warning)

Uh oh!

codecov Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joshbranham May 20, 2026

Choose a reason for hiding this comment

Uh oh!

maxcao13 May 20, 2026

Choose a reason for hiding this comment

Uh oh!

joshbranham commented May 21, 2026

Uh oh!

openshift-merge-bot Bot commented May 21, 2026

Uh oh!

cwbotbot commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

e2e-aws

e2e-aks

Uh oh!

hypershift-jira-solve-ci Bot commented May 21, 2026

AI Test Failure Analysis

Uh oh!

maxcao13 commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshbranham commented May 21, 2026

Uh oh!

enxebre commented May 21, 2026

Uh oh!

openshift-ci Bot commented May 21, 2026

Uh oh!

maxcao13 commented May 21, 2026

Uh oh!

joshbranham commented May 21, 2026

Uh oh!

openshift-merge-bot Bot commented May 21, 2026

Uh oh!

hypershift-jira-solve-ci Bot commented May 21, 2026

AI Test Failure Analysis

Uh oh!

maxcao13 commented May 21, 2026

Uh oh!

hypershift-jira-solve-ci Bot commented May 21, 2026

Uh oh!

maxcao13 commented May 21, 2026

Uh oh!

maxcao13 commented May 22, 2026

Uh oh!

openshift-ci-robot commented May 22, 2026

Uh oh!

openshift-ci Bot commented May 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

maxcao13 commented May 20, 2026 •

edited by coderabbitai Bot

Loading

openshift-ci-robot commented May 20, 2026 •

edited by openshift-ci Bot

Loading

coderabbitai Bot commented May 20, 2026 •

edited

Loading

codecov Bot commented May 20, 2026 •

edited

Loading

cwbotbot commented May 21, 2026 •

edited

Loading

maxcao13 commented May 21, 2026 •

edited

Loading