Allow VERIFY_TIMEOUT override for mcpchecker-eval - #84653
Conversation
openshift/openshift-mcp-server#463 restores each eval task's original (fast) setup/verify timeout as the default and moves the OpenShift-specific increase behind a VERIFY_TIMEOUT env var, so generic/upstream CI is not penalized by timeouts only needed for OpenShift's slower SCC/PSS admission and cloud PVC binding. Declare VERIFY_TIMEOUT on the openshift-mcp-server-mcpchecker-eval ref (default empty, so scripts keep their own defaults unless overridden) and set it to 300s on the mcp-server core-suite tests that run against a live OCP cluster: mcpchecker-eval-google and mcpchecker-eval-anthropic on main and release-0.5, and periodic-mcpchecker-eval on release-0.5. Co-authored-by: Cursor <cursoragent@cursor.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (3)
Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review. WalkthroughThe MCP checker evaluation step now supports an optional verification timeout override. Main and release configurations set the timeout to 300 seconds for Google, Anthropic, and periodic evaluations. ChangesMCP verification timeout
Estimated code review effort: 1 (Trivial) | ~2 minutes Merge Risk: ⚪ Minimal · up to This change adds a configurable 300-second verification timeout to the specified MCP evaluation jobs while retaining empty defaults for other uses. No concrete merge-blocking risk remains. Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cajieh The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
cajieh
changed the title
|
[REHEARSALNOTIFIER]
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals. Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@cajieh: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
openshift/openshift-mcp-server#463 restores each eval task's original (fast) setup/verify timeout as the default and moves the OpenShift-specific increase behind a
VERIFY_TIMEOUTenv var, so generic/upstream CI is not penalized by timeouts only needed for OpenShift's slower SCC/PSS admission and cloud PVC binding.This PR:
VERIFY_TIMEOUTon theopenshift-mcp-server-mcpchecker-evalref (default empty, so scripts keep their own defaults unless overridden).VERIFY_TIMEOUT: 300son the mcp-server core-suite tests that run against a live OCP cluster:mcpchecker-eval-googleandmcpchecker-eval-anthropiconmainandrelease-0.5, andperiodic-mcpchecker-evalonrelease-0.5.Related: openshift/openshift-mcp-server#463
Made with Cursor
Summary by CodeRabbit
This PR lets
mcpchecker-evaltasks overrideVERIFY_TIMEOUT.VERIFY_TIMEOUTsetting to the shared task reference.VERIFY_TIMEOUT: 300sfor live OpenShift core-suite tests onmainandrelease-0.5.