Skip to content

check-issue: require source/target implementations for [Rule] checks and cross-validate against code #264

@isPANN

Description

@isPANN

Problem

The check-issue skill has gaps in both [Rule] and [Model] issue evaluation:

1. [Rule] Missing prerequisite check: source and target must be implemented

In Rule Check 1 (Usefulness), when pred show <source> or pred show <target> fails, the skill only emits a Warn ("unknown problem — may be a new model"). However, if the source or target problem is not yet implemented in the main branch, the reduction rule cannot be implemented — there's no code to reduce from/to.

Current behavior: Warn and continue with all other checks.
Expected behavior: Fail with a clear message like "Source problem X is not implemented in the codebase. The corresponding [Model] issue must be merged first before this [Rule] can be evaluated." This should be a prerequisite gate that runs before all other checks — if either problem is missing, skip the remaining checks and report the blocker.

2. [Rule] Correctness check should cross-validate issue content against code

In Rule Check 3 (Correctness), the skill only verifies literature references (paper titles, DOIs, claimed theorems). It does not check whether the issue's description of the source and target problems is consistent with their actual implementations in the codebase.

Missing checks:

  • Read the actual Problem impl for source and target (e.g., dims(), evaluate(), field names, variant params)
  • Verify that the issue's "Reduction Algorithm" references fields/methods that actually exist on the source and target types
  • Verify that the "Size Overhead" table uses metric names that match the target problem's actual getter methods (already partially done in Check 4c via pred show --json → size_fields, but should also be part of correctness)
  • Check that the issue's description of how the source/target problems work matches the code (e.g., if the issue says "MIS maximizes vertex count" but the code uses weighted sums, that's a correctness error)

3. [Model] No orphan node prevention

In Model Check 1 (Usefulness), the skill checks the Motivation text for mentions of planned reductions, but only as a Warn and without any concrete verification. This means a [Model] issue can pass all checks and get implemented, only to become an orphan node in the reduction graph — a problem with no reduction rules connecting it to anything else.

This is especially problematic given the new Rule prerequisite gate (#1 above): if [Rule] issues require both problems to be implemented first, then we need [Model] issues to prove upfront that they will actually be connected.

Current behavior: Warn if Motivation doesn't mention reductions.
Expected behavior: Verify that at least one [Rule] issue exists on GitHub (open or closed) that references this model as source or target. If none found → Fail with label Orphan.

Proposed Changes

Add Rule Check 0: Prerequisite Gate (new, before all others)

## Rule Check 0: Prerequisites (fail label: `Blocked`)

1. Run `pred show <source> --json` and `pred show <target> --json`
2. If either fails → **Fail** with label `Blocked`
   - Message: "Cannot evaluate: `<problem>` is not implemented. Merge the [Model] issue first."
   - Skip all remaining checks
3. If both succeed → store the JSON output for use in subsequent checks

Enhance Rule Check 3: Code Cross-Validation (add step 3e)

### 3e: Cross-validate Against Code Implementation

Using the `pred show` JSON from Check 0:

1. Verify the issue's description of source problem matches code:
   - Field names, variable semantics, objective direction
2. Verify the issue's description of target problem matches code:
   - Field names, variable semantics, objective direction
3. Verify the reduction algorithm references valid fields/methods
4. If any mismatch found → **Fail** with specifics

Enhance Model Check 1: Orphan Node Prevention (add step 1b)

### 1b: Verify Planned Reductions Exist (fail label: `Orphan`)

1. Parse the problem name from the issue
2. Search GitHub for `[Rule]` issues that reference this problem name (as source or target):
   ```bash
   gh issue list --search "[Rule] <problem-name> in:title,body" --json number,title --limit 20
  1. Also check existing reduction rules in code:
    pred show <problem-name> --json  # check "reductions" field if problem already exists
  2. Decision:
    • At least one [Rule] issue (open or closed) references this modelPass
    • Problem already exists in code with existing reductionsPass (adding a variant)
    • No [Rule] issues found AND no existing reductionsFail with label Orphan
      • Message: "No [Rule] issues reference this problem. Create at least one [Rule] issue (as source or target) before this model can be approved."

This creates a **mutual dependency** that ensures the reduction graph stays connected:
- `[Rule]` issues require both source and target models to be implemented → prevents rules referencing nonexistent problems
- `[Model]` issues require at least one `[Rule]` issue to exist → prevents orphan nodes

The intended workflow becomes:
1. Open `[Model]` issue + at least one `[Rule]` issue referencing it (can be done together)
2. `check-issue` on `[Model]` → passes orphan check because [Rule] issue exists
3. Implement the model via `issue-to-pr`
4. `check-issue` on `[Rule]` → passes prerequisite check because model is now in code
5. Implement the rule via `issue-to-pr`

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions