Skip to content

Improve feasibility checks for interactive/capability-specific specs #2842

@MarkusNeusinger

Description

@MarkusNeusinger

Problem

The line-interactive spec requires true interactivity (hover tooltips, zoom, pan, range selection). However, implementations were generated and approved for matplotlib, seaborn, and plotnine - libraries that can only output static PNG files.

These implementations "faked" interactivity by:

  • Using mplcursors (only works in live display, not in saved PNG)
  • Adding static annotations to "simulate" hover states
  • Adding text hints like "use scroll to zoom" that don't actually work in PNG output

This passed AI quality review with scores of 85-88/100.

Root Cause

  1. impl-generate prompts don't check if a spec's requirements are feasible for the target library
  2. quality-evaluator prompts don't verify that claimed features actually work in the saved output format
  3. No automatic not-feasible detection for capability mismatches

Affected Specs

Any spec requiring features that static libraries can't provide:

  • line-interactive - hover, zoom, pan (matplotlib/seaborn/plotnine deleted)
  • Potentially others with: animations, hover tooltips, click interactions, etc.

Library Capability Matrix

Feature matplotlib seaborn plotnine plotly bokeh altair highcharts letsplot pygal
Static PNG
Interactive HTML ⚠️
Hover tooltips ⚠️
Zoom/Pan
Animations

Proposed Fixes

1. Update prompts/library/*.md with capability declarations

Add to each library prompt:

## Library Capabilities
- Output formats: PNG only (no HTML)
- Interactivity: None (static output)
- NOT FEASIBLE for specs requiring: hover, zoom, pan, animations, click handlers

2. Update prompts/plot-generator.md with feasibility check

Add pre-generation step:

## Before Generating
1. Check if spec requirements match library capabilities
2. If spec requires interactivity but library only outputs static images → STOP
3. Return "NOT_FEASIBLE: {library} cannot provide {required_feature}"

3. Update prompts/quality-evaluator.md with output verification

Add to quality criteria:

## Auto-Reject: Fake Features
- AR-08: FAKE_INTERACTIVITY - Code claims interactivity but output is static
  - Check: If spec requires hover/zoom → verify output is HTML, not just PNG
  - Check: If code uses interactive libraries (mplcursors) but saves as PNG → REJECT

4. Add spec tagging for required capabilities

In specification.yaml:

requires:
  - interactive_output  # Needs HTML, not just PNG
  - hover_tooltips
  - zoom_pan

Then impl-generate can auto-skip incompatible libraries.

Immediate Actions Taken

  • Deleted line-interactive implementations for matplotlib, seaborn, plotnine
  • Deleted corresponding GCS images and metadata

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinginfrastructureWorkflow, backend, or frontend issue

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions