-
Notifications
You must be signed in to change notification settings - Fork 0
feat: add report issue system for specs and implementations #3285
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| name: Report Issue | ||
| description: Report an issue with an existing plot spec or implementation | ||
| title: "Report: " | ||
| labels: ["report-pending"] | ||
| body: | ||
| - type: markdown | ||
| attributes: | ||
| value: | | ||
| ## Report an Issue | ||
|
|
||
| Found a problem with an existing plot? Let us know! | ||
|
|
||
| Our AI will validate and structure your report, then queue it for maintainer review. | ||
|
|
||
| - type: input | ||
| id: spec_id | ||
| attributes: | ||
| label: Specification ID | ||
| description: "The spec ID of the affected plot" | ||
| placeholder: "e.g., scatter-basic, qrcode-basic, heatmap-correlation" | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: dropdown | ||
| id: target | ||
| attributes: | ||
| label: What has the issue? | ||
| description: "Is this an issue with the specification itself or a specific library implementation?" | ||
| options: | ||
| - Specification (affects all libraries) | ||
| - Implementation (specific library) | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: dropdown | ||
| id: library | ||
| attributes: | ||
| label: Library (only for implementation issues) | ||
| description: "Which library implementation has the issue? Select N/A for spec issues." | ||
| options: | ||
| - N/A (spec issue) | ||
| - matplotlib | ||
| - seaborn | ||
| - plotly | ||
| - bokeh | ||
| - altair | ||
| - plotnine | ||
| - pygal | ||
| - highcharts | ||
| - letsplot | ||
| validations: | ||
| required: false | ||
|
|
||
| - type: dropdown | ||
| id: category | ||
| attributes: | ||
| label: Issue Category | ||
| description: "What type of issue is this?" | ||
| options: | ||
| - Visual (ugly, unclear, hard to read) | ||
| - Data (unrealistic values, inappropriate context) | ||
| - Functional (doesn't work as expected, e.g., QR code not scannable) | ||
| - Other | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: description | ||
| attributes: | ||
| label: Issue Description | ||
| description: "Describe the issue. What's wrong? What should it look like instead?" | ||
| placeholder: | | ||
| Example: The QR code generated looks like a QR code but cannot be scanned by any QR reader app. | ||
|
|
||
| Expected: The QR code should be a valid, scannable code that links to the specified URL. | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: additional | ||
| attributes: | ||
| label: Additional Context (optional) | ||
| description: "Screenshots, comparison images, or other helpful information" | ||
| validations: | ||
| required: false | ||
2 changes: 1 addition & 1 deletion
2
.github/ISSUE_TEMPLATE/plot-request.yml → .github/ISSUE_TEMPLATE/spec-request.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
.github/ISSUE_TEMPLATE/plot-update.yml → .github/ISSUE_TEMPLATE/spec-update.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| name: "Report: Validate" | ||
| run-name: "Validate Report: ${{ github.event.issue.title }}" | ||
|
|
||
| # Validates and structures user-submitted issue reports | ||
| # Flow: | ||
| # 1. User submits report → report-pending label auto-added | ||
| # 2. AI validates spec/impl exists, analyzes issue, posts structured comment | ||
| # 3. Labels updated: report-validated + category + report:spec/impl | ||
|
|
||
| on: | ||
| issues: | ||
| types: [labeled] | ||
|
|
||
| concurrency: | ||
| group: report-validate-${{ github.event.issue.number }} | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| validate: | ||
| if: github.event.label.name == 'report-pending' | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| issues: write | ||
|
|
||
| steps: | ||
| - name: Check if already validated | ||
| id: check | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| # Check if already has report-validated label | ||
| LABELS=$(gh issue view ${{ github.event.issue.number }} --json labels -q '.labels[].name' | tr '\n' ' ') | ||
| if echo "$LABELS" | grep -q "report-validated"; then | ||
| echo "::notice::Skipping: Issue already validated" | ||
| echo "should_run=false" >> $GITHUB_OUTPUT | ||
| exit 0 | ||
| fi | ||
| echo "should_run=true" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: Checkout repository | ||
| if: steps.check.outputs.should_run == 'true' | ||
| uses: actions/checkout@v6 | ||
|
|
||
| - name: React with eyes emoji | ||
| if: steps.check.outputs.should_run == 'true' | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh api repos/${{ github.repository }}/issues/${{ github.event.issue.number }}/reactions \ | ||
| -f content=eyes | ||
|
|
||
| - name: Validate with Claude | ||
| if: steps.check.outputs.should_run == 'true' | ||
| id: validate | ||
| continue-on-error: true | ||
| timeout-minutes: 10 | ||
| uses: anthropics/claude-code-action@v1 | ||
| with: | ||
| claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} | ||
| claude_args: "--model sonnet" | ||
| prompt: | | ||
| ## Task: Validate Issue Report | ||
|
|
||
| You are validating a user-submitted issue report for pyplots. | ||
|
|
||
| ### Issue Details | ||
| - **Title:** ${{ github.event.issue.title }} | ||
| - **Number:** #${{ github.event.issue.number }} | ||
| - **Author:** ${{ github.event.issue.user.login }} | ||
| - **Body:** | ||
| ``` | ||
| ${{ github.event.issue.body }} | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Instructions | ||
|
|
||
| 1. **Read the prompt:** `prompts/workflow-prompts/report-analysis.md` | ||
|
|
||
| 2. **Parse the issue body** to extract: | ||
| - spec_id (from "Specification ID" field) | ||
| - target (Specification or Implementation) | ||
| - library (if implementation) | ||
| - category (Visual/Data/Functional/Other) | ||
| - description | ||
|
|
||
| 3. **Validate the spec exists:** | ||
| ```bash | ||
| ls plots/{spec_id}/ | ||
| ``` | ||
| If NOT found: | ||
| - Post comment: "Spec `{spec_id}` not found. Please check the ID." | ||
| - Remove `report-pending` label | ||
| - Close issue | ||
| - STOP | ||
|
|
||
| 4. **If implementation issue, validate the library exists:** | ||
| ```bash | ||
| ls plots/{spec_id}/implementations/{library}.py | ||
| ``` | ||
| If NOT found: | ||
| - Post comment: "Implementation `{library}` not found for `{spec_id}`." | ||
| - Remove `report-pending` label | ||
| - Close issue | ||
| - STOP | ||
|
|
||
| 5. **Read relevant files:** | ||
| - `plots/{spec_id}/specification.md` | ||
| - `plots/{spec_id}/metadata/{library}.yaml` (if impl issue) | ||
|
|
||
| 6. **Post structured analysis comment** following the format in the prompt file. | ||
|
|
||
| 7. **Update issue title:** | ||
| ```bash | ||
| gh issue edit ${{ github.event.issue.number }} --title "[{spec_id}] {brief description}" | ||
| ``` | ||
|
|
||
| 8. **Update labels:** | ||
| ```bash | ||
| # Remove pending, add validated | ||
| gh issue edit ${{ github.event.issue.number }} --remove-label "report-pending" --add-label "report-validated" | ||
|
|
||
| # Add target label | ||
| gh issue edit ${{ github.event.issue.number }} --add-label "report:spec" | ||
| # OR for impl: | ||
| gh issue edit ${{ github.event.issue.number }} --add-label "report:impl" --add-label "report:impl:{library}" | ||
|
|
||
| # Add category label (map from dropdown value) | ||
| # Visual → category:visual | ||
| # Data → category:data | ||
| # Functional → category:functional | ||
| # Other → category:other | ||
| gh issue edit ${{ github.event.issue.number }} --add-label "category:{category}" | ||
| ``` | ||
|
|
||
| ### Important | ||
| - Do NOT add the `approved` label | ||
| - Do NOT trigger any fixes | ||
| - Keep the analysis comment concise but informative | ||
|
|
||
| - name: Add reaction on success | ||
| if: steps.check.outputs.should_run == 'true' && steps.validate.outcome == 'success' | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh api repos/${{ github.repository }}/issues/${{ github.event.issue.number }}/reactions \ | ||
| -f content=rocket | ||
|
|
||
| - name: Handle failure | ||
| if: steps.check.outputs.should_run == 'true' && steps.validate.outcome == 'failure' | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh issue comment ${{ github.event.issue.number }} --body "## :x: Validation Failed | ||
|
|
||
| The AI validation workflow encountered an error. A maintainer will review this manually. | ||
|
|
||
| --- | ||
| :robot: *[report-validate](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})*" | ||
|
|
||
| gh api repos/${{ github.repository }}/issues/${{ github.event.issue.number }}/reactions \ | ||
| -f content=confused |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The report-issue.yml template is missing a title field. Unlike spec-update.yml which provides a title template, this template relies on users to create their own titles which may lead to inconsistent issue titles. Consider adding a title field similar to spec-update.yml to ensure consistency, for example:
title: "Report Issue: "ortitle: "[SPEC-ID] ". The workflow will update the title later with the proper format, but having a template helps guide users.