Skip to content

chore: clarify conformance reports#6699

Merged
LesnyRumcajs merged 1 commit intomainfrom
update-conformance-report-wording
Mar 6, 2026
Merged

chore: clarify conformance reports#6699
LesnyRumcajs merged 1 commit intomainfrom
update-conformance-report-wording

Conversation

@LesnyRumcajs
Copy link
Member

@LesnyRumcajs LesnyRumcajs commented Mar 6, 2026

Summary of changes

This came up yesterday during a discussion with FilOz. This should make it more appealing and clear.

Changes introduced in this pull request:

  • changed wording a bit, explained the current status,
  • changed ❌ to ➖ , the former kind of indicates fiasco
  • added link to docs

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Outside contributions

  • I have read and agree to the CONTRIBUTING document.
  • I have read and agree to the AI Policy document. I understand that failure to comply with the guidelines will lead to rejection of the pull request.

Summary by CodeRabbit

Documentation

  • Enhanced API conformance reports with a Legend section clarifying status labels (Tested, Not tested, N/A)
  • Reclassified test status indicators for improved clarity
  • Added clear distinction for Forest-specific methods marked as not applicable
  • Improved presentation and consistency of conformance testing information

@LesnyRumcajs LesnyRumcajs requested a review from a team as a code owner March 6, 2026 08:36
@LesnyRumcajs LesnyRumcajs requested review from akaladarshi and hanabi1224 and removed request for a team March 6, 2026 08:36
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 6, 2026

Walkthrough

The PR updates API conformance reporting by standardizing status legends across documentation and the script that generates reports. It clarifies status semantics (✅ = Tested, ➖ = Not tested, N/A = Forest-specific) and reclassifies numerous Filecoin and F3 methods from ❌ to ➖ status, reflecting refined testing categorization without implementation changes.

Changes

Cohort / File(s) Summary
API Conformance Reports
docs/docs/users/reports/api_conformance/report_2026-01-20.md, docs/docs/users/reports/api_conformance/report_2026-02-01.md
Added Legend sections clarifying status labels and conformance testing notes. Reclassified numerous Filecoin and F3 method entries from ❌ (not conformance/not tested) to ➖ (not tested). Added Forest-specific methods marked as N/A.
Report Generation Script
scripts/tests/api_compare/convert_report_to_markdown.sh
Updated script to generate Legend section in reports and changed non-Forest-specific status indicator from ❌ to ➖ in generated rows. Table structure and logic flow remain unchanged.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

Suggested reviewers

  • akaladarshi
  • hanabi1224
🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'chore: clarify conformance reports' directly aligns with the main changes: updating documentation to clarify conformance report presentation, adjusting status indicators, and adding legend sections.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch update-conformance-report-wording

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
scripts/tests/api_compare/convert_report_to_markdown.sh (1)

36-41: Minor inconsistency between legend and generated output.

The legend on line 27 describes N/A as "Forest-specific method (not in Lotus)", but the actual generated output on line 38 renders as N/A (Forest-specific). Consider updating the legend to match the exact output format for consistency:

📝 Suggested fix
-    echo "- N/A: Forest-specific method (not in Lotus)"
+    echo "- N/A (Forest-specific): Method only exists in Forest (not in Lotus)"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/tests/api_compare/convert_report_to_markdown.sh` around lines 36 -
41, The legend text and the jq output are inconsistent: the jq filter in the
convert_report_to_markdown.sh script emits "N/A (Forest-specific)" for
Forest-only methods (see the branch that checks startswith("Forest.") in the jq
expression), so update the legend line that currently says 'N/A: Forest-specific
method (not in Lotus)' to exactly match the generated output "N/A
(Forest-specific)" so the legend and the output are identical.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@scripts/tests/api_compare/convert_report_to_markdown.sh`:
- Around line 36-41: The legend text and the jq output are inconsistent: the jq
filter in the convert_report_to_markdown.sh script emits "N/A (Forest-specific)"
for Forest-only methods (see the branch that checks startswith("Forest.") in the
jq expression), so update the legend line that currently says 'N/A:
Forest-specific method (not in Lotus)' to exactly match the generated output
"N/A (Forest-specific)" so the legend and the output are identical.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 18b2d54c-e277-407b-a5e6-49241ae985c2

📥 Commits

Reviewing files that changed from the base of the PR and between b0bfaaa and 4887a27.

📒 Files selected for processing (3)
  • docs/docs/users/reports/api_conformance/report_2026-01-20.md
  • docs/docs/users/reports/api_conformance/report_2026-02-01.md
  • scripts/tests/api_compare/convert_report_to_markdown.sh

@LesnyRumcajs LesnyRumcajs enabled auto-merge March 6, 2026 08:56
@LesnyRumcajs LesnyRumcajs added this pull request to the merge queue Mar 6, 2026
Merged via the queue into main with commit 6af57c1 Mar 6, 2026
45 checks passed
@LesnyRumcajs LesnyRumcajs deleted the update-conformance-report-wording branch March 6, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants