feat: Add automated API documentation generation pipeline #72

rbren · 2025-11-03T19:04:02Z

I have read and reviewed the documentation changes to the best of my ability.
If the change is significant, I have run the documentation site locally and confirmed it renders as expected.

Summary of changes

This PR introduces a comprehensive automated API documentation generation pipeline that creates API reference documentation from the OpenHands SDK source code.

🚀 Features

Core Pipeline

Automated SDK Integration: Clones/pulls the latest software-agent-sdk repository
Sphinx-powered Generation: Uses Sphinx with autodoc, napoleon, and markdown builder extensions
Mintlify-ready Output: Generates clean Markdown files with proper frontmatter for Mintlify integration
Module Organization: Properly organizes documentation by SDK module structure

Scripts & Tools

Python Script (scripts/generate-api-docs.py): Full-featured generation script with comprehensive error handling
Shell Wrapper (scripts/generate-api-docs.sh): Convenient shell script for quick execution
Idempotent Design: Safe to run repeatedly, with clean and incremental build options

Configuration & Setup

Sphinx Configuration: Complete conf.py with optimized settings for API documentation
Navigation Integration: Auto-generates mint.json configuration snippets for easy integration
Dependency Management: Clear documentation of required packages and setup instructions

📁 Generated Structure

api-reference/
├── index.md                    # Main API reference landing page
├── sdk.agent.md               # Agent module documentation
├── sdk.conversation.md        # Conversation module documentation
├── sdk.llm.md                 # LLM module documentation
├── sdk.tool.md                # Tool module documentation
├── sdk.workspace.md           # Workspace module documentation
└── ... (110+ module files)    # Complete SDK coverage

🛠 Usage

Quick Start

# Install dependencies
pip install sphinx sphinx-markdown-builder myst-parser

# Generate documentation
./scripts/generate-api-docs.sh

# Or use Python script directly
python scripts/generate-api-docs.py --verbose

Integration

Copy configuration from scripts/mint-config-snippet.json
Add to your docs.json navigation structure
Generated docs are ready for Mintlify deployment

🔧 Technical Details

Sphinx Extensions: autodoc, napoleon, autosummary, viewcode, intersphinx, myst-parser
Output Format: Clean Markdown with minimal Sphinx-specific syntax
Error Handling: Comprehensive logging and graceful failure handling
Build Artifacts: Excluded from git but generated docs are committed
Documentation Coverage: 110+ module files covering the entire SDK

📋 Files Added

Scripts

scripts/generate-api-docs.py - Main generation script
scripts/generate-api-docs.sh - Shell wrapper
scripts/README.md - Complete setup and usage guide

Configuration

scripts/sphinx/source/conf.py - Sphinx configuration
scripts/sphinx/source/index.rst - Documentation structure
scripts/mint-config-example.json - Integration example

Generated Documentation

api-reference/ - Complete API reference (110+ files)
scripts/mint-config-snippet.json - Ready-to-use navigation config

Infrastructure

Updated .gitignore - Excludes build artifacts, includes generated docs

This pipeline enables automatic, up-to-date API documentation that stays synchronized with the SDK development, providing developers with comprehensive reference materials in a format optimized for the Mintlify documentation platform.

@rbren can click here to continue refining the PR

- Create comprehensive Python script for generating API docs from OpenHands SDK - Add Sphinx configuration with autodoc, napoleon, and markdown builder - Generate clean Markdown output suitable for Mintlify integration - Include proper module organization and navigation structure - Add shell script wrapper for convenience - Update .gitignore to exclude build artifacts but include generated docs - Provide complete setup instructions and mint.json configuration example Co-authored-by: openhands <openhands@all-hands.dev>

…vigation - Move all API reference docs from api-reference/ to sdk/api-reference/ - Update generation script to output to new location - Integrate API Reference section into docs.json under SDK tab - Position API Reference after Architecture section in navigation - Update all documentation and scripts to reflect new paths - Regenerate mint config snippet with correct paths Co-authored-by: openhands <openhands@all-hands.dev>

- Fix clean_build_artifacts to preserve conf.py and index.rst - Recreate missing conf.py and index.rst files - Ensure pipeline works correctly with --clean option - Maintain proper Sphinx configuration for markdown generation Co-authored-by: openhands <openhands@all-hands.dev>

…d titles - Update sphinx-apidoc to target correct SDK directory (openhands/sdk) - Fix title formatting to use proper module names instead of title case - Generate 110+ individual module documentation files - Preserve essential Sphinx configuration files during clean operations - Add proper Python path configuration for module imports - Maintain existing navigation structure in docs.json

…ction - Fix RST module name replacement to avoid double prefixes - Ensure proper return type annotation for run_sphinx_build method - Successfully generate comprehensive API documentation with: - 110+ individual module documentation files - Rich docstring content extracted from openhands-sdk package - Proper cross-references and type annotations - Clean Markdown format suitable for Mintlify - All files now contain substantial content (20-400+ lines each) - Pipeline generates complete API reference documentation

- Remove outdated mint-config-example.json file - File contained outdated paths (api-reference/ instead of sdk/api-reference/) - Proper configuration is generated in mint-config-snippet.json by the script - Keep scripts directory clean with only essential files

- Enhanced clean_markdown_file method to handle problematic syntax - Fix complex type annotations with asterisks and curly braces - Escape curly braces that might be interpreted as template syntax - Remove unbalanced asterisks that break markdown parsers - Specifically handle model_config lines that were causing parsing issues - Should resolve broken link detection errors in CI

- Add handling for <factory> tags that Mintlify interprets as unclosed HTML - Wrap <factory> tags in backticks to render as inline code - Should resolve 'Expected a closing tag for <factory>' deployment errors - Fixes Mintlify deployment failures with Pydantic field defaults

…ike tags - Add comprehensive handling for <secret-hidden> and other HTML-like patterns - Use regex to escape angle bracket patterns that aren't real HTML tags - Clean up complex type signatures by removing ~ prefixes that confuse parsers - Simplify ~typing., ~annotated_types., ~uuid., ~openhands. patterns - Should resolve remaining 'Could not parse expression with acorn' errors - Improve overall Mintlify compatibility for complex API documentation

…ibility - Add multiline JSON pattern handling in clean_markdown_file - Enhance JSON-like pattern detection for single and multiline cases - Improve handling of complex type signatures and HTML-like tags - Add specific fixes for patterns that cause acorn parsing errors - Should resolve most remaining 'Could not parse expression with acorn' issues

- Wrap multiline JSON example in code blocks to prevent JavaScript parsing - Fix specific pattern in sdk.conversation.md that was causing 'Could not parse expression with acorn' error - This should resolve the final parsing issue preventing successful deployment

- Extract clean method/class names as titles instead of long signatures - Format parameters as readable bullet points below the title - Separate parameters from method descriptions for better readability - Apply formatting to both class constructors and method signatures - Maintain all type information while improving presentation

- Fix unbalanced asterisks in Sphinx-generated parameter lists - Convert '* **Parameters:**' to '**Parameters:**' to avoid markdown parsing issues - Apply same fix to Returns, Raises, and Yields sections - Disable custom signature formatting to rely on Sphinx's native output - Maintain clean, readable documentation while ensuring JavaScript parser compatibility

- Fix unbalanced asterisks in code blocks and quoted sections - Escape standalone asterisks that confuse JavaScript parsers - Fix ClassVar[ConfigDict] patterns with unbalanced asterisks - Remove problematic code block patterns that cause parsing issues - Ensure all markdown is compatible with Mintlify's JavaScript parser

… errors - Convert '* **param_name**' to '- **param_name**' to avoid nested emphasis - Fix unbalanced asterisks in Sphinx-generated parameter documentation - Ensure all parameter lists use consistent dash formatting instead of asterisks - Maintain readability while ensuring JavaScript parser compatibility

- Remove all emphasis/bolding from method signatures and parameter names - Convert **Parameters:** to simple 'Parameters:' text - Remove *property*, *method*, *abstract* emphasis from headers - Fix HTML-like tags by wrapping in backticks - Eliminate all unbalanced asterisks that cause JavaScript parser issues - Maintain readability while ensuring maximum compatibility with Mintlify

…rrors - Remove '(\*,' patterns from class signatures that confuse JavaScript parsers - Clean up escaped asterisks in function signatures - Ensure all parameter lists use clean, parser-friendly syntax - Final fix for remaining acorn parsing errors in API documentation

…ple approach - Replace complex signature-based headers with simple class/method names - Document parameters as plain text bullet points instead of in headers - Remove all emphasis, bolding, and complex markdown patterns - Generate clean, parser-friendly documentation that focuses on content over formatting - Use Sphinx autodoc with minimal post-processing for maximum reliability - Headers now contain only essential names: '### ClassName' and '#### method_name' - All parameter details documented in readable text format below headers

- Replace 110+ individual API reference pages with 9 clean module-level pages - Organize API documentation by major SDK components (agent, conversation, event, etc.) - Simplify navigation structure for better user experience - Remove outdated file references that no longer exist

- Document the parser-friendly design philosophy - Update file structure to show new module-level organization - Highlight key features: simple headers, clean formatting, no parsing errors - Reflect the change from 100+ files to 9 organized module pages

Co-authored-by: Engel Nyst <engel.nyst@gmail.com>

openhands-ai · 2025-11-04T19:52:55Z

Looks like there are a few issues preventing this PR from being merged!

GitHub Actions are failing:
- Sync Documentation and API Reference

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #72 at branch `feature/api-docs-generation-pipeline`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

.github/workflows/sync-docs-and-api.yml

…s-generation-pipeline

- Synced code blocks from examples - Generated API reference documentation Synced from agent-sdk ref: main

enyst · 2025-11-06T14:44:14Z

.github/workflows/sync-docs-and-api.yml

@OpenHands this PR deleted the .github/workflows/sync-docs-code-blocks.yml file: restore it from main branch.

Then extend it to also do generate_api_docs.

Goal: we want to add the generate api docs job to the sync from main branch, because that workflow runs these syncs on schedule and creates a PR if there are changes. That is the behavior we want for this too, instead of only commits wherever.

I'm on it! enyst can track my progress at all-hands.dev

Final update summary (no new commits since my last update)

What I changed:

Restored from main and extended:

Added back .github/workflows/sync-docs-code-blocks.yml and extended it to also run API docs generation (scripts/generate-api-docs.py), installing Sphinx deps as needed.

Workflow now:

Checks out docs and software-agent-sdk (ref input, defaults to main)

Syncs code blocks, generates API docs

Detects changes and opens a PR via peter-evans/create-pull-request

Triggers: schedule (daily 2:00 UTC) and workflow_dispatch

Removed redundant workflow:

Deleted .github/workflows/sync-docs-and-api.yml to avoid direct pushes and duplication. This ensures the sync and API docs updates happen via scheduled PRs, per your request.

Commits pushed to feature/api-docs-generation-pipeline:

beeb0f3 ci: restore sync-docs-code-blocks workflow from main and extend to generate API docs

59fce12 ci: remove redundant sync-docs-and-api workflow in favor of restored PR-based workflow

Checklist against your request:

Restore .github/workflows/sync-docs-code-blocks.yml from main: done

Extend it to also do generate_api_docs: done

Ensure it runs on schedule and creates a PR if there are changes: done

Avoid direct commit behavior and duplication: done (removed the push-based workflow)

Scope check:

Only touched two files:

Added .github/workflows/sync-docs-code-blocks.yml

Removed .github/workflows/sync-docs-and-api.yml

No other changes made.

_{View full conversation}

…nerate API docs\n\n- Restores deleted .github/workflows/sync-docs-code-blocks.yml\n- Adds API docs generation step to run alongside code block sync\n- Uses PR-based update flow on schedule via peter-evans/create-pull-request\n\nCo-authored-by: openhands <openhands@all-hands.dev>

…PR-based workflow\n\n- Deletes .github/workflows/sync-docs-and-api.yml which directly pushed commits\n- Ensures scheduled sync + API docs generation happens via PRs only\n\nCo-authored-by: openhands <openhands@all-hands.dev>

enyst

Looks good to me, thank you!

I got rid of some extraneous changes from automated syncs, but I'm still a little concerned that this kind of thing will make it more difficult to review later, so I think we should merge this ASAP.

mintlify bot deployed to staging November 3, 2025 19:05 View deployment

mintlify bot deployed to staging November 3, 2025 19:10 View deployment

mintlify bot deployed to staging November 3, 2025 19:11 View deployment

mintlify bot deployed to staging November 3, 2025 19:23 View deployment

mintlify bot deployed to staging November 3, 2025 19:38 View deployment

mintlify bot deployed to staging November 3, 2025 19:39 View deployment

openhands-agent added 5 commits November 3, 2025 19:42

mintlify bot deployed to staging November 3, 2025 19:54 View deployment

mintlify bot deployed to staging November 3, 2025 20:00 View deployment

openhands-agent added 7 commits November 3, 2025 20:06

mintlify bot deployed to staging November 3, 2025 20:40 View deployment

mintlify bot deployed to staging November 3, 2025 20:41 View deployment

mintlify bot deployed to staging November 4, 2025 16:58 View deployment

Update scripts/README.md

1d24c50

Co-authored-by: Engel Nyst <engel.nyst@gmail.com>

mintlify bot deployed to staging November 4, 2025 16:59 View deployment

Update scripts/README.md

c6ca324

Co-authored-by: Engel Nyst <engel.nyst@gmail.com>

xingyaoww mentioned this pull request Nov 4, 2025

Add auto-generated Python API reference documentation #69

Closed

6 tasks

mintlify bot deployed to staging November 4, 2025 19:52 View deployment

enyst reviewed Nov 5, 2025

View reviewed changes

.github/workflows/sync-docs-and-api.yml Outdated Show resolved Hide resolved

Update .github/workflows/sync-docs-and-api.yml

b743dcd

mintlify bot deployed to staging November 5, 2025 05:23 View deployment

mintlify bot deployed to staging November 5, 2025 05:24 View deployment

Merge branch 'main' of github.com:OpenHands/docs into feature/api-doc…

4e601b7

…s-generation-pipeline

enyst force-pushed the feature/api-docs-generation-pipeline branch from 921fdef to 4e601b7 Compare November 5, 2025 05:45

mintlify bot deployed to staging November 5, 2025 05:46 View deployment

docs: sync documentation from agent-sdk

b82799d

- Synced code blocks from examples - Generated API reference documentation Synced from agent-sdk ref: main

mintlify bot deployed to staging November 5, 2025 05:47 View deployment

enyst and others added 2 commits November 6, 2025 15:25

Merge branch 'main' into feature/api-docs-generation-pipeline

7a72883

docs: sync documentation from agent-sdk

b2c742b

- Synced code blocks from examples - Generated API reference documentation Synced from agent-sdk ref: main

mintlify bot deployed to staging November 6, 2025 14:26 View deployment

Merge branch 'main' into feature/api-docs-generation-pipeline

b09e158

mintlify bot deployed to staging November 6, 2025 14:33 View deployment

enyst reviewed Nov 6, 2025

View reviewed changes

mintlify bot deployed to staging November 6, 2025 14:49 View deployment

mintlify bot deployed to staging November 6, 2025 14:55 View deployment

enyst approved these changes Nov 6, 2025

View reviewed changes

enyst merged commit 3c93ad0 into main Nov 6, 2025
2 checks passed

enyst deleted the feature/api-docs-generation-pipeline branch November 6, 2025 16:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add automated API documentation generation pipeline #72

feat: Add automated API documentation generation pipeline #72

Uh oh!

rbren commented Nov 3, 2025

Uh oh!

openhands-ai bot commented Nov 4, 2025

Uh oh!

Uh oh!

enyst Nov 6, 2025

Uh oh!

openhands-ai bot Nov 6, 2025

Uh oh!

openhands-ai bot Nov 6, 2025

Uh oh!

enyst left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

feat: Add automated API documentation generation pipeline #72

feat: Add automated API documentation generation pipeline #72

Uh oh!

Conversation

rbren commented Nov 3, 2025

🚀 Features

Core Pipeline

Scripts & Tools

Configuration & Setup

📁 Generated Structure

🛠 Usage

Quick Start

Integration

🔧 Technical Details

📋 Files Added

Scripts

Configuration

Generated Documentation

Infrastructure

Uh oh!

openhands-ai bot commented Nov 4, 2025

Uh oh!

Uh oh!

enyst Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

openhands-ai bot Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

openhands-ai bot Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

enyst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants