Skip to content

Add doxygen docs comments #148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 33 commits into from
Nov 11, 2025
Merged

Add doxygen docs comments #148

merged 33 commits into from
Nov 11, 2025

Conversation

@tobyhede
Copy link
Contributor

@tobyhede tobyhede commented Oct 29, 2025 •
edited
Loading

  • complete Doxygen documentation
  • mise tasks for docs:generate and docs:validate
  • test CI workflow uses docs:validate

@auxesis auxesis force-pushed the doxygen-docs-clean branch from 13d4d7d to 58ba053 Compare November 6, 2025 11:42
auxesis
auxesis requested changes Nov 6, 2025
View reviewed changes
Copy link
Contributor

@auxesis auxesis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @tobyhede — this is a really solid foundation for better reference docs.

I've pushed a couple of small fixes to how the mise tasks are organised, and ensured that all the doc validations are running.

Currently docs:validate:documented-sql is failing, and that will need to be fixed before merging.

tobyhede and others added 24 commits November 11, 2025 10:39
@tobyhede
docs(sql): add complete Doxygen documentation (Phases 1-4)
36c56d9 
Add comprehensive Doxygen comments across all SQL modules:

**Phase 1**: Documentation standards, templates, and tooling
**Phase 2**: Config module, encrypted types, operators, comparison functions
**Phase 3**: Index implementations (Blake3, HMAC-256, Bloom Filter, ORE, STE)
**Phase 4**: JSONB functions, config schema, lifecycle management, utilities

This commit consolidates all documentation work from continue-doxygen-sql-comments
branch (commit a398dc8) into a clean history.

Includes documentation for 52+ SQL files with complete @brief, @param, @return,
@throws, @example, and @see tags following established standards.

Source branch: continue-doxygen-sql-comments (pre-merge state)
Original commits: f41981a..a398dc8 (19 commits)
@tobyhede
docs: add Doxygen infrastructure and tooling
a2cbf00 
Add complete Doxygen generation and validation infrastructure:

**Configuration:**
- Doxyfile - Doxygen configuration for HTML/LaTeX output
- CLAUDE.md - Documentation standards and guidelines

**Validation Scripts:**
- tasks/check-doc-coverage.sh - Verify documentation coverage
- tasks/validate-required-tags.sh - Ensure required tags present
- tasks/doxygen-filter.sh - SQL-to-C++ comment filter for Doxygen

**Mise Tasks:**
- docs:generate - Generate API documentation
- docs:validate - Run coverage and tag validation

Source: phase-4-doxygen branch (commits 2e53216, ee96e15, e8debb0, etc.)
@tobyhede
ci: add documentation validation to test workflow
048e26f 
Add validate-docs job to GitHub Actions workflow:
- Runs documentation coverage and tag validation
- Executes before test job (dependency)
- Uses mise run docs:validate

Also update README.md with Documentation section explaining:
- How to install Doxygen
- How to generate documentation (mise run docs:generate)
- Documentation standards overview

Source: phase-4-doxygen (commits 1264b71, 5e37aca)
@tobyhede
docs(sql): add Doxygen comments to version template
1fac7d7 
Document version.template to achieve 100% documentation coverage:
- Add @file tag for version.sql (the generated file)
- Document eql_v2.version() function with @brief, @return, @example
- Add @note explaining auto-generation from template
- Fixes CI validation error for version.template

Source: phase-4-doxygen (commits d4c2257, 01ab2f8)
@tobyhede
docs(sql): fix documentation validation errors
85463d8 
Add missing documentation tags to operators:
- src/operators/<>.sql: Add @param and @return for JSONB overload
- src/operators/~~.sql: Add @brief, @param, @return for operator

Add validation script:
- tasks/validate-documented-sql.sh: SQL syntax validation (optional)

These fixes ensure all documented functions have required @brief, @param,
and @return tags for proper Doxygen output.

Source: phase-4-doxygen commit b4d6b4d, 01ab2f8
@tobyhede
fix: restore correct SQL code from phase-4-doxygen
7de4031 
Fix functional regressions from using continue-doxygen-sql-comments source:
- src/config/functions.sql: Uncomment add_encrypted_constraint call
- src/config/config_test.sql: Better test documentation
- src/encrypted/constraints_test.sql: Enhanced test documentation
- src/encrypted/functions.sql: Improved documentation
- src/encryptindex/functions.sql: Documentation improvements
- src/encryptindex/functions_test.sql: Test documentation
- src/jsonb/functions.sql: Better function documentation

These files use the phase-4-doxygen versions which branched from clean
main and have correct code + better Phase 4 documentation.

Source: phase-4-doxygen (clean main branch + Phase 4 docs)
@tobyhede
fix(docs): skip auto-generated files in validation
4deefe9 
Add check to skip auto-generated files (with '-- AUTOMATICALLY GENERATED FILE'
marker) in validate-required-tags.sh, matching check-doc-coverage.sh behavior.

This fixes validation errors for src/version.sql which is generated during
build from src/version.template (which is properly documented).

Validation now correctly:
- Skips version.sql (auto-generated, in .gitignore)
- Validates version.template (source with documentation)
- Achieves 100% coverage without errors
@tobyhede
docs: add Doxygen configuration and validation scripts
0320a1f
@tobyhede
docs: add documentation build tasks to mise
e8fce88
@tobyhede
docs: add packaging script for documentation archives
000a9f2
@tobyhede
fix(docs): skip auto-generated files in validation
bfb24ee
@tobyhede
ci: add documentation publishing to release workflow
c38d95d
@tobyhede
chore: ignore generated documentation directory
638822a
@tobyhede
refactor: use auto-discovered docs-package task
477b3e2
@auxesis @tobyhede
test: move tests from TOML to standalone shell scripts
b43ac75
@auxesis @tobyhede
test: move documentation tests under docs:validate:*
84ed601
@auxesis @tobyhede
test: check that the source directory exists before running tests
6546640
@auxesis @tobyhede
build: namespace doxygen tasks under docs:*
eb0040d
@auxesis @tobyhede
build: ensure latest bash is used
2498223
@auxesis @tobyhede
test: ensure the documented SQL is validated
e698958
@auxesis @tobyhede
ci: ensure database is available before running docs:validate
e8b7dfa
@auxesis @tobyhede
refactor: go back to relative paths, so the output is less noisy
13b806b
@tobyhede
feat(docs): add XML and Markdown documentation generation
4a7ad7d 
Add comprehensive documentation tooling with Doxygen XML output and
custom Markdown converter for API reference generation.

Changes:
- Configure Doxygen for XML primary output with schema validation
- Add xml-to-markdown.py converter with proper parameter parsing
- Generate single-file API reference with table-formatted parameters
- Add docs:markdown task for Markdown generation
- Update docs:package to include Markdown in distribution
- Document XML format structure and integration patterns
- Disable broken SQL-specific features, add workarounds

The system extracts 84 documented functions with @param, @return, @note
tags into clean Markdown with professional parameter tables.

Note: Doxygen SQL parsing is imperfect (treats SQL as C++), but
documentation comments are extracted correctly. Function signatures
may show incorrect types but descriptions are accurate.
@tobyhede
fix(docs): resolve validation failures due to SQL file dependencies
b5f0fac 
- Update mise.toml to call correct validation script instead of non-existent files
- Modify documented-sql.sh to recognize dependency errors as expected behavior
- Files that depend on types from other files now show as validated with dependency notation

The validation script was failing because it tests SQL files in isolation without
respecting their dependency declarations. This fix allows the validation to distinguish
between real syntax errors and expected dependency issues.
@tobyhede
fix(ci): add default environment variables to docker-compose for CI
034a780 
The docs validation was failing in CI with connection refused errors because
PostgreSQL was not configured to listen on port 7432 inside the container.

Added default values for all PostgreSQL environment variables in docker-compose.yml:
- PGPORT defaults to 7432 (ensures PostgreSQL listens on the expected port)
- PGUSER, POSTGRES_DB, POSTGRES_USER default to 'cipherstash'
- POSTGRES_PASSWORD defaults to 'password'

This ensures the container works correctly even when mise environment variables
are not properly exported to the docker compose subprocess in CI.
@tobyhede
refactor(ci): consolidate validation into test matrix for consistency
dc78bc1 
Reorganized the GitHub workflow to run documentation validation within the test
job matrix instead of as a separate job. This ensures validation and tests run
in identical environments.

Changes:
- Removed separate validate-docs job (was using mise-action@v2 with v2025.1.6)
- Added validation step within test job (uses mise-action@v3 with v2025.11.2)
- Validation now runs for all PostgreSQL versions (17, 16, 15, 14)

This eliminates the environment mismatch that was causing validation to fail
while tests passed. Both now use the same mise version, setup, and PostgreSQL
configuration.
@tobyhede
ci: remove validate from release
46117b6
tobyhede and others added 3 commits November 11, 2025 11:47
@tobyhede
fix(docs): use dedicated package script instead of inline task
fab1b00 
The docs:package task in mise.toml was trying to package files that didn't
exist (markdown/API.md and README.md in docs/api/). Replaced the inline
script with a call to the existing package.sh script which properly packages
the HTML documentation.

The package.sh script correctly:
- Validates documentation exists before packaging
- Packages only the HTML documentation that's actually generated
- Creates both .tar.gz and .zip archives for distribution
@tobyhede
fix(docs): correct Doxygen parameter documentation format
896c2bb 
When Doxygen parses SQL files as C++, it misinterprets function signatures.
For CREATE FUNCTION foo(val type), Doxygen sees 'type' as the parameter name,
not 'val'.

Changed @param documentation from:
  @param val eql_v2_encrypted Description
To:
  @param eql_v2_encrypted Description

This fix applies only to functions with unique parameter types.
Functions with duplicate parameter types (e.g., compare functions with
two eql_v2_encrypted params) still generate warnings and need a
different approach.

Results:
- Fixed 88 @param lines across 20 SQL files
- Reduced parameter warnings from ~64 to 44 (-31%)
@auxesis
refactor: remove duplicate jobs
7834c35
auxesis
auxesis requested changes Nov 11, 2025
View reviewed changes
@auxesis
fix: remove output about a file that doesn't exist
6bb1bc1
@auxesis
refactor: make markdown generation idiomatic
78aa5a6 
- Re-namespace the markdown generation task, from `docs:markdown`, to
  `docs:generate:markdown`, so it's clearer what it's doing.
  This could be folded into the `docs:generate` task as an option, but
  we can tackle that separately if we find it's more ergonomic.
- Call `docs:generate` if the XML doesn't yet exist, instead of nagging
  the user to do it. Make the computer do the work, not the user.
- Hide xml-to-markdown.py from `mise tasks` output, because it's
  internal implementation that users shouldn't be calling directly.
- Ensure Python is installed, so xml-to-markdown.py works out of the box
@tobyhede
fix(docs): correct XML to Markdown conversion for SQL documentation
2dda98d 
- Extract return types from argsstring element for accurate SQL function types
- Filter out SQL intrinsics (AS, CAST, CHECK, OPERATOR) incorrectly identified as functions
- Fix GitHub anchor generation for functions with underscores
- Remove hardcoded local paths, use dynamic path resolution
- Clean up duplicate backticks in return type formatting

Previously showed incomplete return types (e.g., 'eql_v2' instead of 'eql_v2.blake3')
and included SQL DDL keywords as functions. Now generates clean API documentation
with 75 correctly documented functions and proper type signatures.
auxesis
auxesis approved these changes Nov 11, 2025
View reviewed changes
Copy link
Contributor

@auxesis auxesis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for all the cleanups.

@tobyhede tobyhede merged commit 23a8d6f into main Nov 11, 2025
6 checks passed
@tobyhede tobyhede deleted the doxygen-docs-clean branch November 11, 2025 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@auxesis auxesis auxesis approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tobyhede @auxesis