Skip to content

docs(sql): Add Doxygen-style documentation (Phase 1-2: Core modules) #141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 16 commits into from
Oct 27, 2025
Merged

docs(sql): Add Doxygen-style documentation (Phase 1-2: Core modules) #141

merged 16 commits into from
Oct 27, 2025

Conversation

@tobyhede
Copy link
Contributor

@tobyhede tobyhede commented Oct 27, 2025 •
edited
Loading

💪 What

Adds Doxygen-style documentation to 8 critical SQL files covering configuration and core encryption functionality:

  • Adds documentation standards and templates defining required tags (@brief, @param, @return, @example) and formatting conventions for all SQL object types
  • Documents 16 configuration functions (10 public + 6 internal) for managing encryption configuration, including add_column, add_search_config, and configuration lifecycle functions
  • Documents core encrypted type (eql_v2_encrypted) with detailed structure explanation of JSONB payload fields
  • Documents 8 encrypted utility functions for extracting ciphertext, metadata, and aggregating encrypted values
  • Documents 18 operator overloads for equality (=), less-than (<), less-than-or-equal (<=), and pattern matching (~~) comparisons on encrypted data
  • Creates tracking infrastructure for managing discrepancies between SQL code and reference documentation

Coverage: 8 of 52 SQL files documented (15% complete), ~43 functions/operators/types with full Doxygen tags

Quality metrics:

  • All customer-facing functions include @example tags with practical usage
  • Internal/private functions marked with @internal tag
  • Index requirements (unique, match, ore) explicitly noted in operator documentation
  • All SQL validated for syntax correctness

Testing: Documentation changes only - no functional code changes, existing tests continue to pass

tobyhede and others added 10 commits October 24, 2025 14:01
@tobyhede @claude
docs: add comprehensive plan for SQL Doxygen documentation
c53aa13 
Add detailed implementation plan for adding Doxygen-style comments to all
53 SQL implementation files across the EQL codebase.

Plan includes:
- 6 phases with 29 tasks (21-31 hour estimate)
- Complete file checklist (53 files across 13 modules)
- Validation scripts for coverage, syntax, and required tags
- Templates and standards documentation
- Quality assurance and handoff procedures

Related to SQL documentation generation RFC.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@tobyhede
docs(sql): add documentation standards, templates, and tooling (Phase 1)
295e72e 
Setup for SQL Doxygen documentation project:
- Documentation standards with required tags
- Templates for all SQL object types
- Inventory of 52 SQL files to document
- Cross-reference sync rules (SQL is source of truth)
- Tracking files for discrepancies and issues

Part of: add-doxygen-sql-comments plan
PR: Phase 0 + Phase 1 (Setup)
@tobyhede
docs(sql): document config module and core types (Phase 2 checkpoint)
ad579b1 
Add Doxygen-style documentation to core customer-facing modules:

Config module (public):
- add_search_config: Configure searchable encryption indexes
- remove_search_config: Remove index configurations
- modify_search_config: Update index configurations
- migrate_config: Transition pending to encrypting state
- activate_config: Activate encrypting configuration
- discard: Discard pending configuration
- add_column: Configure column for encryption
- remove_column: Remove column from encryption
- reload_config: Placeholder for config sync
- config: Query configuration in tabular format

Config module (private):
- config_default: Initialize default config structure
- config_add_table: Add table to configuration
- config_add_column: Add column to table config
- config_add_cast: Set cast type for column
- config_add_index: Add search index to column
- config_match_default: Generate match index defaults

Encrypted types:
- eql_v2_encrypted: Core composite type for encrypted data

Operators:
- eq: Equality comparison helper (internal)
- = operator: Three overloads for encrypted/JSONB comparisons

Coverage: 21 objects documented across 4 files
All include @brief, @param, @return tags
Customer-facing functions include @example tags
Internal functions marked with @internal

Part of: add-doxygen-sql-comments plan
Phase: 2 (Core modules - checkpoint 1/7)
@tobyhede
docs(sql): document encrypted functions and comparison operators (Pha…
3e96b26 
…se 2 checkpoint 2)

Add Doxygen documentation for encrypted column operations and operators:

Encrypted functions:
- ciphertext: Extract ciphertext from encrypted values (2 overloads)
- meta_data: Extract index terms/metadata (2 overloads)
- grouped_value: Aggregate for first non-null value in groups
- _first_grouped_value: Internal state function for aggregate
- add_encrypted_constraint: Add validation constraints
- remove_encrypted_constraint: Remove validation constraints

Comparison operators:
- lt: Less-than helper (internal)
- < operator: Three overloads for ORE-based comparisons

Pattern matching operators:
- like: LIKE helper using bloom filters (internal)
- ilike: Case-insensitive LIKE helper (internal)
- ~~ operator: Three overloads for pattern matching (LIKE)
- ~~* operator: Case-insensitive pattern matching (ILIKE)

Coverage: 16 functions/operators documented across 3 files
All include @brief, @param, @return tags
Customer-facing functions include @example tags
Internal functions marked with @internal

Part of: add-doxygen-sql-comments plan
Phase: 2 (Core modules - checkpoint 2/7)
@tobyhede
docs(sql): document <= comparison operator (Phase 2 checkpoint 3)
13bdcdd 
Add Doxygen documentation for less-than-or-equal operator:
- lte: Internal comparison helper
- <= operator: Three overloads (encrypted/encrypted, encrypted/jsonb, jsonb/encrypted)

All include @brief, @param, @return tags with example usage.

Part of: add-doxygen-sql-comments plan
Phase: 2 (Core modules - checkpoint 3/7)
@tobyhede
docs(sql): update plan with execution strategy details
2fa0d40 
Add execution strategy section explaining:
- Batch-based documentation approach
- Parallel subagent usage for Phases 3-4
- Updated time estimates (26-39 hours)
- Pre-flight checks phase

Part of: add-doxygen-sql-comments plan
Phase: 0 (Pre-flight)
@tobyhede
docs(sql): document comparison operators (>, >=, <>)
9f75204 
Add Doxygen documentation for remaining comparison operators:
- > operator: Greater-than comparison (3 overloads)
- >= operator: Greater-than-or-equal comparison (3 overloads)
- <> operator: Not-equal comparison (3 overloads)

Each includes gt/gte/neq internal helpers marked @internal.
All include @brief, @param, @return tags with ORE index examples.

Part of: add-doxygen-sql-comments plan
Phase: 2 (Core modules - comparison operators complete)
@tobyhede @claude
docs(operators): complete Phase 2 - document JSONB and support functions
f4ea5ec 
Add comprehensive Doxygen documentation to remaining operator files:

JSONB field accessor operators:
- -> (field accessor, 3 overloads: text, encrypted, integer)
- ->> (text-returning accessor, 2 overloads)
- @> (contains operator for ste_vec)
- <@ (contained-by operator, reverse of @>)

Comparison support functions:
- compare.sql: Core comparison with ORE priority cascade
- order_by.sql: ORE term extraction for ORDER BY clauses
- operator_class.sql: Btree operator family/class definitions

All files now include:
- @brief descriptions explaining operator purpose
- @param/@return type documentation
- @example usage demonstrations
- @note for important implementation details
- @see cross-references to related functions

Phase 2 (Core Modules - Operators) now complete.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@tobyhede
docs(operators): clarify ->> alias semantics
7a1d46d
@tobyhede
docs(config): align modify_search_config throws
b96945c
@tobyhede tobyhede marked this pull request as ready for review October 27, 2025 03:50
@tobyhede
docs(sql): add Doxygen comments to hash and bloom filter index module…
cebefcd 
…s (Phase 3 batch 1)

Documented three index module types with comprehensive Doxygen comments:

Modules documented:
- blake3: 6 database objects (type, constructor, extractor, comparator, caster, compare function)
- hmac_256: 6 database objects (type, constructor, extractor, comparator, caster, compare function)
- bloom_filter: 5 database objects (type, constructor, extractor, comparator, caster)

Changes:
- Added @brief descriptions for all types and functions
- Documented @param and @return tags for all parameters and return values
- Clarified "three-way comparison" terminology in compare functions
- Standardized documentation format across all index modules

Progress: 17 database objects documented across 8 files
@tobyhede
docs(sql): add Doxygen comments to ORE block module (Phase 3 batch 2)
96fd995 
Documented 27 database objects implementing Order-Revealing Encryption
for range queries on encrypted data:
- 2 types (ore_block_u64_8_256_term, ore_block_u64_8_256_v1)
- 8 functions (make, extract, comparison, equality)
- 1 comparison function (ore_block_u64_8_256_term_cmp)
- 2 casts (to/from JSONB)
- 6 operator functions (=, <>, <, <=, >, >=)
- 6 operators supporting encrypted range queries
- 2 operator class definitions (btree, hash)

Note: operators.sql and operator_class.sql marked as DISABLED
(not included in build due to performance/architectural decisions)

Progress: Phase 3 batch 2 - ORE block cryptographic module complete
freshtonic
freshtonic approved these changes Oct 27, 2025
View reviewed changes
Copy link
Contributor

@freshtonic freshtonic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Did not read the entire diff though.

Left a comment.

docs/development/reference-sync-rules.md
**Example:**
```sql
--! @brief Extract ciphertext from encrypted value
--! @param encrypted JSONB Raw encrypted value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of JSONB Raw encrypted value, something like "jsonb containing encrypted EQL payload" ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, changed.

@tobyhede
docs(sql): add Doxygen comments to remaining ORE and STE modules (Pha…
41a3cc9 
…se 3 final)

Completes Phase 3 documentation of advanced index modules:

Modules completed:
- ore_cllw_u64_8: 7 objects documented (CLLW ORE fixed-width u64)
  - Types: ore_cllw_u64_8_term_v1, ore_cllw_u64_8_encrypted_v1
  - Functions: ore_cllw_u64_8_term_v1, ore_cllw_u64_8_encrypted_v1,
    ore_cllw_u64_8_compare, ore_cllw_u64_8_lt, ore_cllw_u64_8_gt
  - Shared comparison functions for order-preserving encryption

- ore_cllw_var_8: 7 objects documented (CLLW ORE variable-width)
  - Types: ore_cllw_var_8_term_v1, ore_cllw_var_8_encrypted_v1
  - Functions: ore_cllw_var_8_term_v1, ore_cllw_var_8_encrypted_v1,
    ore_cllw_var_8_compare, ore_cllw_var_8_lt, ore_cllw_var_8_gt
  - Parallel to fixed-width with variable-length support

- ste_vec: 12 objects documented (STE vector for containment queries)
  - Types: ste_vec_term_v1, ste_vec_encrypted_v1, ste_vec_value_v1
  - Functions: 9 specialized functions for STE vector operations
  - Critical for @> (contains) and <@ (contained by) operators

Phase 3 Statistics:
- 26 total objects documented across 7 files
- 286 insertions, 38 deletions (net +248 lines)
- All ORE (order-revealing encryption) modules now documented
- All STE (searchable text encryption) modules now documented
- All index modules in EQL now have complete Doxygen documentation

All index modules (blake3, hmac_256, bloom_filter, ORE variants, STE)
now have comprehensive Doxygen comments following Phase 1 patterns.
@tobyhede
docs(sql): standardize JSONB parameter descriptions
6795a8c 
Update all JSONB parameter descriptions to use consistent format:
- Change from 'JSONB Raw encrypted value' or 'JSONB Encrypted data payload'
- Change to 'jsonb containing encrypted EQL payload'

Changes:
- Use lowercase 'jsonb' (correct PostgreSQL type convention)
- Use consistent phrase 'containing encrypted EQL payload'
- Updated 19 parameters across 8 files (Phase 2 + Phase 3)

Files modified:
- src/encrypted/functions.sql
- src/blake3/functions.sql
- src/hmac_256/functions.sql
- src/bloom_filter/functions.sql
- src/ore_block_u64_8_256/functions.sql
- src/ore_cllw_u64_8/functions.sql
- src/ore_cllw_var_8/functions.sql
- src/ste_vec/functions.sql
@tobyhede
chore(sql): remove disabled ORE block operator files
bcca4b8 
Remove operators.sql and operator_class.sql from ore_block_u64_8_256 module.
These files were disabled (!REQUIRE) and are no longer needed.

Files removed:
- src/ore_block_u64_8_256/operators.sql
- src/ore_block_u64_8_256/operator_class.sql
@tobyhede
Revert "chore(sql): remove disabled ORE block operator files"
619049f 
This reverts commit bcca4b8.
@tobyhede tobyhede merged commit 246701b into main Oct 27, 2025
4 checks passed
@tobyhede tobyhede deleted the add-doxygen-sql-comments branch October 27, 2025 06:04
This was referenced Oct 27, 2025
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@freshtonic freshtonic freshtonic approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tobyhede @freshtonic