docs(sql): add comprehensive Doxygen comments to Phase 4 modules #142
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added cs_config() function for easier config inspection
Drop operators more good
ORE Operators and Sorting
Use same drop setup on install and uninstall
This change reverts this repo back to 01dcc24. The changes reverted include commits from: - #86 - #87 We mostly want to revert the changes in #86 (since they aren't working as intended with Proxy), but #87 is also included since it's more recent (and also includes some ORE-related changes that would be tedious to untangle). Since there aren't many changes after #86, the most pragmatic option is to revert to the last known-good state and redo the install/uninstall changes by hand on top of that. Commands used to revert: ``` git reset --hard 01dcc24 git reset --soft ed460fc ``` This change doesn't use `git revert` because there were > 20 commits to revert and merge commits also don't play well with `git revert`.
Revert back to commit 01dcc24
This change updates `cs_ore_64_8_v1` to parse ORE indexes (the `'o'` field) as JSON arrays of hex-encoded strings (instead of casting from the Postgres text format). The corresponding change for encoding ORE indexes as JSON arrays of hex-encoded stings has already been merged in Proxy. This is similar to some of the changes in #86, but we're parsing into the composite types for ORE indexes instead of into a plain `bytea[]`. Parsing into the composite type allows for ordering with an operator class on the output from `cs_ore_64_8_v1`.
Parse ORE indexes as JSON arrays of hex-encoded strings
Setup for SQL Doxygen documentation project: - Documentation standards with required tags - Templates for all SQL object types - Inventory of 52 SQL files to document - Cross-reference sync rules (SQL is source of truth) - Tracking files for discrepancies and issues Part of: add-doxygen-sql-comments plan PR: Phase 0 + Phase 1 (Setup)
Add Rust/SQLx tests covering JSONB functionality in encrypted payloads: **JSONB Array Tests:** - jsonb_array_elements_text - array element extraction - jsonb_array_length - array size validation **JSONB Path Query Tests:** - jsonb_path_query - complex path expressions - jsonb_path_query_first - single result queries - jsonb_path_exists - path existence checks - Array-specific path operations **JSONB Structure Tests:** - Encrypted selector validation (ct, k, i, p, ob fields) - JSONB field type verification - Payload structure correctness **Test Helpers:** - Function call tracking verification - Test framework meta-tests These tests migrate all JSONB-related assertions from the original SQL test suite to the new Rust/SQLx framework, ensuring consistent behavior across PostgreSQL versions 14-17.
Add comprehensive Rust/SQLx tests for encrypted data equality: **HMAC Index Equality:** - Operator tests with matching encrypted values - Operator tests with non-matching encrypted values - Plain JSONB comparison validation **Blake3 Index Equality:** - Operator tests with matching encrypted values - Operator tests with non-matching encrypted values - eq() function tests for exact equality - JSONB payload structure preservation (including 'ob' field) **Key Differences Tested:** - HMAC removes 'ob' field during equality comparison - Blake3 preserves 'ob' field for complete payload equality - Both support operator and function-based equality checks These tests ensure encrypted data can be reliably compared for equality using both index-based operators and explicit eq() function calls, maintaining SQL parity across all supported PostgreSQL versions.
Add Doxygen-style documentation to core customer-facing modules: Config module (public): - add_search_config: Configure searchable encryption indexes - remove_search_config: Remove index configurations - modify_search_config: Update index configurations - migrate_config: Transition pending to encrypting state - activate_config: Activate encrypting configuration - discard: Discard pending configuration - add_column: Configure column for encryption - remove_column: Remove column from encryption - reload_config: Placeholder for config sync - config: Query configuration in tabular format Config module (private): - config_default: Initialize default config structure - config_add_table: Add table to configuration - config_add_column: Add column to table config - config_add_cast: Set cast type for column - config_add_index: Add search index to column - config_match_default: Generate match index defaults Encrypted types: - eql_v2_encrypted: Core composite type for encrypted data Operators: - eq: Equality comparison helper (internal) - = operator: Three overloads for encrypted/JSONB comparisons Coverage: 21 objects documented across 4 files All include @brief, @param, @return tags Customer-facing functions include @example tags Internal functions marked with @internal Part of: add-doxygen-sql-comments plan Phase: 2 (Core modules - checkpoint 1/7)
…se 2 checkpoint 2) Add Doxygen documentation for encrypted column operations and operators: Encrypted functions: - ciphertext: Extract ciphertext from encrypted values (2 overloads) - meta_data: Extract index terms/metadata (2 overloads) - grouped_value: Aggregate for first non-null value in groups - _first_grouped_value: Internal state function for aggregate - add_encrypted_constraint: Add validation constraints - remove_encrypted_constraint: Remove validation constraints Comparison operators: - lt: Less-than helper (internal) - < operator: Three overloads for ORE-based comparisons Pattern matching operators: - like: LIKE helper using bloom filters (internal) - ilike: Case-insensitive LIKE helper (internal) - ~~ operator: Three overloads for pattern matching (LIKE) - ~~* operator: Case-insensitive pattern matching (ILIKE) Coverage: 16 functions/operators documented across 3 files All include @brief, @param, @return tags Customer-facing functions include @example tags Internal functions marked with @internal Part of: add-doxygen-sql-comments plan Phase: 2 (Core modules - checkpoint 2/7)
Add Doxygen documentation for less-than-or-equal operator: - lte: Internal comparison helper - <= operator: Three overloads (encrypted/encrypted, encrypted/jsonb, jsonb/encrypted) All include @brief, @param, @return tags with example usage. Part of: add-doxygen-sql-comments plan Phase: 2 (Core modules - checkpoint 3/7)
Add execution strategy section explaining: - Batch-based documentation approach - Parallel subagent usage for Phases 3-4 - Updated time estimates (26-39 hours) - Pre-flight checks phase Part of: add-doxygen-sql-comments plan Phase: 0 (Pre-flight)
Add Doxygen documentation for remaining comparison operators: - > operator: Greater-than comparison (3 overloads) - >= operator: Greater-than-or-equal comparison (3 overloads) - <> operator: Not-equal comparison (3 overloads) Each includes gt/gte/neq internal helpers marked @internal. All include @brief, @param, @return tags with ORE index examples. Part of: add-doxygen-sql-comments plan Phase: 2 (Core modules - comparison operators complete)
Add comprehensive Doxygen documentation to remaining operator files: JSONB field accessor operators: - -> (field accessor, 3 overloads: text, encrypted, integer) - ->> (text-returning accessor, 2 overloads) - @> (contains operator for ste_vec) - <@ (contained-by operator, reverse of @>) Comparison support functions: - compare.sql: Core comparison with ORE priority cascade - order_by.sql: ORE term extraction for ORDER BY clauses - operator_class.sql: Btree operator family/class definitions All files now include: - @brief descriptions explaining operator purpose - @param/@return type documentation - @example usage demonstrations - @note for important implementation details - @see cross-references to related functions Phase 2 (Core Modules - Operators) now complete. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Fix assertion count: 513 total (was incorrectly 558) - Fix migrated count: 56 assertions (equality + JSONB, not 40) - Recalculate all coverage percentages with correct totals - Split Task 15 into 8 incremental sub-tasks (15.1-15.9) - Add index type constants module (tests/sqlx/src/index_types.rs) - Add fixture schema documentation (tests/sqlx/fixtures/FIXTURE_SCHEMA.md) - Update success criteria and performance expectations - Remaining: 457/513 assertions (89.1%) to migrate
…s (Phase 3 batch 1) Documented three index module types with comprehensive Doxygen comments: Modules documented: - blake3: 6 database objects (type, constructor, extractor, comparator, caster, compare function) - hmac_256: 6 database objects (type, constructor, extractor, comparator, caster, compare function) - bloom_filter: 5 database objects (type, constructor, extractor, comparator, caster) Changes: - Added @brief descriptions for all types and functions - Documented @param and @return tags for all parameters and return values - Clarified "three-way comparison" terminology in compare functions - Standardized documentation format across all index modules Progress: 17 database objects documented across 8 files
Documented 27 database objects implementing Order-Revealing Encryption for range queries on encrypted data: - 2 types (ore_block_u64_8_256_term, ore_block_u64_8_256_v1) - 8 functions (make, extract, comparison, equality) - 1 comparison function (ore_block_u64_8_256_term_cmp) - 2 casts (to/from JSONB) - 6 operator functions (=, <>, <, <=, >, >=) - 6 operators supporting encrypted range queries - 2 operator class definitions (btree, hash) Note: operators.sql and operator_class.sql marked as DISABLED (not included in build due to performance/architectural decisions) Progress: Phase 3 batch 2 - ORE block cryptographic module complete
test(sqlx): add comprehensive JSONB operator tests
…se 3 final)
Completes Phase 3 documentation of advanced index modules:
Modules completed:
- ore_cllw_u64_8: 7 objects documented (CLLW ORE fixed-width u64)
- Types: ore_cllw_u64_8_term_v1, ore_cllw_u64_8_encrypted_v1
- Functions: ore_cllw_u64_8_term_v1, ore_cllw_u64_8_encrypted_v1,
ore_cllw_u64_8_compare, ore_cllw_u64_8_lt, ore_cllw_u64_8_gt
- Shared comparison functions for order-preserving encryption
- ore_cllw_var_8: 7 objects documented (CLLW ORE variable-width)
- Types: ore_cllw_var_8_term_v1, ore_cllw_var_8_encrypted_v1
- Functions: ore_cllw_var_8_term_v1, ore_cllw_var_8_encrypted_v1,
ore_cllw_var_8_compare, ore_cllw_var_8_lt, ore_cllw_var_8_gt
- Parallel to fixed-width with variable-length support
- ste_vec: 12 objects documented (STE vector for containment queries)
- Types: ste_vec_term_v1, ste_vec_encrypted_v1, ste_vec_value_v1
- Functions: 9 specialized functions for STE vector operations
- Critical for @> (contains) and <@ (contained by) operators
Phase 3 Statistics:
- 26 total objects documented across 7 files
- 286 insertions, 38 deletions (net +248 lines)
- All ORE (order-revealing encryption) modules now documented
- All STE (searchable text encryption) modules now documented
- All index modules in EQL now have complete Doxygen documentation
All index modules (blake3, hmac_256, bloom_filter, ORE variants, STE)
now have comprehensive Doxygen comments following Phase 1 patterns.
Fixes 4 HIGH severity issues identified in plan review:
1. Remove duplicate ore table creation
- Deleted ore_data.sql fixture creation (Task 2, Step 2-3)
- ore table already exists from migrations/002_install_ore_data.sql
- Tests now use #[sqlx::test] without fixtures
2. Fix ORE helper function calls
- Replace create_encrypted_json_with_index(&pool, 42, "ore64")
- With get_ore_encrypted(&pool, 42) to select from pre-seeded table
- Applied across Tasks 2-6, 8-9, 12 (all ORE tests)
- ore table has 'ob' key (ORE_BLOCK), not ore64 index
3. Fix invalid test IDs
- Replace id=91347 with id=4 for "different record" tests
- 91347 * 10 = 913470 outside ore range (1-99) → NULL → panic
- 4 * 10 = 40 is within ore range → valid data
- Applied in Task 1 (inequality tests)
4. Remove ore_data fixture references
- Removed all scripts("ore_data") fixture attributes
- Updated tests to use migrations instead
- Added comments explaining ore table source
Updated documentation:
- FIXTURE_SCHEMA.md: Document ore from migrations (NOT fixture)
- Added helper function documentation (get_ore_encrypted)
- Added valid id range explanation (1-9 → lookups 10-90)
Coverage: All ORE-related tests now use correct patterns
Update all JSONB parameter descriptions to use consistent format: - Change from 'JSONB Raw encrypted value' or 'JSONB Encrypted data payload' - Change to 'jsonb containing encrypted EQL payload' Changes: - Use lowercase 'jsonb' (correct PostgreSQL type convention) - Use consistent phrase 'containing encrypted EQL payload' - Updated 19 parameters across 8 files (Phase 2 + Phase 3) Files modified: - src/encrypted/functions.sql - src/blake3/functions.sql - src/hmac_256/functions.sql - src/bloom_filter/functions.sql - src/ore_block_u64_8_256/functions.sql - src/ore_cllw_u64_8/functions.sql - src/ore_cllw_var_8/functions.sql - src/ste_vec/functions.sql
Remove operators.sql and operator_class.sql from ore_block_u64_8_256 module. These files were disabled (!REQUIRE) and are no longer needed. Files removed: - src/ore_block_u64_8_256/operators.sql - src/ore_block_u64_8_256/operator_class.sql
This reverts commit bcca4b8.
test(sqlx): add equality operator and eq() function tests
docs(sql): Add Doxygen-style documentation (Phase 1-2: Core modules)
Add detailed Doxygen documentation to 13 SQL files across Phase 4: **Encrypted Supporting Files (4 files):** - src/encrypted/aggregates.sql - MIN/MAX aggregate functions for ORE - src/encrypted/casts.sql - Type conversion between JSONB/text/encrypted - src/encrypted/compare.sql - Fallback comparison for btree correctness - src/encrypted/constraints.sql - 5 validation functions with @example tags **JSONB Functions (1 file, 15 functions):** - src/jsonb/functions.sql - Path query operations and array manipulation - jsonb_path_query (3 overloads) - Query for matching elements - jsonb_path_exists (3 overloads) - Check path existence - jsonb_path_query_first (3 overloads) - Get first matching element - jsonb_array_length (2 overloads) - Get array length - jsonb_array_elements (2 overloads) - Extract array elements - jsonb_array_elements_text (2 overloads) - Extract as ciphertext **Config Schema (4 files):** - src/config/types.sql - Configuration state ENUM documentation - src/config/tables.sql - eql_v2_configuration table structure - src/config/indexes.sql - Partial unique indexes for state constraints - src/config/constraints.sql - 5 validation functions **Encryptindex Functions (1 file, 7 functions):** - src/encryptindex/functions.sql - Configuration lifecycle management - diff_config, select_pending_columns, select_target_columns - ready_for_encryption, create_encrypted_columns - rename_encrypted_columns, count_encrypted_with_active_config **Root Utilities (3 files):** - src/common.sql - Constant-time comparison, JSONB conversion, logging - src/crypto.sql - pgcrypto extension enablement - src/schema.sql - eql_v2 schema creation with warnings **Documentation Quality:** - ✅ Consistent use of @brief, @param, @return, @throws, @note, @see tags - ✅ @internal tags mark implementation details vs customer-facing APIs - ✅ @example sections show concrete usage for customer functions - ✅ Cross-references create navigable documentation graph - ✅ File-level @file documentation provides module context - ✅ Security notes highlight timing attack prevention - ✅ All tests pass - documentation doesn't break functionality **Statistics:** - Files modified: 13 - Functions documented: 32+ - Lines added: +718 gross, +555 net - Test status: ✅ All 40+ test files passing **Code Review:** - Status: APPROVED (see CODE_REVIEW_PHASE_4_DOXYGEN.md) - BLOCKING issues: 0 - NON-BLOCKING issues: 6 (addressed in follow-up commit) - Review verified documentation accuracy against implementation
Address code review recommendations to improve documentation accuracy
and add safety warnings for DDL-executing functions.
**Documentation Improvements:**
1. **src/encryptindex/functions.sql:102-103** - Clarified select_target_columns
- Enhanced note to explain full JOIN logic (checks both column_name
and column_name_encrypted with type verification)
- Previous: "LEFT JOIN returns NULL when no match"
- Updated: Explains both name matching variants and type requirement
2. **src/jsonb/functions.sql:296** - Enhanced jsonb_array_length exception
- More specific @throws documentation with exact error message
- Previous: "if value is not an array (missing 'a' flag)"
- Updated: "'cannot get array length of a non-array' if 'a' flag
is missing or not true"
3. **src/jsonb/functions.sql:29** - Clarified NULL handling in jsonb_path_query
- Precise SETOF return semantics
- Previous: "Returns NULL if val is NULL"
- Updated: "Returns a set containing NULL if val is NULL"
**Safety Warnings:**
4. **src/encryptindex/functions.sql:152** - Added @warning to create_encrypted_columns
- Highlights dynamic DDL execution (ALTER TABLE ADD COLUMN)
- Alerts users to schema modification side effects
5. **src/encryptindex/functions.sql:180** - Added @warning to rename_encrypted_columns
- Highlights dynamic DDL execution (ALTER TABLE RENAME COLUMN)
- Alerts users to schema modification side effects
**Quality Assurance:**
- ✅ All tests passing (59 test files)
- ✅ Code review approved (CODE_REVIEW_PHASE_4_COMPLETE.md)
- ✅ 0 blocking issues
- ✅ All NON-BLOCKING recommendations addressed
**Note:** User fixed blocking code bug in jsonb_path_query_first separately
…ed_with_active_config **Problem:** Documentation incorrectly stated that the 'v' field stores the active configuration ID, contradicting src/encrypted/constraints.sql:56 which correctly documents that 'v' must be the literal payload version "2". **Root Cause:** The 'v' field serves a single, consistent purpose across the entire codebase: it stores the EQL payload format version (currently "2"). It does NOT store configuration IDs. **Fix:** - Removed incorrect claim that 'v' stores configuration ID - Clarified that 'v' field stores payload version "2" - Updated function description to avoid implementation details - Added note explaining the distinction **Files Changed:** - src/encryptindex/functions.sql:201-209 **Verification:** - ✅ All tests passing (59 test files) - ✅ Documentation now consistent with src/encrypted/constraints.sql:56 - ✅ No code changes - documentation-only fix **Context:** This addresses feedback about major documentation inconsistency where two different files made contradictory claims about the 'v' field's purpose. The payload version is always "2" for EQL v2, as enforced by the _encrypted_check_v validation function.
tobyhede
force-pushed
the
add-phase-4-doxygen
branch
from
eee5f5d to
703ac04
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add comprehensive Doxygen documentation to 13 SQL files across Phase 4, completing systematic documentation of the EQL codebase's core infrastructure.
Phase 4 Coverage:
Files Modified (13 total)
Encrypted Supporting Files (4 files)
JSONB Functions (1 file, 15 functions)
jsonb_path_query(3 overloads) - Query for matching elementsjsonb_path_exists(3 overloads) - Check path existencejsonb_path_query_first(3 overloads) - Get first matching elementjsonb_array_length(2 overloads) - Get array lengthjsonb_array_elements(2 overloads) - Extract array elementsjsonb_array_elements_text(2 overloads) - Extract as ciphertextConfig Schema (4 files)
Encryptindex Functions (1 file, 7 functions)
diff_config- Compare configurationsselect_pending_columns- Get columns needing encryptionselect_target_columns- Map to encrypted columnsready_for_encryption- Check readinesscreate_encrypted_columns- Create encrypted columnsrename_encrypted_columns- Finalize encryptioncount_encrypted_with_active_config- Track progressRoot Utilities (3 files)
Documentation Quality Standards
Comprehensive Doxygen Tags:
@brief- Clear one-line summaries for all symbols@param- Detailed parameter descriptions with types@return- Return value descriptions@throws- Exception conditions documented@note- Important usage notes and context@see- Cross-references creating navigable docs@internal- Implementation details marked clearly@example- Concrete usage examples for customer-facing functions@file- Module-level documentationKey Documentation Features:
Code Review
Thorough Implementation Verification:
Review Highlights:
Test Results
✅ All 40+ test files passing
Build Status: ✅ Clean build, no errors or warnings
Statistics
Quality Assurance
Related Work
This PR builds on previous documentation phases:
Next Steps
After merge: