Skip to content

Conversation

@tobyhede
Copy link
Contributor

U̶p̶d̶a̶t̶e̶ Rewrite all the docs

I've completed a thorough and intensive documentation review of the Encrypt Query Language (EQL) project.

Major Changes

  1. Created New Documentation
  • docs/reference/eql-functions.md (498 lines) - Comprehensive API reference for all 40+ EQL functions, organized by category with both operator and function equivalents
  • docs/reference/database-indexes.md (391 lines) - Complete guide to PostgreSQL B-tree index creation, usage requirements, troubleshooting, and best practices based on operator_class_test.sql
  1. Completely Rewrote Documentation
  • docs/reference/json-support.md - Reduced from 887 to 297 lines, removed all references to non-existent functions (ste_vec_value, ste_vec_term, ste_vec_terms), documented actual API (jsonb_path_query,
    jsonb_array_elements, operators)
  1. Fixed Documentation
  • docs/tutorials/proxy-configuration.md - Corrected to show direct operator usage instead of wrapped function calls
  • docs/reference/index-config.md - Fixed all function names from cs_* to eql_v2.*, added missing cast types (real, double)
  • docs/README.md - Fixed broken links, added references to new documentation files
  • README.md - Enhanced with step-by-step getting started, troubleshooting section
  • DEVELOPMENT.md - Updated with merged content from src/README.md

Cross-References Established

All documentation files now have proper cross-references:

  • database-indexes.md ↔ eql-functions.md ↔ json-support.md ↔ proxy-configuration.md ↔ index-config.md
  • All links verified to exist and point to correct locations

Key Corrections Made

  1. Operators work directly on encrypted columns - no function wrappers needed in normal environments
  2. Function versions exist only for Supabase or operator-restricted environments
  3. PostgreSQL indexes require the eql_v2.encrypted_operator_class operator class
  4. Index terms must exist in data BEFORE creating PostgreSQL indexes
  5. Proper type casting to eql_v2_encrypted is required for index usage

Verification Process

All documentation has been verified against:

  • Actual test files (operator_class_test.sql, functions_test.sql, etc.)
  • Source code implementations (src/config/functions.sql, src/jsonb/functions.sql, etc.)
  • Cross-referenced between docs to ensure consistency

Add two new reference documents:
- eql-functions.md: Complete API reference for all 40+ EQL functions
  organized by category (configuration, query, JSONB, array, helpers,
  aggregates). Includes both operator and function forms with examples.
- database-indexes.md: PostgreSQL B-tree index guide covering creation,
  usage requirements, troubleshooting, and best practices. Based on
  operator_class_test.sql implementation.

These documents provide the missing comprehensive reference material for
EQL's public API and PostgreSQL index integration.
Complete rewrite of json-support.md to match actual EQL API:

- Remove references to non-existent functions (ste_vec_value,
  ste_vec_term, ste_vec_terms)
- Document actual functions: jsonb_path_query, jsonb_array_elements,
  jsonb_path_exists, grouped_value
- Show correct operator usage (@>, <@, ->, ->>)
- Add proper selector-based querying examples
- Reduce from 887 to 297 lines, removing outdated content
- Add explanation of how ste_vec indexing works

All examples now verified against test files and source code.
Fix multiple documentation issues verified against implementation:

- index-config.md: Correct function names from cs_* to eql_v2.*,
  add missing cast types (real, double)
- proxy-configuration.md: Change from wrapped function calls to direct
  operator usage (= instead of hmac_256() wrappers), add function
  alternatives for Supabase
- docs/README.md: Fix broken links, add references to new documentation
  (database-indexes.md, eql-functions.md)

All changes verified against test files to ensure accuracy.
Add comprehensive user guidance to main README.md:

- Add step-by-step getting started example with SQL code showing
  table creation, column configuration, and index setup
- Add troubleshooting section with common errors, causes, and solutions
- Clarify that CipherStash Proxy/Protect.js are required for encryption
- Update TOC: change "Developing" to "Contributing"
- Improve installation instructions and upgrade process

Makes the README more actionable for new users with concrete examples
and solutions to common setup issues.
Add "Adding SQL" section to DEVELOPMENT.md with guidelines for:
- Modular SQL file organization
- Operator wrappers vs implementation separation
- Dependency declarations using REQUIRE comments
- Test file naming conventions
- Build system usage with tsort

Provides clear guidance for contributors developing new SQL features.
Fix critical documentation issues and add missing function documentation:

- Fix return type names: change eql_v2_* to eql_v2.* for index term
  extraction functions (hmac_256, blake3, bloom_filter, ore_block_u64_8_256)
- Add missing function equivalents for Supabase compatibility:
  ilike(), lt(), lte(), gt(), gte()
- Document configuration lifecycle functions: migrate_config(),
  activate_config(), discard(), reload_config()
- Add aggregate functions: min(), max()
- Document jsonb overloads for JSONB path and array functions
- Add new Utility Functions section with version(), to_encrypted(),
  to_jsonb(), check_encrypted()
- Standardize terminology: change "context" to "prefix" for ste_vec
  index configuration
- Update table of contents to include new sections

This update brings the documentation in line with actual implementation
and test behavior, addressing 15+ previously undocumented functions.
- Recreate the index, OR
- Truncate and repopulate the table

**Correct order:**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens when we are using Protect? Do those functions do something special?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch.
The add_search_config function is just how the config is added when using EQL/Proxy.

For protect.js, assuming the config is setup and EQL is installed, just needs to be:

CREATE INDEX ON users (encrypted_email eql_v2.encrypted_operator_class);

Maybe these docs should mention the config in more general terms, and explicitly reference other config options?

Definitely another step here to take this reference documentation and turn into reference for protect and support content for website.

Copy link
Contributor

@freshtonic freshtonic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the prior state of the docs this is a massive improvement.

Added non-blocker comment about improving code examples.

```
-- Range comparison - use comparison operators directly
SELECT * FROM events
WHERE encrypted_date < '{"v":2,"k":"pt","p":"2023-10-05","i":{"t":"events","c":"encrypted_date"}}'::eql_v2_encrypted;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker but more of a seed to think about how we can improve this later. The raw payload is very noisy and grabs attention when what we really care about here is drawing attention to the <.

Same for the other examples.

- Fix check_encrypted() description to reflect actual validation (v,c,i fields not v,k,i)
- Update check_encrypted() behavior: raises exceptions instead of returning false
- Fix version() example to not show hardcoded version string
- Enhance JSON path operator examples with specific selector types (text, encrypted, integer)
- Add array index access example for -> operator
@tobyhede tobyhede merged commit 6ba78f6 into main Oct 20, 2025
@tobyhede tobyhede deleted the update-docs branch October 20, 2025 22:44
tobyhede added a commit that referenced this pull request Oct 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants