Skip to content

Conversation

@skwowet
Copy link
Member

@skwowet skwowet commented Nov 4, 2025

Changes

Before:

  • Query building created individual clauses for each identity (60 orgs, 75 members)
  • Hit OpenSearch maxClauseCount limit (1024) with organizations having 70+ identities
  • Sequential query construction without deduplication
  • No chunking strategy for large identity sets

After:

  • Increased identity processing limit (100 for both orgs and members)
  • Categorize identities into query types before building OpenSearch queries (weak/fuzzy/prefix for orgs; verified/unverified exact/fuzzy for members)
  • Deduplicate fuzzy and prefix values before querying
  • Chunk queries into groups of 20 to stay under clause limits
  • Extract common chunkArray utility for reuse
  • Use filter instead of must for tenant filtering (performance optimization)
  • Change error logging from info to error for better visibility

Fixes: Organizations with 70+ identities now process successfully without hitting OpenSearch clause limits while maintaining query quality.

@skwowet skwowet self-assigned this Nov 4, 2025
@skwowet skwowet requested a review from themarolt November 6, 2025 11:25
@skwowet skwowet merged commit 957f676 into main Nov 6, 2025
18 of 19 checks passed
@skwowet skwowet deleted the refactor/organization-merge-suggestions branch November 6, 2025 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants