Skip to content

OpenConceptLab/ocl_issues#2599 | Performance improvements for Expansion concepts/mappings indexing - #888

Merged
snyaggarwal merged 9 commits into
masterfrom
issues#2599
Jul 2, 2026
Merged

OpenConceptLab/ocl_issues#2599 | Performance improvements for Expansion concepts/mappings indexing#888
snyaggarwal merged 9 commits into
masterfrom
issues#2599

Conversation

@snyaggarwal

Copy link
Copy Markdown
Contributor

Linked Issue

Closes OpenConceptLab/ocl_issues#2599

Summary

Collection Resources Full re-index Partial update Speedup
gigantic-collection/1 2,841 concepts + 12,545 mappings 194s 4.7s 42x
PEPFAR MER_FY22/v1.0 3,073 concepts + 15,006 mappings 184s 3.9s 47x

@snyaggarwal
snyaggarwal requested a review from paynejd July 2, 2026 05:42
@snyaggarwal snyaggarwal self-assigned this Jul 2, 2026

@paynejd paynejd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ~40x speedup is legitimate — partial field-append via Painless script vs. re-serializing the whole concept/mapping document. And the URL-derivation / partial-update approach matches the tasks in #2599. But I found two correctness regressions the benchmark and tests don't cover (both removal-related and value-equivalence), plus one edge note. Details inline.

Comment thread core/collections/models.py Outdated
Comment thread core/collections/models.py
Comment thread core/collections/models.py Outdated

@paynejd paynejd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more minor efficiency note inline (the correctness item — HEAD collection_url — is in the thread above).

Comment thread core/common/mixins.py Outdated
@snyaggarwal
snyaggarwal merged commit 6ac3a2a into master Jul 2, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed up Expansion indexing by using partial doc update in ES

2 participants