Optimize HEAD export generation using previous export plus updated resources diff

[filiperochalopes]

User story

As a CIEL Lab maintainer, I want HEAD exports to be generated faster by reusing the previous HEAD export and applying only changed resources so nightly jobs and manual test refreshes do not depend on a full export processing every time.

Use case

CIEL Lab currently uses HEAD export to populate/update its local data in a nightly job. In staging/production, generating a HEAD export can take at least ~2 hours, which creates a bottleneck when the refresh must be triggered manually multiple times during testing.

Requirements

• Optimize HEAD export generation by building from the previous HEAD export when possible.
• Use the existing export content as the cold data baseline.
• Detect whether HEAD changed since the previous export.
• Reuse IDs, timestamps, and checksums already present in the previous export.
• Fetch only resources updated since the previous export timestamp using OCL API.
• Apply changed concepts and mappings to the previous export content to generate a new HEAD export ZIP.
• Preserve existing IDs and checksums correctly in the generated export.
• Identify and document any missing OCL API capabilities needed to fetch changed resources efficiently.
• Keep the current full HEAD export path as a fallback when no previous export exists or incremental generation is not possible.

Acceptance criteria

• Given a previous HEAD export exists and HEAD has changed, when a new HEAD export is requested, then the system generates it by reusing the previous export and fetching only updated resources.
• Given no previous HEAD export exists, when a HEAD export is requested, then the system falls back to the full export process.
• Given updated concepts or mappings exist since the previous export, then the new export includes those updates.
• Given unchanged resources exist in the previous export, then they are preserved without being rehydrated unnecessarily.
• The optimized process is measurably faster than the current full HEAD export process for CIEL in staging/production.
• Any API gaps required for efficient incremental export generation are documented as follow-up tasks.

If I had a webhook available, I could already handle some of these updates locally instead of depending on a refresh process based on the export file.

More details (collapsible)

Context

CIEL Lab currently uses HEAD export to populate local data through a nightly job. This creates a bottleneck, especially during testing when the refresh must be triggered manually more than once.

Recent observed timing:

• staging HEAD export on 2026-05-31 01:44:06 UTC: 7218s
• production/staging: currently at least ~2h
• local environment: ~20min

Related permission improvements:

• Repo HEAD export should be allowed by Repo Authors #2571

Slack reference:

• https://openmrs.slack.com/archives/C0A99PV25NG/p1780920825418859

Implementation direction discussed:

• The first export remains expensive.
• Subsequent exports should be generated from an existing export plus changed resources.
• Timestamps, IDs, and checksums are already present in exports.
• OCL API should be used to fetch resources updated since a timestamp.
• Any gaps in the API should be handled as small follow-up enhancements.

[ocl-automation]

Automated Triage — 2026-06-17

Dimension	Score
Complexity	4
Scope	4
Clarity	3
Dependency	2
Risk	3

Decision: decision/planning-needed
Type: type/feature
Signals: signal/large-scope

Complex optimization requiring differential processing logic and coordination across export generation, storage, and job systems. Moderate clarity as the goal is clear but implementation approach and edge cases (like handling deletions or corrupted diffs) are unspecified.

---

Generated by ticket-triage pipeline v3. Model: anthropic/claude-sonnet-4 (openrouter).

[filiperochalopes]

Probably it will be a won't fix for now because of priorities on the backlog, at least not of these requirements. But quick wins were accomplished. Should we close it or put in some sort of inactive archive/backlog? @snyaggarwal @jamlung-ri

[filiperochalopes]

/uat @jamlung-ri

[snyaggarwal]

@filiperochalopes we can keep this ticket in backlog

[filiperochalopes]

Not this approach, but we had performance improvement. Keep in Backlog @snyaggarwal says