Problem: Existing data needs to be optimized for digest operation

[MicBun]

Existing historical data needs to be queued for processing in the new digest system.

Acceptance Criteria:

• Create backfill_pending_days action
• Implement batch processing for large datasets
• Add progress tracking and resumability
• Include performance monitoring
• Create rollback capability
• Test with production data volume
• Document backfill procedures

Performance Requirements:

• Process in batches of 10,000 records maximum
• Include progress reporting every 10,000 processed
• Limit memory usage during processing
• Allow cancellation if performance issues occur

[outerlook]

Process in batches of 10,000 days maximum

I believe it's records, right?

[MicBun]

I believe it's records, right?

yes

[outerlook]

Create rollback capability

I'm not sure about this task. Rollback in case of deletions is having a snapshot and being able to recover with a new fork in case of a disaster, right?

i.e. rolling back by inserting all back is another approach, but it seems as risky as deleting data. In any case, I'll understand this during development

[outerlook]

I'll drop this in favor of #1124

[vinarmani]

@outerlook @MicBun This appears to be the final blocker on the parent goal. Can you please give an update on the status of this task?

[outerlook]

Hey @vinarmani , I'm currently working on this.

This includes doing tests against a snapshot of the real database, and with it I'm finding some bottlenecks on performance that needs to be worked before proceeding

[MicBun]

Even though the majority of problem issues have been resolved and the code is ready, deploying it to production is challenging since prod already contains extensive data. We need to test thoroughly and avoid any risks that could impact the existing production environment