Implement retry limits to prevent ingestion pipeline blockage

[Copilot]

Sequential ingestion gets stuck indefinitely on a single bad message, blocking all subsequent sources from processing.

Changes

Schema

• Added retryCount?: number to InternalMessageSchema to track failed ingestion attempts

Ingestion logic (from-sources.ts)

• MAX_RETRY_ATTEMPTS = 3 - sources exceeding this limit are skipped
• getAlreadyIngestedSet() now returns { alreadyIngestedIds, maxRetriesReachedIds } for filtering
• incrementRetryCount() updates Firestore on actual failures (excludes boundary/filter outcomes)
• Added maxRetriesReached to summary logging

Example flow

// Attempt 1: FAIL → retryCount = 1
// Attempt 2: FAIL → retryCount = 2  
// Attempt 3: FAIL → retryCount = 3
// Next run: SKIPPED (logged as warning)

Configuration

• Removed shared/ from lint-staged (schema-only package, no eslint config needed)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Ingestion is sequential which means it is blockable</issue_title>
<issue_description>Since the ingestion process is sequential, if a message fails to ingest and fails the process the next time the process runs it will attempt to process the same message. This means that the process may go completely stuck because of one wrong/bad message,

We should only retry one message a limited number of times.

We could try and see what we have stored in the process field to determine how many times a message has been attempted. If such info is not available in plain in process we could define a separate field where we store that.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Ingestion is sequential which means it is blockable #149

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
oborishte-map	Skipped		Feb 12, 2026 3:58pm