Move ingestion and scanner work to durable jobs/workflows

[ishaanxgupta]

We performs ingestion/scanning through request-time waits, asyncio.create_task, local cancellation flags, and singleton process pipelines in memory.py and scanner.py.

Acceptance criteria:

• Introduce durable job records with status, retry count, timeout, and error state.
• Add queue/workflow worker for ingest, batch ingest, scraping, and scanner jobs.
• Make jobs idempotent by content hash/repo commit/user/org.
• Add dead-letter handling and status APIs.

Bounty 30$ API Credits

[ishaanxgupta]

Phase 1 - Task Queue: Introduce a task queue (Celery + Redis). The /v1/memory/ingest endpoint should: Validate and enqueue the job, returning a job_id immediately, Workers process the LangGraph pipeline asynchronously, Add a GET /v1/memory/ingest/{job_id}/status endpoint
Phase 2 - Step Persistence: Store intermediate state between LangGraph nodes (classification result, extraction results, judge decisions) in Redis. If a step fails, resume from the last checkpoint.
Phase 3 - Retry + Timeout: Wrap each agent call with asyncio.wait_for(timeout=...) and add exponential backoff retries. We already has some timeout logic from a previous conversation but it needs to be systematized.