fix: recover expired Redis task reservations

[suguanYang]

Summary

• run expired Redis reservation recovery in an independent watchdog subprocess inside every existing worker container
• use a fresh Kombu connection for bounded, configurable recovery sweeps
• coordinate replicas through the application Redis periodic lock while retaining Kombu's broker mutex as the correctness boundary
• remove the temporary Celery Beat schedule, task route, and maintenance queue
• require both worker and watchdog heartbeats in the existing container health check
• stop colocated Beat and watchdog processes within bounded worker shutdown cleanup
• add contract coverage for recovery, watchdog resilience, heartbeat freshness, and child-process lifecycle

Why

Staging showed that hard-killed gevent work could remain in Redis's unacked set after the 4,500-second visibility window. An approved one-time call through a fresh Kombu channel restored the stranded reservations and allowed redelivery.

The original implementation scheduled that operation through Celery Beat. Review found two distributed-runtime gaps:

• a saturated gevent pool could starve the recovery task;
• mixed-version RedBeat processes could remove a static schedule unknown to the older image.

The watchdog remains outside the Celery execution pool and introduces no new AWS service or broker queue. Mixed old/new EKS and Fargate workers can therefore share the broker without unknown-task routing or RedBeat schedule conflicts.

Runtime behavior

• Every upgraded worker container starts one watchdog subprocess.
• The watchdog attempts one sweep immediately, then every 30 seconds by default.
• Each sweep makes up to 10 calls of 100 reservations each; period, batch size, and batch count are configurable.
• Replica watchdogs use the existing application Redis periodic lock. If that lock store is unavailable, recovery fails open and Kombu's broker-side mutex still serializes restoration.
• Broker failures are logged without terminating the watchdog.
• The watchdog writes its own heartbeat before and after each attempt. A dead or wedged watchdog eventually makes the existing container health check fail so the orchestrator replaces the task.
• Results report attempted rather than claiming that Kombu restored a specific count or acquired its internal mutex.

Evidence

• Staging had two expired reservations after the configured visibility window.
• An approved one-time fresh-channel Kombu restore requeued both reservations.
• Staging redelivered the parse task, and billing idempotency reported Job already charged.
• A real local watchdog subprocess configured logging, acquired the application-Redis coordination path, attempted recovery, handled SIGTERM, removed its heartbeat, and exited cleanly.

Verification

• focused worker contracts: 11 passed
• API + worker contract suite: 399 passed
• Ruff: passed
• Pyright: 0 errors, 0 warnings
• git diff --check: passed
• post-fix Standards and Spec/distributed-runtime reviews: no correctness blocker

Remaining live gates

This code does not by itself complete issue Ontos-AI/knowhere-api-infra#22. Staging still needs the deployed hard-stop/redelivery drill, duplicate-side-effect checks, mixed-replica and application-Redis-outage drills, watchdog kill/hang restart timing, backlog recovery above 1,000 reservations, and the ECS 120-second shutdown check.

This PR is review-only. Do not merge, promote to staging, deploy, or rerun the staging test job without explicit approval.