fix: improve exception handling with try-except, debug logging, and specific error types

[tangym]

Problem

Several code paths across p2m modules could throw raw, unhandled exceptions with no context — missing file paths in FileNotFoundError, lost tracebacks in concurrent workers, and bare except Exception catch-alls that obscure the root cause.

Changes

File I/O & parse errors — Wrap unguarded read_text(), yaml.safe_load(), and json.loads() calls with specific exception handlers that include the file path in the error message:

• config.py, otel.py, tools.py, systematization_convert.py

Session imports — Catch ModuleNotFoundError and AttributeError in CallableSession.open(), OTelTracedSession.open() with actionable messages. Catch aiohttp errors in HTTPEndpointSession.run_turn().

Worker tracebacks — Add log.debug() with full traceback and seed_id in rollout and judge workers before returning errors, so tracebacks are preserved when --verbose is used.

Specific exceptions before catch-alls — Add LLMAuthError/LLMInputError/LLMRateLimitError/LLMProviderError re-raise and ValueError/KeyError/JSONDecodeError handlers before the generic except Exception in rollout and judge workers.

Phoenix collector — Catch ConnectionError separately from generic Exception with a targeted connectivity message.

Policy & systematization — Include model name, raw response text, or file path in error messages for parse failures.

Tests

21 new test cases in tests/test_exception_handling.py covering all error paths. Full suite: 565 passed, 0 regressions.

[jakepresent]

Nice work - the contextual error messages and debug tracebacks will make 1k+ scale runs much more debuggable.

One question before I approve: re-raising LLMRateLimitError and LLMProviderError from the rollout/judge workers (vs the previous catch-all-Exception path) means a single transient Azure hiccup at e.g. seed 500 of a 1k run will now kill the whole stage instead of marking one row failed. Is there retry-with-backoff above the worker layer that I'm missing, or is the expectation that callers rely on cache-resume from #32 to restart the run? Want to make sure that's intentional before signing off, especially with scaling work landing this week.

Two tiny things:

• import aiohttp inside HTTPEndpointSession.run_turn should probably hoist to module level - it's imported on every turn call. Minor.
• HTTPEndpointSessionErrorTest uses port 1 to force connection-refused; could be flaky in containers/CI environments where port 1 behavior differs. A closed high port would be more portable. Nit.

[tangym]

Thanks for the thorough review Jake!

Re: LLMRateLimitError / LLMProviderError re-raise killing the stage — this is intentional and there's a two-layer safety net:

1. Retry with backoff (PR #17, already merged): model_client.py wraps all generate calls with _with_retries — rate limits get 5 retries with coordinated per-model cooldown (honors Retry-After), provider 5xx gets exponential backoff (1s × 2^attempt, capped at 120s). By the time LLMRateLimitError reaches the worker, it's already exhausted all retry attempts.

2. Cache-resume (PR #32, already merged): If the stage does abort after max retries, p2m run picks up where it left off — completed seeds are hashed and skipped on the next run. So the user just re-runs the same command and only the failed seeds are re-attempted.

The flow is: transient failure → _with_retries handles it silently (up to 5×) → if still failing, worker raise → stage aborts → user re-runs → cache-resume skips completed seeds.

Without the re-raise, a sustained rate limit would silently mark every seed as failed (since the old catch-all just returned the error), and the user would get a full "scores" artifact with 100% errors — misleading.

Nits fixed in b551234:

• Hoisted import aiohttp from run_turn to open(), stored as self._aiohttp
• Changed test port from 1 to 59123 for container/CI portability

[jakepresent]

Thanks Yeming, that fully clears it - the _with_retries layer below the worker plus cache-resume above is the right shape, and you're right that silent-error-with-full-artifact would be worse than abort-with-resume. The fail-fast signal is more useful at the worker boundary.

Nits look good. Approving.