fix: repair a scheme-less WANDB_BASE_URL and stop losing W&B sink failures

[shehabyasser-scale]

Closes #52 (pending the live confirmation noted at the bottom).

The symptom

No harbor benchmark run has ever reported anything to the self-hosted W&B. Entity shehab-yasser listed zero projects. Every run's exported session/artifacts/wandb/state.json looked like this:

{"evaluation_ids": [], "next_step": 0, "run_id": "vero-a25e1cd8b7064d7e", "request_log_files": {}}

…from the 2026-07-25 swe-atlas-qna run whose session contains 11 completed evaluations. So "nothing was logged" was never "nothing happened".

The mechanism

That exact fingerprint is what SidecarWandbSink.__init__ leaves behind when wandb.init() throws. _save_state() runs at runtime/wandb.py:225, _open_wandb_run() at :230. harbor/deployment.py:265-287 catches anything out of the constructor and continues with wandb_sink = None, by design — observability must not take the eval path down.

The cause, reproduced locally at $0

$ WANDB_BASE_URL=scaleai.wandb.io python -c "import wandb; wandb.init(project='vero-egress-probe', ...)"
pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
base_url
  Input should be a valid URL, relative URL without a base
    [type=url_parsing, input_value='scaleai.wandb.io', input_type=str]

W&B parses base_url as a URL. The natural way to write a self-hosted host is rejected, and the rejection is indistinguishable from "W&B is unavailable".

Same script with https:// prepended:

INIT OK -> https://scaleai.wandb.io/shehab-yasser/vero-egress-probe/runs/vero-probe-0001
wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s)
FINISH OK

The project auto-created server-side; query_wandb_entity_projects(entity="shehab-yasser") went from [] to [vero-egress-probe]. Metrics logged, wandb.Artifact uploaded. Egress was never blocked, credentials were never missing, and the code was never unplumbed.

Note the repo's own swe-bench-pro/baseline/secrets.env.example:23 already gets it right (WANDB_BASE_URL=https://scaleai.wandb.io); the live gitignored secrets.env had drifted to the scheme-less form. Since the deployed secrets file is not in the repo, fixing only the file would leave the trap armed for the next operator.

Also worth correcting

Issue #52 is titled "Weave traces". There is no Weave in this codebase — weave is not a dependency and is not importable in the built env. wandb.log_traces drives SidecarWandbSink._log_trace, which uploads a wandb.Artifact of type evaluation_trace. So count_weave_traces would return nothing even on a perfectly healthy run; the right server-side check is whether the project exists and carries runs and artifacts.

Changes

• normalize_wandb_base_url() prepends https:// to a scheme-less WANDB_BASE_URL and warns; called before both wandb.init() sites. Values that already carry a scheme (including http://localhost:8080) pass through untouched; unset stays unset.
• The swallowed init failure now also lands in session/artifacts/wandb/init-error.json with the exception type and message. The existing logger.warning only reaches the sidecar container's stderr, which no run artifact captures — grepping every artifact of the failing run for "wandb" returns nothing. That invisibility is what made a one-line config bug survive multiple runs.

Tests: test_scheme_less_wandb_base_url_is_repaired_before_init (repair, passthrough, unset, and that a sink repairs before opening its run) and test_wandb_init_failure_is_recorded_in_the_session_artifacts (eval path survives, reason is durable).

Still to confirm

The local probe proves the URL fix and that scaleai.wandb.io is reachable from this laptop. Modal-sandbox egress to it is not yet proven. A live benchmark run on this branch is queued; I will post the resulting project URL and artifact count here before this merges.

Greptile Summary

This PR fixes a silent W&B reporting failure caused by a scheme-less WANDB_BASE_URL (e.g. scaleai.wandb.io instead of https://scaleai.wandb.io) being rejected by W&B's URL validator, and adds durable error recording so a disabled sink is no longer invisible in exported run artifacts.

• Adds normalize_wandb_base_url() which prepends https:// to a scheme-less env var and warns; called before both wandb.init() call sites (_open_wandb_run and WandbEventSink.__init__).
• Writes wandb/init-error.json to session artifacts when the SidecarWandbSink constructor fails, preserving the exception type and message in the archived run record.
• Adds unit tests covering repair, passthrough, unset, and the durable-error recording path.

Confidence Score: 5/5

Safe to merge; the URL normalization and error recording are well-isolated, and the eval path is unaffected on any failure.

The changes are tightly scoped: a pure utility function that mutates one env var idempotently, and an error-recording block that is fully wrapped in a try/except. Neither touches the evaluation engine or any data path. The only minor concern is a narrow except OSError guard that could theoretically let a non-OS exception escape, but the write operation involves only plain strings so this is not a realistic failure mode today.

Files Needing Attention: The except OSError guard in deployment.py is the one place that could be marginally tightened; all other files are straightforward.

Important Files Changed

Filename	Overview
vero/src/vero/runtime/wandb.py	Adds normalize_wandb_base_url() and calls it in both _open_wandb_run and WandbEventSink.init before wandb.init(); logic is correct and idempotent.
vero/src/vero/harbor/deployment.py	Captures the W&B init failure into wandb/init-error.json; the inner except OSError guard on the write is slightly narrower than the observability-must-never-crash-eval principle warrants.
vero/tests/test_v05_wandb.py	New test block covers scheme-less repair, already-qualified passthrough, unset, and that SidecarWandbSink normalizes before opening its run.
vero/tests/test_v05_harbor_deployment.py	New async test verifies eval path survives a sink init failure and that init-error.json is written with the correct fields.

Sequence Diagram

sequenceDiagram
    participant D as deployment.py build_harbor_components
    participant S as SidecarWandbSink.__init__
    participant O as _open_wandb_run
    participant N as normalize_wandb_base_url
    participant W as wandb.init()
    participant A as ArtifactStore

    D->>S: SidecarWandbSink(project, session_id, ...)
    S->>A: _save_state() → wandb/state.json
    S->>O: _open_wandb_run(wandb_dir, ...)
    O->>N: normalize_wandb_base_url()
    alt WANDB_BASE_URL scheme-less
        N->>N: prepend https://, warn
        N-->>O: https://scaleai.wandb.io
    else already has scheme or unset
        N-->>O: passthrough / None
    end
    O->>W: "client.init(**init_kwargs)"
    alt wandb.init() succeeds
        W-->>S: run object
        S-->>D: SidecarWandbSink instance
    else wandb.init() raises
        W-->>O: Exception
        O-->>S: Exception
        S-->>D: Exception
        D->>A: write_json(wandb/init-error.json)
        Note over D: wandb_sink = None, eval path continues
    end

Loading

_{Reviews (3): Last reviewed commit: "fix: repair a scheme-less WANDB_BASE_URL..." | Re-trigger Greptile}

[shehabyasser-scale]

Confirmed live from inside a Modal sandbox

The open question on this PR was whether Modal egress to scaleai.wandb.io works, since the $0 repro only proved it from a laptop. It does.

A swe-atlas-qna run started at 08:58:16 on a branch carrying this fix. One minute later:

query_wandb_entity_projects(entity="shehab-yasser")
 -> [{"name": "vero-swe-atlas-qna", "created_at": "2026-07-25T05:59:21Z"}, ...]

That entity listed zero projects before today. The project was created by the eval-sidecar running inside the Modal sandbox, on its wandb.init().

And it is not just an empty project — it is streaming:

run vero-974cd10513e641f9   state: running   group: swe-atlas-qna
_step: 59        47 metric keys        has_history: true
inference/producer/requests: 129        inference/producer/total_tokens: 10,670,979
inference/evaluation/requests: 284      inference/evaluation/total_tokens: 245,524
validation/agent/num_cases: 48          validation/agent/metric/error_rate: 0.854
budget/harbor-validation/agent/remaining_runs: 5
diagnostics: "infrastructure_invalidity_threshold_exceeded"
status: "invalid"

Compare the failing run's state.json: evaluation_ids: [], next_step: 0, request_log_files: {}.

So the entire SidecarWandbSink pipeline — metrics, budget ledger, per-scope inference telemetry, diagnostics — was correct all along and was disabled by one missing https://.

Worth noting what this buys beyond the ticket: those inference/*/upstream_errors series are now readable mid-run, which is how I confirmed #54 and the model half of #58 without waiting for the run to finish or extracting a tarball. The observability fix immediately became the instrument for measuring the others.

Ready to merge from my side.

[varunursekar]

why not just fix the url input?