Git marketplace refresh starts a full clone per concurrent `codex exec` process

[stan-lu]

Summary

When a Git-sourced marketplace is enabled, concurrent codex exec launches each start their own full Git clone of the same marketplace into a separate marketplace-upgrade-* staging directory.

This happened before the agent prompt was processed. A workload that spawned many short-lived, read-only codex exec --json processes amplified the behavior into dozens of concurrent HTTPS connections and several gigabytes of duplicate temporary data.

The update/materialization itself may reasonably use a staging directory for atomic replacement. The problem is that concurrent processes do not appear to share a refresh result or coordinate with a cross-process lock, and unchanged revisions are not cheaply rejected before the full clone.

Environment

• macOS on Apple Silicon
• Codex Desktop 26.721.81911, using its embedded Codex CLI
• Git-sourced marketplace configured in ~/.codex/config.toml:

[marketplaces.compound-engineering-plugin]
source_type = "git"
source = "https://github.com/EveryInc/compound-engineering-plugin.git"
ref = "main"

[plugins."compound-engineering@compound-engineering-plugin"]
enabled = true

The installed marketplace metadata had a persisted last_revision, and the on-disk marketplace clone was already present under ~/.codex/.tmp/marketplaces/compound-engineering-plugin.

Observed behavior

Each child process independently ran the equivalent of:

git clone https://github.com/EveryInc/compound-engineering-plugin.git \
  ~/.codex/.tmp/marketplaces/.staging/marketplace-upgrade-<random>

Observed during one incident:

• approximately 50 concurrent git clone processes / 47 active GitHub HTTPS connections
• all clones targeted the same repository and main ref
• ~/.codex/.tmp/marketplaces/.staging reached about 4.3 GB
• hundreds of marketplace-upgrade-* directories accumulated while the parent workload was still launching new codex exec processes

After the parent workload stopped, the remaining clone children were orphaned (PPID 1) and needed manual termination before the temporary directories could be cleaned safely.

Expected behavior

For each configured Git marketplace:

1. Use a lightweight ref check (for example git ls-remote) and compare it with the stored revision before downloading the marketplace payload.
2. When the ref is unchanged, reuse the existing local materialization with no full clone.
3. When an update is required, use a per-marketplace cross-process lock / single-flight refresh so one process performs the staging download and other concurrent CLI launches wait for or reuse its result.
4. Keep the existing atomic staging-and-swap behavior only for the one actual update.

This is especially important because codex exec is commonly launched programmatically and in parallel by test runners or orchestration tools.

Why this matters

A single accidental parallel workload can multiply one marketplace refresh into tens of identical full repository downloads, saturating bandwidth and consuming gigabytes of disk. The behavior is independent of whether the agent run itself is read-only.

Related issues

• Plugin catalog cache is rewritten in full after every plugin/list — no TTL, no content check, ~65 GB/day of disk writes #35401 reports repeated plugin-catalog refreshes/writes without TTL or content comparison. This report is specifically about the Git marketplace upgrade/materialization path and concurrent full clones.
• OpenAI-curated plugin manifests are repeatedly validated during normal turn/subagent setup #19382 reports repeated marketplace manifest work during normal turn/subagent setup.
• Plugin cache refresh ignores marketplaces defined in config.toml #23902 notes that Git-sourced marketplaces use a separate marketplace_upgrade refresh path.

I can provide sanitized process listings and staging-directory measurements if useful.

[taegyun-rapportlabs]

Confirming the same cross-process/full-clone behavior through an Orca integration path, with the parent/child lifecycle captured live on macOS.

• Orca 1.4.167
• Codex CLI 0.146.0
• One private Git marketplace, ~305 MB per full clone
• 580 orphaned ~/.codex/.tmp/marketplaces/.staging/marketplace-upgrade-* directories, approximately 150 GB

Orca periodically spawns this short-lived process for its usage status bar:

/opt/homebrew/bin/codex -s read-only -a untrusted app-server

It initializes JSON-RPC, sends account/rateLimits/read, and terminates the app-server as soon as the response arrives. Codex startup has already launched marketplace auto-upgrade on a background thread. The child git clone ... marketplace-upgrade-XXXXXX then survives with PPID 1, while the process owning the Rust TempDir is gone; activation/config revision persistence and cleanup never run. The next quota poll sees the old revision and repeats the clone.

A foreground codex plugin marketplace upgrade <name> --json completed successfully, aligned the live checkout/config/install metadata with remote HEAD, and a controlled short-lived app-server -> account/rateLimits/read -> terminate test then produced zero new staging directories. This confirms that the repeated full clone was caused by the interrupted startup upgrade rather than marketplace contents.

Detailed Orca-side report and process tree: stablyai/orca#7725 (comment)

This path reinforces the need for a cross-process per-marketplace lock/single-flight refresh, stale staging GC, and shutdown coordination that does not rely only on process-local TempDir drop.