feat: add ability to enable mutex profile

[sambhav-jain-16]

What?

Add the ability to enable mutex profile fraction in all services of Kraken

Why?

As of now, there's no way to enable mutex profile contention. As we are exploring different types of hosts for kraken we need to verify the performance changes by profiling. For debugging, a mutex profile is needed rather than taking a trace every time; it is much cleaner and provides clear code paths for the contending mutexes.

Why not enable it by default without using a flag?

mutex profile has an overhead since every mutex action needs to be sampled (decided by the rate provided) ref: https://cs.opensource.google/go/go/+/go1.24.0:src/runtime/mprof.go;l=892. We will use this flag to selectively enable mutex profile

Testing

Enabled by default in devcluster
Output:

> curl http://localhost:16002/debug/pprof/mutex\?debug\=1
--- mutex:
cycles/second=2449999982
sampling period=1
2770415 29 @ 0x9d1fa1
#       0x9d1fa0        runtime._LostContendedRuntimeLock+0x0   /usr/local/go/src/runtime/proc.go:5444

> curl http://localhost:15003/debug/pprof/mutex\?debug\=1
--- mutex:
cycles/second=2450000025
sampling period=1
162315 6 @ 0x4504a1
#       0x4504a0        runtime._LostContendedRuntimeLock+0x0   /usr/local/go/src/runtime/proc.go:5444

[Copilot]

Pull request overview

Adds a new CLI flag to enable Go mutex contention profiling across Kraken services, and turns it on in the devcluster startup scripts to make /debug/pprof/mutex data available for performance investigations.

Changes:

• Add --mutex-profile-fraction flag to agent/origin/proxy/tracker/build-index and apply it via runtime.SetMutexProfileFraction.
• Enable mutex profiling by default in devcluster process/container startup scripts.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
agent/cmd/cmd.go	Adds flag + sets mutex profile fraction during service startup.
origin/cmd/cmd.go	Adds flag + sets mutex profile fraction during service startup.
proxy/cmd/cmd.go	Adds flag + sets mutex profile fraction during service startup.
tracker/cmd/cmd.go	Adds flag + sets mutex profile fraction during service startup.
build-index/cmd/cmd.go	Adds flag + sets mutex profile fraction during service startup.
examples/devcluster/herd_start_processes.sh	Enables mutex profiling (fraction=1) for multiple devcluster processes.
examples/devcluster/agent_one_start_container.sh	Enables mutex profiling (fraction=1) for the agent container.
examples/devcluster/agent_two_start_container.sh	Enables mutex profiling (fraction=1) for the agent container.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The mutex profiling flag parsing + runtime.SetMutexProfileFraction wiring is duplicated across multiple cmd packages (agent/build-index/origin/proxy/tracker). To reduce future drift (e.g., if help text/validation changes), consider factoring this into a small shared helper (e.g., in utils/) that registers the flag and applies it early in Run.

[gkeesh7]

The CLAUDE.md debugging section should be updated to document the new --mutex-profile-fraction flag and the /debug/pprof/mutex endpoint. This helps contributors discover the profiling capability without digging through CLI flags.

[gkeesh7]

One Comment about updating the Claude.md rest LGTM.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.