Reference | Delegated

Delegated builds

Available since Xake 3.2

Delegated builds let an external system run rules across many machines, share results, and deduplicate concurrent demands — while every existing local build stays unchanged.

The guiding principle: distribution is a property of a rule, not a new verb. Script authors keep writing need ["target"]; whether a target runs locally, remotely, or is fetched from a shared store is decided at rule-definition time.

Overview

Three opt-in mechanisms work together:

Mechanism	Purpose
delegated executor	Delegates a rule's up-to-date check and execution to a caller-supplied function
runDetached	Releases the CPU slot for I/O-bound waits in non-delegated rules
Resource	Throttles concurrent access to a shared budget

Resources are covered in detail on the Resources page.

Delegated rule hook

delegated wraps any rule so its execution is delegated to a caller-supplied executor:

"run-suite:*" => recipe {
    do! runSuite()
} |> delegated myExecutor

The executor type:

type DelegatedExecutor<'ctx> =
    'ctx -> Target list -> (unit -> Async<BuildResult>) -> Async<BuildResult>

It receives:

• the execution context
• the target list
• a body thunk that runs the inner recipe locally

The executor owns the full rebuilder decision:

1. Compute an identity key for the target
2. Look it up in a shared result store
3. On a hit — return the stored result without calling body
4. On an in-flight match — join the existing execution
5. On a miss — call body, publish the result, return it

Writing an executor

A minimal executor with an in-memory store and in-flight dedup:

let makeExecutor (budget: Resource) =
    let store = ConcurrentDictionary<string, BuildResult>()
    let inFlight = ConcurrentDictionary<string, Lazy<Task<BuildResult>>>()

    fun ctx targets runBody ->
        let key = identityOf targets
        async {
            match store.TryGetValue key with
            | true, cached -> return cached
            | _ ->
                let fresh = lazy (
                    (async {
                        return! Resource.withAcquired budget 1 (async {
                            let! result = runBody()
                            store.[key] <- result
                            return result
                        })
                    }) |> Async.StartAsTask)

                let entry = inFlight.GetOrAdd(key, fresh)
                let! result = entry.Value |> Async.AwaitTask
                inFlight.TryRemove key |> ignore
                return result
        }

How the engine runs a delegated rule

• The rule still goes through the worker pool, so local per-target dedup by target name is preserved.
• The NeedRebuild gate is skipped — the executor is the rebuilder.
• The entire task runs detached — it never holds a CPU slot. Concurrency is governed by whatever dispatch Resource the executor uses.
• The returned BuildResult is stored in the local .xake database so downstream need calls resolve correctly.

runDetached is redundant in delegated rules

The engine already detaches delegated rule execution. Use runDetached only for non-delegated I/O-bound rules:

// non-delegated rule — needs runDetached to avoid pinning a CPU slot
"slow-io" => recipe {
    do! runDetached (recipe {
        do! awaitRemoteResult()
    })
}

// delegated rule — already detached, runDetached is unnecessary
"test:*" => recipe {
    do! runSuite()
} |> delegated executor

Detached execution

runDetached releases the CPU slot before the body runs and reacquires it after. Many more detached rules than CPU cores can be in flight at once:

runDetached : Recipe<ExecContext,'b> -> Recipe<ExecContext,'b>

Use it for non-delegated rules that block on I/O. Wrap in a Resource to cap the concurrency:

let remoteSlots = Resource.newResource "remote" 10

"fetch:*" => recipe {
    do! runDetached (recipe {
        do! withResource remoteSlots 1 (recipe {
            do! fetchFromRemote()
        })
    })
}

Sharing resources between executors

A Resource is a standalone object. Pass the same instance to multiple executors to share a dispatch budget:

let budget = Resource.newResource "dispatch" 5

let compileExecutor = makeExecutor budget
let testExecutor    = makeExecutor budget

rules [
    "build:*" => recipe { ... } |> delegated compileExecutor
    "test:*"  => recipe { ... } |> delegated testExecutor
]

Both executors compete for the same 5 slots. Different stores, different dedup, shared budget.

Incremental discovery

Targets discovered in waves all flow through the same executor instance:

"test:(name:*)" => recipe {
    do! runSuite()
} |> delegated executor

command "run-all" {
    let! wave1 = discoverTests "fast"
    do! need wave1                       // 50 tests dispatched

    let! wave2 = discoverTests "slow"
    do! need wave2                       // 50 more, same executor queue
}

Wave 2 targets join the same executor's store and in-flight map. If a target from wave 1 is still running when wave 2 requests it, the executor deduplicates. If it already finished, the store returns a cache hit.

Note that do! need [...] is a barrier — wave 2 starts only after wave 1 completes. For overlapping dispatch, demand batches from separate rules connected with <==.

Two dedup layers

Layer	Scope	Key
Worker pool	Local process	Target.FullName
Executor	Cross-target, cross-process	Identity key (caller-defined)

These are orthogonal. The executor's dedup is the one that spans machines.

Composing with requiring

delegated and requiring compose via |>:

"build:*" => recipe { ... }
    |> requiring compileLock
    |> delegated executor

Samples

Build the library first: dotnet fsi build.fsx -- -- build

Sample	What it shows
delegated.fsx	All three mechanisms: delegated rule with external process, in-memory store with cache hit and in-flight dedup, runDetached for a non-delegated rule, dispatch Resource
delegated-tests.fsx	100 test suites via one pattern rule, dispatch budget of 20. With THREADS=4, peak concurrency = 20 — proves delegated rules are not bounded by the CPU pool
delegated-proc.fsx + worker.fsx	Two-process variant: orchestrator dispatches to separate worker processes. Phase 1 spawns workers; phase 2 serves from cache with no workers
resources.fsx	Resource patterns without delegated execution: download throttle, restore mutex, DB write lock