Repository navigation
v0.9.0
Per-job environment variables
lanq run --env NAME=VALUE(repeatable; single runs,--shards,--dry-run) sets a variable for this job only.
Shards get it too. The split is at the first=, and a name may be given only once (names are compared ignoring
case).- The developer decides which names jobs may set: machines.yml
policy.allowed_env, exact names or prefixes ending in
*([CONFIGURATION, TEST_FILTER, "MYAPP_*"]). It is empty by default, so every--envis refused until the
developer lists names. A name not listed isRejected(2) with anask_developerstep. - A hard deny table applies on top, even to listed names, because these variables can make an allowlisted command run
other code or move where tools keep their files: the loader and shell startup names machines.yml already refused,
PATH, Node (NODE_OPTIONS,NODE_PATH), .NET (DOTNET_STARTUP_HOOKS,DOTNET_ROOT*,CORECLR_*,COR_*, ...),
Java, Python, Perl and Ruby startup and path variables, the Go toolchain and module settings andCC/CXX/CGO_*,
Rust (RUSTFMT,RUSTUP_TOOLCHAIN,RUSTC,RUSTDOC,RUSTFLAGS,RUSTDOCFLAGS,RUSTC_WRAPPER,
RUSTC_WORKSPACE_WRAPPER,CARGO_ENCODED_RUSTFLAGS,CARGO_HOME,CARGO_TARGET_*,CARGO_BUILD_*,
CARGO_REGISTRIES_*,CARGO_SOURCE_*), build tools (JAVA_OPTS,GRADLE_USER_HOME,GRADLE_OPTS,MAVEN_OPTS,
MAVEN_ARGS,
MAKEFILES,MAKEFLAGS,DOTNET_HOST_PATH,NUGET_PLUGIN_PATHS,COMPLUS_*,DOTNET_MSBUILD_SDK_RESOLVER_*),
TLS trust and proxies (NODE_TLS_REJECT_UNAUTHORIZED,NODE_EXTRA_CA_CERTS,SSL_CERT_FILE,SSL_CERT_DIR,
REQUESTS_CA_BUNDLE,CURL_CA_BUNDLE,HTTP_PROXY,HTTPS_PROXY,ALL_PROXY,NO_PROXY), package indexes
(PIP_*,UV_INDEX_URL,UV_EXTRA_INDEX_URL,UV_INDEX,UV_DEFAULT_INDEX,UV_FIND_LINKS,UV_PYTHON), OpenSSL and glibc module paths, profile and system folders (HOME,USERPROFILE,APPDATA,TEMP,
TMP,XDG_CONFIG_HOME, ...),IFS,CDPATH, and every name startingGIT_,SSH_,NPM_CONFIG_,YARN_,
MSBUILD,LD_,DYLD_orLANQ_.NODE_ENV,GOOSandGOARCHstay allowed. Anallowed_enventry the
table fully covers (GIT_*,HOME,PIP_*,CARGO_BUILD_*,CARGO_REGISTRIES_*,CARGO_SOURCE_*) makes the
config fail to load; a prefix that only
partly overlaps (NODE_*,UV_*,CARGO_*) loads, andlanq doctorprints an informationalOKline (allowed_env) saying those names stay refused, for
examplepolicy.allowed_env CARGO_*: names on lanq's deny list (CARGO_ENCODED_RUSTFLAGS, CARGO_HOME, CARGO_TARGET_*, ...) stay refused. - A deny-table name on the command line is a
fix_commandstep ("this variable can never be set by a job; drop it or
ask the developer: ..."); fromlanq.ymlit isask_developer("the developer drops it from that job's env: in
lanq.yml"). - Each value must be non-empty and free of control characters, at most 4,096 characters, and 16,384 for all of a job's
variables together (NAME=VALUEeach). Values are never echoed in a refusal. - A machine whose own
env:in machines.yml sets the same name is not used for the job (env_conflict), and neither
is a Windows machine on which lanq's largest job script plus the values would pass 8,000 characters
(windows_command_too_long). Both are permanent and judged from the config. When every capable machine clashes,
the run ends at once withNoCapacity(75,retryable: false) and anask_developerstep ("ask the developer:
they can drop --env CONFIGURATION or remove it from the machine's env (build-01, big-01)"); when the length rules
out the last ones, it isRejected(2), naming each clashing machine too, with anask_developerstep ("they can
shorten or drop the --env values, or change machines.yml"; for a plan, "that job's env: values in lanq.yml"). An
agent stops and reports these; it never edits the config or drops variables to get through. lanq.ymlgainsenv:in a job and indefaults:, a mapping ofNAME: value. It is merged by name, the job's value
winning: the one exception to "a job's value replaces the default". Each entry becomes an--envflag.- Variables go after the machine's
env:and before lanq's ownLANQ_*, quoted as data like the machine's.
Exclusive jobs
lanq run --exclusive(lanq.ymlexclusive: true) takes every slot of the machine (max_jobs), all or nothing, so
no other lanq job, from you or anyone else, runs there while it does.- It waits while any live slot is in use (
busy, retryable: "busy (slots 1/4; --exclusive needs all 4 free)"). There
is no reservation, so on a machine that is never idle it waits out--waitand endsNoCapacity(75). Use--onwith a
machine that is idle, or run it in quiet hours. --exclusive --shards Nputs one shard on each machine;--shards autocounts the idle machines. A fixed N above the
machines that can take the job by config (capabilities,size:,--env) is refused (2) before any network.
--memand Windows arguments are not part of that count, so extra shards wait for a machine that is idle.- In a plan, the dry run marks an exclusive job's machine full, so a sibling job waits.
- Each slot of an exclusive job is an ordinary slot file, so
lanq unlockand the stale-slot reclaim handle it like any
other (slot release itself now checks the holder; see Security). - An exclusive claim also backs out when a live
slot-Nabove its ownmax_jobsexists: another developer's
machines.ymlwith a highermax_jobsfor the samework_roothas a job there. That is one look at claim time, so
such a developer can still start a job in a higher slot afterwards: everyone sharing a worker should use the same
max_jobs. - To fit the Windows command-line limit, every Windows slot claim (normal and exclusive) carries its fixed PowerShell
slot functions compressed (deflate, base64) and dot-sources them in memory, as the helper install already does. Some endpoint-protection tools flag that
pattern; the inflated text is lanq's constant claim code, and AMSI scans it as such.
Baseline rerun: is this failure new?
lanq run --baseline(lanq.ymlbaseline: true): when a job, a shard or a plan job endsFailed(its own non-zero
exit), lanq runs the same request again at the base commit, a clean tree, and says whether the failure is new.
Never afterTimedOut,WindowClosed,InfraError,NoCapacity,RejectedorCancelled.- The base is the merge-base of HEAD and the default branch (
origin/HEAD, elseorigin/main,origin/master,
main,master).--baseline-ref REF(lanq.ymlbaseline_ref:, implies--baseline) uses the merge-base of HEAD
and REF instead. On the base commit with uncommitted changes, the base is HEAD without them; with none, the baseline
isskipped. Refused (Rejected, 2) before anything is sent: a REF that is empty, over 256 characters, has control
characters, starts with-or contains..; a REF that is not a commit; a repository with no commit; no default
branch; unrelated history. A shallow clone that lacks the merge-base is not refused: the baseline isinconclusive. - lanq.yml
baseline_ref:must name a branch or tag (underrefs/heads/,refs/remotes/orrefs/tags/): a commit
id,HEAD/*_HEADand revision characters are refused ("baseline_ref must name a branch or tag (a commit can be
given on the command line with --baseline-ref)"), so a repository's file cannot make lanq send any old tree to an
archive worker.--baseline-refon the command line still takes any commit. A short name must exist under exactly
one of those (baseline_ref NAME is both a branch and a tag; write refs/heads/NAME or refs/tags/NAME,baseline_ref NAME names more than one ref (refs/heads/NAME, refs/remotes/NAME); write one of them); a fullrefs/heads/NAMEor
refs/tags/NAMEis accepted; git's short formsheads/main,tags/v1.0andremotes/origin/main, which git would
accept, are refused with the full ref to write (baseline_ref heads/main: write refs/heads/main). The baseline'srefand label show the name as written. - How the keys combine: a job's own
baseline: falsewith its ownbaseline_ref:is an error; its ownbaseline: false
drops an inherited ref quietly; its ownbaseline_ref:turns the baseline on even whendefaults:sayfalse; its
ownbaseline: trueunderdefaults:withbaseline: falseand a ref gives a plain--baseline. - The verdict is
new(the base completed),preexisting(the base failed too, any exit code),inconclusive
(anything else: timed out, no machine, the base could not be read or sent, Ctrl+C) orskipped. It is information
only: the run's status and exit code never change. - Shards: only the failed shards rerun, with the same
LANQ_SHARD/LANQ_SHARDS. Plans: only the failed jobs that
asked for it, after the whole plan has ended. Each rerun gets its own--wait. - A rerun goes to the machine the job failed on: while that machine could take it but is busy, it waits up to 2 looks
(about 15 s each, never past what is left of its--wait) for a slot there, then takes any eligible machine; with
--wait 0(the default) it does not wait for it. A held or ruled-out machine is not waited for. On another machine,
baseline.sameMachineisfalseand the message and thenewreason end(on big-01, not build-01 where it failed: the result may reflect the machine). Rerun lines are prefixed[baseline],[baseline 2/4]or
[baseline unit 2/4]([baseline] [lanq] waiting for a slot on build-01, where the job failed, before taking another machine (look 1 of 2), or... (the other baseline reruns hold some)while sibling reruns hold slots). - Ctrl+C: a rerun not yet started is
inconclusive: cancelled (Ctrl+C) before it startedon every path (single,
shards, plans), withstatusnullbefore it began waiting andCancelled(130) while it waited or claimed a slot;
a running one is stopped and cleaned up (Cancelled, 130, inconclusive). A rerun that finds no machine is
NoCapacity(75), inconclusive. - A rerun of an older commit leaves the worker's newer stored base (and
bases.json) alone, so the next normal job
still gets a small incremental bundle. - Output: a
[lanq] baseline: ...line per rerun before the status line, abaselineobject in--json(run, each
shard, each plan job), and the verdict in thefix_codereason.--dry-runprints the base (Baseline: on failure, rerun at 1a2b3c4 (merge-base with origin/main) (a dry run runs nothing); the JSON message has no suffix). When the
merge-base is the branch tip the base is named by the branch:at 1a2b3c4 (origin/main). - The base commit's payload gets the same committed-name and package-config secret scans as any snapshot (a hit:
inconclusive: the base commit was not sent: Refused: possible secrets in payload (...)). On an archive worker the
rerun receives the base commit's tree, an older version of the files. - A red run costs about twice as long with
--baseline.
JSON and records
- Run, shard and plan job objects (and the dry runs) gain
env(the variable names only, never values),
exclusive: trueandbaseline({ref, sha, status, exitCode, machine, jobId, logPath, durationSeconds, verdict, message}, plussameMachinewhen the rerun ran somewhere). Each is left out when it does not apply. - New permanent reason codes
env_conflictandwindows_command_too_long. lanq health --jsonrows gainslotsStale(only when above 0), andfreeSlotscounts live slots only; the text
line adds(4 stale: reclaimed by the next claim)and busy reasons, 1 stale: reclaimed by the next claim.jobs.jsonlgains optionalenvNames,exclusive, and on baseline rerunsbaselineOf(the failed job's id) and
commit(the base's short id);vstays 2.lanq stats,--target-minutesand the plan's history order leave
baseline reruns out.- Like the other job options,
--env,--exclusive,--baselineand--baseline-refare refused with--plan
(set them inlanq.yml).
Security
--envis gated bypolicy.allowed_env(empty by default) and the hard deny table, and eachNAME=VALUEgoes
through the secret scan (Rejected2, value never echoed). A repository'slanq.ymlcan set only names the
developer allowed. The table is a guard against the obvious ways a variable can turn an allowlisted command into
other code; it is not a sandbox.- Values are visible to other users of the worker (its process list) and appear in the dry run and in suggested
commands innextSteps, which repeat your command line. Never pass secrets with--env. - Slot release now checks the holder: cleanup removes each slot (a normal job's single slot too) only while its first
line is still this job's id, so a late cleanup can no longer delete a slot another job reclaimed meanwhile. - A claim that did not report back (timed out, lost connection, unreadable output, Ctrl+C mid-claim) is followed by a
best-effort release of the slots that hold this job's id (15 seconds, never fails the job).
Fixes
--shardswith--mem: a shard that started while lanq was probing the machines was counted as running, as if its
memory showed in the probe's answer, although the machine answered before it ran. The next round could then place
one shard too many on that machine. Which shards are not running yet is now taken before the probe.- Stale slots (a crashed run's, more than 30 minutes past their deadline) no longer make a machine busy. The probe
reports them (slots_stale=), placement andfreeSlotscount only live slots, and the next claim reclaims them as
it passes. Before, one dead job's slot kept a machine busy for every--exclusivejob for ever, and a dead
exclusive job's slots blocked the machine for everyone until something reclaimed them.lanq healthshows them as
(4 stale: reclaimed by the next claim). - A stale slot a claim cannot reclaim counts as busy for the rest of the run. The claim reports it
(RECLAIM=STUCK slot-N) only when the slot's.reclaimlock is older than about a minute (a dead reclaimer's), no lock
could be made, or the file could not be moved or deleted; a younger lock (a reclaim in progress) or a claim that
simply lost the slot to another job never marks the machine. lanq prints one line per machine, counting the slots
the claim named:[lanq] build-01: 1 stale slot could not be reclaimed (a .reclaim lock or file permissions); counting it as busy for this run; lanq unlock build-01 clears it. Busy reasons then readbusy (slots 2/2, 1 stale, not reclaimable). On Windows an unreadable slot abovemax_jobsnow counts as live, as on Linux. - Linux workers: stopping a stale job from inside the stale-slot reclaim or
lanq unlockoverwrote the loop's own
variables, which could miss the reclaim, leave a.reclaimlock behind and report a slot asREMOVED=that was not.
Tests
- Two tests that failed now and then on GitHub's Windows runner are now deterministic. The
--memplacement test's
fake machine now reports less free memory once a shard's upload is done, as a real job would use it. The newline-free flood test now measures the reader's
memory on one thread, not process-wide, where work left by earlier tests counted too. - The exclusive-claim race tests run the real sh scripts behind a ready/go barrier, including an exclusive claim that
must back out while two normal claims race it. - The plan test for
--waitcounting only while no plan job holds a slot (WaitCountsOnlyWhileNoPlanJobHoldsASlot)
no longer depends on timing: it releases the first job only after the second has said it waits.
Dependencies
- The native binaries carry the .NET 10.0.15 runtime (was 10.0.14).
Docs
- README sections for
--env,--exclusiveand--baseline:
https://github.com/jangolath/lanqueue/blob/main/README.md - Plan keys
env,exclusive,baselineandbaseline_ref: https://github.com/jangolath/lanqueue/blob/main/docs/PLANS.md - The new JSON fields and reason codes: https://github.com/jangolath/lanqueue/blob/main/docs/JSON.md
- What an agent may and may not do with each: https://github.com/jangolath/lanqueue/blob/main/AGENTS.md and
https://github.com/jangolath/lanqueue/blob/main/docs/AGENTS-GUIDE.md (also the snippets, the skill and llms.txt). machines.example.ymlshows a commentedpolicy.allowed_env.
Full changelog: https://github.com/jangolath/lanqueue/blob/main/CHANGELOG.md
What's Changed
- v0.8.1: lanq-makeready 0.3.0, safe .NET SDK suggestions by @jangolath in #14
- Deflake two CI tests; count not-running shards before the probe by @jangolath in #15
- Pin the .NET 10.0.15 runtime for the native binaries by @jangolath in #16
Full Changelog: v0.8.1...v0.9.0