contextlake 8.9.0
Added
-
AWS and Azure adapters.
--platform awscreates an EventBridge Scheduler
schedule firing an ECS task;--platform azurecreates a Container Apps Job on
a cron trigger. Both shell out to an already-authenticatedawsoraz, so
contextlake still ships no cloud SDK. On EKS and AKS use--platform k8s
instead: both are Kubernetes, so theCronJobadapter serves them and brings
concurrencyPolicy: Forbidwith it. Neither cloud service has an equivalent of
Forbid, so two runs there can overlap and the second skips on the store's
advisory lock rather than never starting, which both adapters report. They round
differently by design: EventBridge takesrate(N minutes)and rounds to whole
minutes, while a Container Apps Job trigger is a cron expression and rounds the
way cron does. Registered but never auto-detected, for the same reason as the
Kubernetes adapter. Verified by asserting the rendered request documents and the
exact CLI arguments: there is no account here, so neither is verified by
execution. -
A Kubernetes adapter, covering OpenShift as well. Renders a
CronJoband
applies it withkubectl, falling back tooc. One adapter serves both,
because OpenShift is Kubernetes with a stricter default security context and
the manifest satisfies the stricter one: norunAsUser, since the restricted
SCC assigns an arbitrary UID and rejects a pinned one, plusrunAsNonRoot,
a dropped capability set and anfsGroupso the mounted state directory stays
writable.concurrencyPolicy: Forbidgives single-writer semantics from the
cluster, so the second of two overlapping runs never starts. State mounts a
PersistentVolumeClaimrather than anemptyDir, because an ephemeral store
re-indexes the whole fleet every run. The schedule is a cron expression and
rounds through the same function the cron adapter uses. Nothing is patched in
the cluster on its own: changing an interval means re-running
schedule install, since a background rewrite would need cluster-write rights
for the life of the schedule. Reachable with--platform k8sand never
auto-detected:kubectlon a PATH does not mean a schedule belongs in that
cluster. -
A Windows adapter, so
contextlake scheduleworks through Task Scheduler.
Creates a task withschtasks /SC MINUTE /MO nunder a\contextlakefolder.
Two limits are reported rather than hidden./MOcounts whole minutes, so an
interval is rounded the way cron's is, down above a minute and up below it, and
installsays when it rounded.schtaskscannot set StartWhenAvailable, so a
run missed while the machine was off is lost; the adapter reports that with the
same phrase cron uses, which is what stopsstatusprinting the fact twice. The
command is quoted with Windows rules rather than POSIX ones, because a venv path
containing a space is the ordinary case there. Verified by asserting the exact
schtasksarguments: the development machine is Linux, so this backend is not
verified by execution. -
A launchd adapter, so
contextlake scheduleworks on macOS. Renders a
LaunchAgent plist withStartIntervalin seconds, installs it with
launchctl bootstrap gui/$UID(not the deprecatedload, which can return 0
while doing nothing), and reads the interval back off the installed plist
rather than reporting what was requested. launchd replays a run missed while
the machine was asleep, like systemd and unlike cron.schedule statusreports
no next-fire time for it, because launchd exposes none for an interval agent
and a computed guess would drift from what it actually does. Verified by
asserting the rendered plist and the exactlaunchctlarguments: the
development machine is Linux, so this backend is not verified by execution. -
schedule listreports units whose job record is gone.state()can only
answer "is job X installed?", which can only be asked about a job that still
has a record. The reverse had no reader: delete a record and its unit keeps
firing on schedule, is absent fromlist, anduninstallcannot reach it,
because it resolves a job name through the record that is gone. Adapters gain
installed_names(), implemented by reading the unit directory for systemd and
the marked crontab blocks for cron.listnames each orphan and its platform,
and says how to remove it. It also names any platform it could not enumerate:
skipping a platform and finding nothing on it both produce an empty result, so
reporting only the empty one would let "never looked" read as "checked, clean". With
--jsonthe two arrive as_orphaned_unitsand_unchecked_platforms, added
alongside the jobs rather than nesting them: a script reading this output keeps
working. The leading underscore is what makes that safe, since a job name must
start with an alphanumeric and so can never collide. -
schedule recommendsays when the activity bound was never measured. The
freshness half of the interval formula needs a count of how many repositories
changed, which only the index stage records. On an install without thekb
extra nothing records it, so the bound never engages and the interval rests on
the duty-cycle floor alone. Theactivity floorline was omitted entirely in
that case, which read the same as the bound being switched off. It now states
that it was not measured and what records it.--jsongains anactivity
field readingnot-measured,no-changeormeasured:floor_activity_seconds
is null for the first two of those, so the number alone could not tell them
apart. -
A memory-budget guard on
kb index. One repository in a real fleet
needed more than 9.3 GB in a single worker and never finished; no worker
count survives a repository that size. A repository whose parsed shard
exceeds 2,000,000 combined nodes and edges (well above the largest
repository known to index successfully, at roughly 356,000) is now skipped
with a named, explicit log line instead of being persisted, and the run
continues with the rest. This is a guard, not a fix: it stops one
pathological repository from taking a whole run down, it does not make that
repository indexable, and the exit code reflects that the run was not fully
clean. -
--workers Nonkb indexandbootstrap. Caps how many repositories the
index stage parses in parallel._index_workspacealready accepted and honoured
aworkersvalue, but nothing wired a flag to it, so the default (one fewer than
the CPU count, capped at 8) could not be overridden anywhere, including on
bootstrap, the default scheduled job. Lower it to cut peak memory on a large
fleet or a small machine.
Fixed
-
Every scheduled job read every other job's run history. All jobs append to
one history file and nothing in a record said which job wrote it. Two things
followed.decide_kindasks whether a successful full rebuild is older than
schedule_full_every, so a rebuild run by one job answered that question for
a job that had never run one and postponed its rebuild by a whole cycle. The
recommender's median run duration mixed every job's durations, so a
two-minutekb indexand a forty-minutebootstrapproduced one interval
that fitted neither. Records now carry the job name, passed to the child in
CONTEXTLAKE_SCHEDULE_JOB, and reads are scoped to one job. Records written
before this carry no job name and count as the default job's, so no existing
install loses the measurements it has earned. If you created a named job with
schedule intervalon 8.8.0, its earlier records are reattributed to
default, so that job starts from an empty history and runs one extra full
rebuild on its next cycle. Every record it writes after that is tagged. -
kb index's parallel path leaked every completed repository's parsed graph
for the whole run. The worker pool'sfutsdict was keyed byFuture, and
fut.result()does not clear a future's cached result, so a completed
repository'sGraphShardstayed reachable throughfutsuntil the pool's
withblock exited. Measured by A/B on the same 45 repositories at the same
worker count: 2,130 MB retained versus 95 MB released, 22x apart on
identical work, growing with repos indexed rather than with worker count.
Each future's dict entry is now dropped once its(repo_id, path, head)is
read off it, both on the fast path and the serial fallback after a broken
pool. -
A timed-out scheduled run orphaned its worker pool.
contextlake schedule run
spawned its child withsubprocess.run(..., timeout=...), which on a timeout kills
only the direct child. The child is usuallybootstraporkb index, which runs a
ProcessPoolExecutorof up to 8 workers, so a timed-out run left the whole pool
running, reparented to init and still holding memory. Measured on a real machine:
one killed run left 8 orphaned workers holding 12.4 GB. On a schedule, every
timed-out run leaked another pool. The child now starts in its own process group
and, on a timeout, the whole group is signalled (SIGTERM, then SIGKILL if it does
not exit within a few seconds), so nothing below it survives. Windows has neither process
groups norSIGKILL, so there the timeout falls back totaskkill /F /T, which
walks the child tree and reclaims the pool the same way.