Skip to content

contextlake 8.9.0

Choose a tag to compare

@github-actions github-actions released this 30 Aug 14:39
· 46 commits to main since this release

Added

  • AWS and Azure adapters. --platform aws creates an EventBridge Scheduler
    schedule firing an ECS task; --platform azure creates a Container Apps Job on
    a cron trigger. Both shell out to an already-authenticated aws or az, so
    contextlake still ships no cloud SDK. On EKS and AKS use --platform k8s
    instead: both are Kubernetes, so the CronJob adapter serves them and brings
    concurrencyPolicy: Forbid with it. Neither cloud service has an equivalent of
    Forbid, so two runs there can overlap and the second skips on the store's
    advisory lock rather than never starting, which both adapters report. They round
    differently by design: EventBridge takes rate(N minutes) and rounds to whole
    minutes, while a Container Apps Job trigger is a cron expression and rounds the
    way cron does. Registered but never auto-detected, for the same reason as the
    Kubernetes adapter. Verified by asserting the rendered request documents and the
    exact CLI arguments: there is no account here, so neither is verified by
    execution.

  • A Kubernetes adapter, covering OpenShift as well. Renders a CronJob and
    applies it with kubectl, falling back to oc. One adapter serves both,
    because OpenShift is Kubernetes with a stricter default security context and
    the manifest satisfies the stricter one: no runAsUser, since the restricted
    SCC assigns an arbitrary UID and rejects a pinned one, plus runAsNonRoot,
    a dropped capability set and an fsGroup so the mounted state directory stays
    writable. concurrencyPolicy: Forbid gives single-writer semantics from the
    cluster, so the second of two overlapping runs never starts. State mounts a
    PersistentVolumeClaim rather than an emptyDir, because an ephemeral store
    re-indexes the whole fleet every run. The schedule is a cron expression and
    rounds through the same function the cron adapter uses. Nothing is patched in
    the cluster on its own: changing an interval means re-running
    schedule install, since a background rewrite would need cluster-write rights
    for the life of the schedule. Reachable with --platform k8s and never
    auto-detected: kubectl on a PATH does not mean a schedule belongs in that
    cluster.

  • A Windows adapter, so contextlake schedule works through Task Scheduler.
    Creates a task with schtasks /SC MINUTE /MO n under a \contextlake folder.
    Two limits are reported rather than hidden. /MO counts whole minutes, so an
    interval is rounded the way cron's is, down above a minute and up below it, and
    install says when it rounded. schtasks cannot set StartWhenAvailable, so a
    run missed while the machine was off is lost; the adapter reports that with the
    same phrase cron uses, which is what stops status printing the fact twice. The
    command is quoted with Windows rules rather than POSIX ones, because a venv path
    containing a space is the ordinary case there. Verified by asserting the exact
    schtasks arguments: the development machine is Linux, so this backend is not
    verified by execution.

  • A launchd adapter, so contextlake schedule works on macOS. Renders a
    LaunchAgent plist with StartInterval in seconds, installs it with
    launchctl bootstrap gui/$UID (not the deprecated load, which can return 0
    while doing nothing), and reads the interval back off the installed plist
    rather than reporting what was requested. launchd replays a run missed while
    the machine was asleep, like systemd and unlike cron. schedule status reports
    no next-fire time for it, because launchd exposes none for an interval agent
    and a computed guess would drift from what it actually does. Verified by
    asserting the rendered plist and the exact launchctl arguments: the
    development machine is Linux, so this backend is not verified by execution.

  • schedule list reports units whose job record is gone. state() can only
    answer "is job X installed?", which can only be asked about a job that still
    has a record. The reverse had no reader: delete a record and its unit keeps
    firing on schedule, is absent from list, and uninstall cannot reach it,
    because it resolves a job name through the record that is gone. Adapters gain
    installed_names(), implemented by reading the unit directory for systemd and
    the marked crontab blocks for cron. list names each orphan and its platform,
    and says how to remove it. It also names any platform it could not enumerate:
    skipping a platform and finding nothing on it both produce an empty result, so
    reporting only the empty one would let "never looked" read as "checked, clean". With
    --json the two arrive as _orphaned_units and _unchecked_platforms, added
    alongside the jobs rather than nesting them: a script reading this output keeps
    working. The leading underscore is what makes that safe, since a job name must
    start with an alphanumeric and so can never collide.

  • schedule recommend says when the activity bound was never measured. The
    freshness half of the interval formula needs a count of how many repositories
    changed, which only the index stage records. On an install without the kb
    extra nothing records it, so the bound never engages and the interval rests on
    the duty-cycle floor alone. The activity floor line was omitted entirely in
    that case, which read the same as the bound being switched off. It now states
    that it was not measured and what records it. --json gains an activity
    field reading not-measured, no-change or measured: floor_activity_seconds
    is null for the first two of those, so the number alone could not tell them
    apart.

  • A memory-budget guard on kb index. One repository in a real fleet
    needed more than 9.3 GB in a single worker and never finished; no worker
    count survives a repository that size. A repository whose parsed shard
    exceeds 2,000,000 combined nodes and edges (well above the largest
    repository known to index successfully, at roughly 356,000) is now skipped
    with a named, explicit log line instead of being persisted, and the run
    continues with the rest. This is a guard, not a fix: it stops one
    pathological repository from taking a whole run down, it does not make that
    repository indexable, and the exit code reflects that the run was not fully
    clean.

  • --workers N on kb index and bootstrap. Caps how many repositories the
    index stage parses in parallel. _index_workspace already accepted and honoured
    a workers value, but nothing wired a flag to it, so the default (one fewer than
    the CPU count, capped at 8) could not be overridden anywhere, including on
    bootstrap, the default scheduled job. Lower it to cut peak memory on a large
    fleet or a small machine.

Fixed

  • Every scheduled job read every other job's run history. All jobs append to
    one history file and nothing in a record said which job wrote it. Two things
    followed. decide_kind asks whether a successful full rebuild is older than
    schedule_full_every, so a rebuild run by one job answered that question for
    a job that had never run one and postponed its rebuild by a whole cycle. The
    recommender's median run duration mixed every job's durations, so a
    two-minute kb index and a forty-minute bootstrap produced one interval
    that fitted neither. Records now carry the job name, passed to the child in
    CONTEXTLAKE_SCHEDULE_JOB, and reads are scoped to one job. Records written
    before this carry no job name and count as the default job's, so no existing
    install loses the measurements it has earned. If you created a named job with
    schedule interval on 8.8.0, its earlier records are reattributed to
    default, so that job starts from an empty history and runs one extra full
    rebuild on its next cycle. Every record it writes after that is tagged.

  • kb index's parallel path leaked every completed repository's parsed graph
    for the whole run.
    The worker pool's futs dict was keyed by Future, and
    fut.result() does not clear a future's cached result, so a completed
    repository's GraphShard stayed reachable through futs until the pool's
    with block exited. Measured by A/B on the same 45 repositories at the same
    worker count: 2,130 MB retained versus 95 MB released, 22x apart on
    identical work, growing with repos indexed rather than with worker count.
    Each future's dict entry is now dropped once its (repo_id, path, head) is
    read off it, both on the fast path and the serial fallback after a broken
    pool.

  • A timed-out scheduled run orphaned its worker pool. contextlake schedule run
    spawned its child with subprocess.run(..., timeout=...), which on a timeout kills
    only the direct child. The child is usually bootstrap or kb index, which runs a
    ProcessPoolExecutor of up to 8 workers, so a timed-out run left the whole pool
    running, reparented to init and still holding memory. Measured on a real machine:
    one killed run left 8 orphaned workers holding 12.4 GB. On a schedule, every
    timed-out run leaked another pool. The child now starts in its own process group
    and, on a timeout, the whole group is signalled (SIGTERM, then SIGKILL if it does
    not exit within a few seconds), so nothing below it survives. Windows has neither process
    groups nor SIGKILL, so there the timeout falls back to taskkill /F /T, which
    walks the child tree and reclaims the pool the same way.