Skip to content

Repository navigation

v0.0.644

Choose a tag to compare

View all tags
@github-actions github-actions released this 18 Oct 23:36
· 839 commits to main since this release
Immutable release. Only release title and notes can be modified.
v0.0.644
a3e1aa9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Cache hits copy files, remote transfers keep pace, globs are standard

A local cache hit now copies plain files instead of unpacking an
archive, and remote downloads and uploads no longer wait on the run's
workers. A cache server learns which scope each request means, GitLab CI
gets the branch scopes GitHub Actions has, task globs read [...] the
way Turbo, Nx and most glob tools do, and vx upgrade re-locks the
workspace for you.

This release is breaking: every cached task misses once after the
upgrade, two cache.store settings are renamed, cache.retention is
gone, and a glob with a literal [...] directory needs an escape.
Read the breaking changes first.

Enjoy the release!

In this release

Cache hits copy files instead of unpacking archives

Each local cache hit used to decompress and untar a tar.zst, even
though the same entry was restored again and again on one machine.

The local cache now keeps each entry as its plain files, unpacked once
when it is saved or downloaded. A download of up to 4 MiB is fetched
whole, decompressed once and written straight into the store, with no
temp archive between. Filling 200 entries of 1 MB each from a remote
went from 1,015 ms to 773 ms. A hit checks each file's size and
mtime, then copies it out. The kernel does the copy, and can clone the
file on a filesystem that supports it. vx never hard-links, so a build
that later edits a restored file leaves the cache's copy as it was. A
remote still gets one tar.zst per entry, packed when it is uploaded.
Small entries, up to 32 KiB by default, stay inside the cache index as
before.

// .vx/workspace.ts
import { defineWorkspace } from '@vzn/vx/config'

export default defineWorkspace({
  cache: { store: { embeddings: '32KiB', decompress: true, verify: 'stat' } }, // the defaults
})
  • embeddings is the largest entry kept inside the index, as a size;
    false keeps every entry as files.
  • decompress: false keeps each entry as one tar.zst. That optimizes
    disk space but trades speed, as every restore decompresses it just in
    time.
  • verify sets what a hit checks first: 'stat' checks size and mtime,
    'hash' checks every byte, and 'none' checks nothing.

Each setting also has an env var: VX_CACHE_STORE_EMBEDDINGS,
VX_CACHE_STORE_DECOMPRESS and VX_CACHE_STORE_VERIFY. See the
workspace config.

Remote cache transfers set their own pace

Downloads from a remote cache ran only as many at once as the run had
workers, and uploads ran four at once, so on a remote with real latency
most of a run was spent waiting on round trips.

Downloads and uploads now each have their own limit. By default each is
min(96, max(cores × 3, 64)), the formula pnpm uses, and the upload
limit applies to each remote. Local restores stay at twice the run's
worker count. On a remote 30 ms away, uploading 1,000 entries went from
9.4 s to 3.8 s, and restoring them from the remote from 5.1 s to 2.6 s.

// .vx/workspace.ts
import { defineWorkspace } from '@vzn/vx/config'

export default defineWorkspace({
  cache: { concurrency: { download: 64, upload: 64, restore: 16 } },
})

A run sets the same limits with --cache-concurrency-download,
--cache-concurrency-upload and --cache-concurrency-restore, or with
VX_CACHE_CONCURRENCY_DOWNLOAD, VX_CACHE_CONCURRENCY_UPLOAD and
VX_CACHE_CONCURRENCY_RESTORE.

Your cache server knows which scope each request means

A branch's entries live under their own scope, but the server saw only
a key. A server that keeps a bucket per branch had to guess, or look in
several.

turboCache() and nxCache() now send an x-vx-cache-scope header on
every request: trusted for a task's own key (main's entries), else the
branch scope, such as acme/app:feat-x. The key is the same as before.
The header states intent only: the server still decides from the CI ID
token whether a write to trusted is allowed, and a request without the
header, from an older client, leaves the layering to the server.

A GitHub merge queue's run (merge_group) builds what is about to land
on main, so github() now treats it as trusted: it reads and writes
main's entries, compared with the queue's base commit. Before, it wrote
under its gh-readonly-queue/... branch, a scope nobody reads. See
Cache scopes and tokens.

Branch cache scopes on GitLab CI

On GitLab every pipeline wrote main's entries, so a branch's builds
landed where main reads, and a laptop on the branch found nothing CI had
built for it.

gitlab() from @vzn/vx-ci names the run's branch from GitLab's own
variables, as github() does on Actions:

// .vx/workspace.ts
import { defineWorkspace } from '@vzn/vx/config'
import { gitlab } from '@vzn/vx-ci'

export default defineWorkspace({ plugins: [gitlab()] })

The default branch keeps main's entries. A branch, a tag or a merge
request writes <project path>:<branch>: a subgroup's project is
group/sub/app:feat-x, and a fork's merge request names the fork. A
merge request is compared with its diff base, so the job needs no git
history, and a merge train reads and writes main's entries. A laptop on
feat-x keeps a subgroup's whole project path too, so it reads the same
scope. See GitLab CI.

Task globs are standard globs

A [...] in a task glob was literal, so a Next.js route directory like
app/[id] needed no escape, but src/*.[jt]s matched nothing, unlike Turbo, Nx and most glob tools. Now brackets are a character class everywhere:
src/*.[jt]s matches .js and .ts, and a literal route directory is
escaped.

inputs: { files: ['src/**/*.[jt]s', 'app/\\[id\\]/**'] }

@vzn/vx-migrate passes Turbo's and Nx's classes through unchanged and
escapes brackets in the paths it writes.

vx upgrade re-locks the workspace

A new release can record .vx/projects.lock.json differently, so after
vx upgrade you had to remember to run vx lock yourself.

Now, inside a workspace, an upgrade that installed something runs the
new vx's vx lock at the workspace root. If the lock fails, the upgrade
still stands and vx says so:

vx upgrade
! vx upgrade  vx lock failed: run vx lock

vx upgrade -g moves only the global vx and never locks.

Other noteworthy changes

  • A task that looks up the remote before the run's batch probe answers
    now waits for that answer, so a key the batch reports missing costs no
    request of its own.
  • On a branch, a plugin's ctx.cache.get, has and hasMany make one
    request: the branch's entry by default, the default branch's with
    false. Writes on a branch stay the branch's.
  • @vzn/vx-history orders a branch by its own CI's timings:
    CI on a branch writes its commit's timings under the branch's scope,
    and a run on the branch reads main's timings, then its last commit's.
    A task the branch has timed uses the branch's time.
  • With embeddings off, a remote hit for a task with no outputs
    used to fail to store and ran as a miss. It now stores.
  • Opening the cache no longer walks the shared store's directories, a
    cost that grew with every stored entry and hit every run, no-op runs
    included.
  • The run footer no longer prints a "saved" time: time is the run's
    wall time.
  • A run no longer cleans up on exit: no pruning of month-old history, no
    eviction. vx cache clean is the one cleaner; run it from a schedule
    or CI, such as vx cache clean --older-than 30d --max-size 10G.
  • Size settings also read an iB unit, such as 32KiB.

Breaking changes

  • The cache version moves to vx-cache-v48: every cached task misses
    once after the upgrade.
  • cache.store.embedded is now cache.store.embeddings
    (VX_CACHE_STORE_EMBEDDINGS): false or a size, default '32KiB'.
    true is no longer a value.
  • cache.store.compress is now cache.store.decompress with the
    meaning inverted (compress: true is decompress: false), and its
    env var is VX_CACHE_STORE_DECOMPRESS.
  • A [...] in a task glob is a character class. Escape a literal route
    directory: app/\[id\]/**.
  • Plugin API: on a branch, ctx.cache.get, has and hasMany no
    longer fall back to the default branch's entry. Pass false to read
    it.
  • cache.retention (VX_CACHE_RETENTION_OLDER_THAN,
    VX_CACHE_RETENTION_MAX_SIZE) is removed: no run evicts. Run
    vx cache clean --older-than / --max-size instead. @vzn/vx-migrate
    no longer maps Turbo's cacheMaxSize / cacheMaxAge or Nx's
    maxCacheSize.

See also Upgrading.

How to update

vx upgrade

Read it on the site

Assets 8
Loading