Skip to content

v4.9.0

Choose a tag to compare

@github-actions github-actions released this 16 Oct 20:47
· 20 commits to main since this release
  • A walk of sources or generates no longer leaves files out on an error.
    It moves from directory to directory by handle, one name per lookup, so the
    symlinks along a path no longer add up to the kernel's limit (a single link
    whose own target spells out a long chain still can, and then ends its branch),
    and a wildcard stops at the first directory it would revisit: a symlink loop
    is cut there, the link kept as an entry. 4.8.x followed a loop until the
    kernel refused, which past about 20 links happens at a varying depth on a busy
    machine and changed the fingerprint from run to run. A name that is gone,
    unreadable or not a directory (ENOENT, EACCES, ENOTDIR), or a link the kernel
    will not resolve (ELOOP), ends a branch as in the shell; a directory removed
    or made unreadable between its listing and its opening is skipped that way.
    Any other error (too many open files, an I/O error) fails the task, naming the
    path, where 4.8.x silently skipped the directory. --status reports such
    errors too, instead of empty file lists. On Linux, a directory that can be
    searched but not read is still crossed by a literal path, as by the shell.
    Hashing still opens each file by its path, so a file behind more than about 20
    nested links can fail to hash on a busy machine: an error, never an omission.
    On Windows lookups still go through paths, and a loop is recognised by its
    canonical path (type-checked, not run-tested).
  • Sources are hashed at compile time only when something reads
    {{.CHECKSUM}}.
    Compiling a task, the quick compile that checks its
    platforms and required vars, and compiling every task a task: or from:
    reference reaches no longer hash sources just to fill the variable. A task
    reading it (in commands and their vars, task names, preconditions, the cache
    block, deferred commands or requires, through a ref: var or Jinja
    debug()), and every task under --summary or templated output: group
    markers, keeps the compile-time value as before. For the others, a task with
    generates computes the value once just before its build-once lock, after
    setup: the sources hash in the lock name (<task>:<cache checksum>), the
    cache key and the cache entry's recorded sources is now taken there rather
    than before setup. A task without generates takes no lock and has no cache
    entry, and simply reuses its up-to-date reading. A cache entry is now restored
    only when the run's cache key matches what its up-to-date check read, and
    saved only when it matches what the run built from: a key taken before
    preconditions, a lock wait or another writer changed the sources is skipped
    (noted with -v), for these tasks and for those keeping the compile-time
    value alike.
  • Fingerprinting walks each directory once and takes file types from the
    listing,
    so hashing a regular file costs one stat instead of about four. A
    source or generated file matched by the walk and removed before it is hashed
    now fails the task, naming it, where it could be left out before, and so does
    one whose type changed in between (a file replaced by a symlink, say). On
    Windows that check opens reparse points without following them; it is
    type-checked there but not run-tested, and reading files behind filter-driver
    reparse points (deduplicated files, OneDrive placeholders) is unverified.
  • A cache restore cancelled before it is verified is undone, and so is one
    that cannot durably mark itself kept: the task then runs normally, and an
    --import-cache fails. Recovering a restore cut short by a crash puts back
    the originals it moved aside over entries it never verified, but only where an
    entry is still exactly what the restore wrote: an entry changed since is kept,
    with a warning naming the original, saved beside it as
    .task-restore-original-<name>-<random>. A restore cut short while committing
    keeps its entries.
  • Fingerprints are read off the thread that schedules tasks. Tasks running
    in parallel now fingerprint at the same time, and interrupts, output and lock
    renewals are handled while a large tree is being hashed.
  • A run reads its sources and generates once after its commands. The
    drift check, the saved fingerprint and the cache entry all use that one
    reading, instead of reading the files up to three more times. The drift
    warning now names the sources whose content changed, rather than those whose
    size or modification time did.
  • Fingerprinting digests files in parallel, on at most as many threads as
    there are CPUs available, shared by every task hashing at the same time. The
    checksum does not depend on the number of threads.
  • Fingerprints use a new checksum format. Each file is recorded with its
    path relative to the task directory, its kind (file or symlink) and its own
    digest, in path order, and the task's rules by their fields, so adjacent
    inputs no longer run together, and sources outside the task directory no
    longer make the checksum depend on the checkout directory's name. Checksums
    are 32 hex digits and no longer match the Go implementation: every task with
    sources rebuilds and misses the cache once, and one that renders
    {{.CHECKSUM}} in a field its identity covers (commands, their vars, task
    names in deps/setup and their vars, preconditions, the cache block) also
    gets a new checksum file name. A source that is neither a regular file nor a
    symlink (a named pipe, a device) now fails the task instead of being read, and
    fingerprinting errors name the file.