0.37

Highlights:

• ch-image can get into a state where it has images in the storage directory that are marked cached, but the build cache has no record of them. We were not able to find a root cause, but this is now a warning rather than a crash (#1824). If you see the warning, please get in touch so we can find the root cause.

• ch-image: new option --pdb that drops into the Python debugger at the specified module and line (#1837)

• Build/install: Version 0.36 introduced a dependency on pkg-config at ./autogen.sh time that gives misleading errors about “undefined macro” if not met (#1845). This is now fixed.

What else is new:

• ch-image:
  • RUN: trailing backslashes no longer sometimes parsed literally (#1679)
  • now errors out if blob downloaded incompletely or incorrectly (#1758)
  • better pretty-printing of Dockerfile parse trees (#1816)
  • new hot pink log function for printf(3)-style debugging (#1825)
  • fixed syntax warnings with Python 3.12 (#1834)
• ch-fromhost:
  • misleading warnings from ldconfig suppressed (#732)
  • now supports --quiet (#1732)
• ch-completion.bash: new man page (#1839)
• ch-convert: tab completion now works for images with colon in name (#1841)
• docs: revised contributor’s guide (#1178)
• miscellaneous bug fixes & improvements in:
  • refactoring (#1838, #1840)
  • documentation & logging/errors (#1750, #1773, #1818, #1823, #1831)
  • test suite and examples (#1759, #1810, #1815, #1826, #1827, #1830)

More details: https://github.com/hpc/charliecloud/milestone/43?closed=1
Even more details: v0.36...v0.37

0.36

Highlights:

• ch-run: new option --write-fake/-W that overlays a writeable tmpfs atop a read-only image (#1793). This can also be used to bind-mount on directories that do not exist in read-only images (#96). The feature does require recent-ish kernel support (see FAQ).

• ch-image build with seccomp root emulation (the default) now intercepts mknod(2) only if creating a device (#1808). In particular, Dockerfiles can now create named pipes (FIFOs) (#1779).

• Extended attributes (xattrs) support now defaults to disabled (#1787). The feature proved to be too finicky for a default setting, and xattrs do not seem to be well supported across the Linux ecosystem.

What else is new:

• ch-image:
  • bundled Lark library updated to 1.1.9 (#1792)
  • upgrading storage directories from 0.29 and earlier is no longer supported (#1590)
  • fix pull of certain images (#1805)
  • build:
    • two curly brace bugs fixed in Dockerfile processing (#1751, #1794)
    • now correctly enforces minimum permissions (#1765)
    • --force=seccomp now fails noisily if not supported by architecture (#1771)
    • --force=seccomp now supported on s390x (#1772)
    • no longer follows symlinks when setting xattrs (#1782)
• ch-run:
  • now supports --quiet (#1660)
  • builds correctly against SquashFUSE 0.5.0 (#1784)
• ch-convert:
  • now supports --quiet (#1735)
  • tab completion added (#1804)
• ch-test:
  • now supports only Bats-core, not legacy Bats (#555)
  • now runs doctests in addition to shell-based tests (#1744)
• configure
  • no longer reports the presence of Buildah and Docker (#1757)
• miscellaneous bug fixes & improvements in:
  • bugs both introduced and fixed during release cycle (#1778, #1791, #1800, #1802)
  • refactoring (#1745, #1761)
  • documentation & logging/errors (#1695, #1785, #1786)
  • test suite and examples (#1756, #1760, #1777, #1781)

0.35

Highlights:

• (Bad news) Previously, the build cache’s optional large file support (i.e. ch-image --cache-large set to a non-default value) used hard links to bring large files and out of images. This assumes that hard links are copy-on-write, which they are not. Therefore, use of this feature was likely to corrupt the cache. We now copy large files instead of hard linking them, which is often a significant performance impact. (#1740)

• (Good news) ch-image now uses kernel fast paths for file copying when available. In particular, some filesystems offer a “reflink” operation that copies only metadata (notably BTRFS, XFS, and recent ZFS). This should provide a general performance increase, but in particular lets the build cache’s large file operations remain fast meta-data only operations. See the “Best practices” document for a detailed discussion. (#1742)

• (Good news) ch-image build supports a new non-standard Dockerfile instruction, RSYNC (#1708). This copies files from the context directory into the image with rsync(1) semantics, which are better defined than COPY and provide more control, notably for symlinks.

• (Good news) Interrupting ch-image with SIGINT or SIGTERM (notably, control-C) is now much less likely to leave the storage directory in an invalid date (#1518).

What else is new:

• ch-image build and ch-run: $HOME is always set now (#1687)
• ch-image: new argument --quiet to reduce verbosity (#1613)
• configure --libexecdir and similar arguments now work (#683)
• miscellaneous bug fixes & improvements in:
  • refactoring (#1730)
  • documentation & logging/errors (#1731)
  • test suite and examples (#1722, #1725, #1726, #1736)

More details: https://github.com/hpc/charliecloud/milestone/41?closed=1
Even more details: v0.34...v0.35

0.34

Highlights:

• ch-image: seccomp root emulation mode (ch-image --force=seccomp, or bare --force in previous versions) is now the default behavior (#1572). Routine use of --force is no longer needed, and in fact bare --force without an argument will likely mean a more intrusive root emulation mode in the near future.

• A pre-print of our build cache performance study is now available. TL;DR: it works, has structural advantages, and may even outperform overlay-based caches in many scenarios.

What else is new:

• ch-image and (sometimes) ch-run now re-print warnings when exiting (#1674)
• Shell programs no longer hang if docker(1) has been aliased to podman(1) (#1656)
• Bash tab completion now deals correctly with mid-line editing (#1709)
• ch-image:
  • build cache now saves and restores ACLs and xattrs (#1287)
  • now recognizes extension .df for image name inference (#1717)
  • minimum storage directory upgradability is now v0.28 (#1589)
• ch-run:
  • new option --set-env0 that accepts zero-separated files (#1124)
• miscellaneous bug fixes & improvements in:
• refactoring & cleanup (#1693, #1697, #1714)
• documentation & logging/errors (#1672, #1689, #1691, #1696, #1698, #1703, #1715, #1719, #1721)
• test suite and examples (#1707, #1716, #1723)
• bugs both introduced and fixed during release cycle (#1718)

More details: https://github.com/hpc/charliecloud/milestone/40?closed=1
Even more details: v0.33...v0.34

0.33

Highlights:

• ch-image and ch-run now have tab-completion for Bash (#1618, #1652)
• ch-ssh has been removed (#1379). We were able to identify no one who wanted to keep it.

What else is new:

• ch-convert:
  • now accepts empty directory as valid output path (#1612)
  • now accepts -s/--storage like the other executables (#1650)
• ch-image:
  • --force=seccomp (the default force mode) now works for PowerPC (#1663).
  • no longer warns if $CH_GROW_STORAGE is set (#1007)
  • build cache no longer:
    • tries to use multiple Git versions simultaneously (#1606)
    • erroneously re-uses imported base images (#1658)
  • delete:
    • now accepts a list of images, not just one (#1626)
    • now removes target image from the cache too (#1485)
  • no longer crashes ...
    • on pushing imported images (#1513)
    • when stdout does not support curly quotes (#1629)
    • when using some storage directories that have been moved (#1657)
• miscellaneous bug fixes & improvements in:
  • documentation & logging/errors (#1547, #1611, #1643, #1667)
  • test suite and examples (#210, #1509, #1614, #1615, #1621, #1623, #1624, #1628, #1642, #1647, #1649, #1662, #1675)
  • bugs both introduced and fixed during release cycle (#1648, #1659)

More details: https://github.com/hpc/charliecloud/milestone/39?closed=1
Even more details: v0.32...v0.33

0.32

Request for feedback:

We are considering removing ch-ssh, a utility program to facilitate SSH connections from one Charliecloud container into an equivalent container on another host. Please respond to and/or comment on our poll, especially if you use this tool, in discussion #1600.

Highlights:

• ch-image build --force now uses a new mode seccomp by default, based on seccomp(2), which is simpler, faster, completely agnostic to libc, and mostly agnostic to distribution (#1527). See §6.7.3 in the ch-image man page for details. The old mode is still available with --force=fakeroot. Because it is prohibitively difficult to detect if seccomp mode would be useful, and we do not want to promote an obsolete mode, ch-image build no longer recommends --force on build failures, and the option --no-force-detect is gone (#1563).

  Note that if seccomp mode proves successful, we plan to remove fakeroot mode (#1565) and possibly make --force the default (#1572).

• Per our new one-year support policy, storage directories created by versions 0.24 and earlier can no longer be upgraded (#1588). Also, storage directories in the default location of 0.25 and earlier will no longer be moved to the new default location (#1243).

• ch-test now works with the maintained bats-core fork of Bats that is in most current distributions (#582). The old Bats 0.4.0 appears to still work but is no longer tested or supported.

• New examples to demonstrate use of libfabric (#1443).

What else is new:

• ch-image:
  • error messages of failed subcommands are now always printed (#1566)
  • build:
    • LABEL now supported (#781)
    • no longer crashes if Git username and e-mail are not configured (#1535)
    • now references multi-stage build aliases correctly (#1571)
  • build-cache --gc: no longer crashes if large files are present (#1586)
  • pull:
    • “duplicate non-empty layer” is now a warning, not an error (#1568)
    • can now pull images containing certain hard links (#1569)
    • download manifests more efficiently (#1101)
  • push: now authenticates before preparing image (#1426)
• ch-run:
  • no longer crashes when built against recent glibc (#1550)
  • deprecated no-op option --no-home no longer accepted (#1524)
• ch-convert:
  • --help now provides explanation and example (#1599)
  • can now convert from Docker even if UID > 2²¹ (#1573)
• ch-fromhost and examples are now more compatible with Flux (#1597)
• miscellaneous bug fixes & improvements in:
  • build/install (#1553, #1583, #1584)
  • documentation & logging/errors (#1585)
  • test suite and examples (#1218, #1476, #1483, #1549, #1558, #1559, #1561, #1574, #1587, #1593)
  • bugs both introduced and fixed during release cycle (#1602)
  • refactoring/code quality (#1069, #1285, #1528, #1579)

More details: https://github.com/hpc/charliecloud/milestone/38?closed=1
Even more details: v0.31...v0.32

0.31

Highlights:

• The tutorial has been completely overhauled, and a new best practices section added to the docs (#536).
• ch-run can now run images in ch-image storage by name (e.g., ch-run debian:11), and running images in ch-image storage by path is now disallowed (e.g., ch-run /var/tmp/charlie.ch/img/debian:11) (#1058).
• ch-run no longer bind-mounts home directories by default, i.e., --no-home is the default behavior (#1470). Your home directory can be bind-mounted with --home, i.e., this is how to get the previous default behavior. Option --no-home is still accepted as a no-op but will be removed in 0.32.
• ch-convert now can convert to and from Podman (#1360).

What else is new:

• ch-image:
  • storage directories only upgradeable if created by a Charliecloud version less than one year old (#1507)
  • parsing Dockerfile lines with leading whitespace is much faster (#1503)
  • Docker Hub rate limits now logged clearly (#1390)
• test suite now passes with ShellCheck 0.9.0 (#1510)
• miscellaneous bug fixes & improvements in:
  • build/install (#1498, #1538)
  • documentation & logging/errors (#1486)
  • test suite (#1430)
  • bugs both introduced and fixed during release cycle (#1516, #1520, #1544)
  • refactoring/code quality (#1092, #1478)

More details: https://github.com/hpc/charliecloud/milestone/37?closed=1
Even more details: v0.30...v0.31

0.30

Highlights:

• The build cache is now enabled by default if an appropriate Git is installed (#1344).

• Significant build cache performance improvements (#1411, #1412, #1451, #1459, #1464).

What else is new:

• ch-image:
  • build:
    • ARG before FROM now supported (#779)
    • COPY can now create multiple directory levels (#1482)
    • COPY from previous stage now works even if no context (#1382)
    • COPY with build cache no longer always misses (#1396)
  • build-cache:
    • more details printed in verbose mode (#1441)
  • delete:
    • now accepts globs (#1447)
    • now removes all stages of multi-stage images (#1368)
  • list:
    • new option -l for a more verbose listing (#1413)
    • now shows digest of remote images (#1365)
  • new subcommand undelete (build cache enabled only) (#1438)
  • Lark module no longer enforces a maximum version (#1432)
  • relative storage paths now rejected instead of breaking (#1403)
  • new option --no-lock to disable storage collision checking (#1418)
  • ch-run:
    • error “can’t join user namespace of pid” fixed (#1270)
  • ch-convert: obscure permissions bug fixed (#1484)
  • web docs: search no longer returns results that are 404 (#1461)
  • test suite and examples:
    • base images have clearer names (#1323)
    • now correctly propagates examples failures (#1475)
  • miscellaneous bug fixes & improvements in:
    • documentation & logging/errors (#1342, #1370, #1399, #1437, #1440, #1492)
    • test suite (#1385, #1386, #1387, #1388, #1389)
    • bugs both introduced and fixed during release cycle (#1469)
    • refactoring/code quality (#992, #1455)
    • development/debug tools (#934, #1434, #1446, #1458)

More details: https://github.com/hpc/charliecloud/milestone/36?closed=1
Even more details: v0.29...v0.30

0.29

Highlights:

• All Charliecloud workflows are once again fully unprivileged end-to-end (#1415). Previously, the SquashFS workflow required a setuid executable (fusermount3) in order to initialize the FUSE mount. In 0.29, fusermount3 does not need to be setuid, and in fact Charliecloud actively suppresses its setuid bit if set. To our knowledge, Charliecloud is the first container implementation to deploy a release with a fully unprivileged SquashFS workflow.

• Build cache enabled by default has been postponed to 0.30 (#1421).

Known bugs of note:

• We are seeing intermittent problems with ch-run --join erroring “can’t join user namespace of pid” (#1270). Currently the workaround is to configure with --disable-syslog.

What else is new:

• ch-image:
  • build:
    • failed builds no longer break later builds when cache is enabled (#1405)
    • don’t crash if context directory not given (#1372)
  • build-cache --reset: fix race with Git garbage collection (#1406)
• miscellaneous bug fixes & improvements in:
  • documentation & logging/errors (#1402)

More details: https://github.com/hpc/charliecloud/milestone/35?closed=1
Even more details: v0.28...v0.29

0.28

Highlights:

• ch-image now has a build cache, based on Git (#583); version 2.28.1 or greater is required. Because this is an experimental feature, it is disabled by default; use --cache or set $CH_IMAGE_CACHE=enabled to enable it. We anticipate enabling by default in 0.29 if an appropriate Git is available. See the documentation for more details. We are especially interested in feedback on this feature. Please let us know how it works for you!

• ch-image now accesses registries anonymously unless the new switch --auth is specified or the sub-command is push (#1329). In those cases, registry access is always authenticated. Among other things, this enables a private Docker Hub rate limit.

• ch-image pull now updates the specified image if the registry has a newer version, without needing --no-cache (#1305).

Known bugs of note:

• We are seeing intermittent problems with ch-run --join erroring “can’t join user namespace of pid” (#1270). Currently the workaround is to configure with --disable-syslog.

What else is new:

• ch-image:
  • build:
    • --force: now supports Arch (#1295), Alpine (#1296), and Ubuntu 22.04 (#1348) images
    • can now build multi-stage tagged images (i.e., colon in name) (#1369)
  • delete: now works even if image in storage directory is broken (#1321)
  • pull:
    • second arg is destination image name, not a directory (#1337)
    • now works from nVidia NGC (#1318) and SuSE (#1336) registries
    • now works with ARM images that don’t specify a variant (#1249)
  • push: now preserves environment variables (#1307)
  • no longer crashes instead of erroring if Requests is not installed (#1384)
  • errors if another instance is using the same storage directory (#766)
  • new option --debug that adds a compact stack trace to fatal errors (#1284)
• ch-run:
  • now re-tries squash mount, in case previous unmount is not complete (#1364)
  • --unset-env now supports extended globs if the underlying libc does (#1311)
• ch-convert -i ch-image -o dir now has a progress meter (#1332)
• miscellaneous bug fixes & improvements in:
  • refactoring for greater robustness (#1283)
  • documentation & logging/errors (#1001, #1265, #1319, #1378)
  • test suite and examples (#186, #678, #1240)
  • bugs introduced and fixed between releases (#1345, #1349, #1351, #1373, #1383)

More details: https://github.com/hpc/charliecloud/milestone/34?closed=1
Even more details: v0.27...v0.28