v0.7.0-rc3

The documentation for this version is available at the Earthly 0.7 documentation page.

Earthly CI

Earthly 0.7 is the first version compatible with Earthly CI.

Earthly 0.7 introduces the new keywords PIPELINE and TRIGGER to help define Earthly CI pipelines.

my-pipeline:
    PIPELINE --push
    TRIGGER push main
    TRIGGER pr main
    BUILD +my-target

For more information on how to use PIPELINE and TRIGGER, please see the reference documentation.

Podman support

Podman support has now been promoted out of beta status and is generally available in 0.7. Earthly will automatically detect the container frontend, whether that's docker or podman and use it automatically for running Buildkit locally, or for outputting images locally resulting from the build.

Please note that rootful podman is required. Rootless podman is not supported.

VERSION is now mandatory

The VERSION command
is now required for all Earthfiles, and an error will occur if it is missing. If you are not ready to update your
Earthfiles to use 0.7 (or 0.6), you can declare VERSION 0.5 to continue to use your Earthfiles.

.env file is no longer used for ARG or secrets

The .env file will only be used to automatically export environment variables, which can be used to configure earthly command line flags.
As a result, values will no longer be propagated to Earthfile ARGs or RUN --secret=... commands.

Instead if you want build arguments or secrets automatically passed into earthly, they must be placed in .arg or .secret files respectively.

Note that this is a backwards incompatible change and will apply to all Earthfiles (regardless of the defined VERSION value).

Pushing no longer requires everything else to succeed

The behavior of the --push mode has changed in VERSION 0.7 and is backwards incompatible with VERSION 0.6. Previously, --push commands would only execute if all other commands had succeeded. This precondition is no longer enforced, to allow for more flexible push ordering via the new WAIT clause. To achieve the behavior of the previous --push mode, you need to wrap any pre-required commands in a WAIT clause. For example, to push an image only if tests have passed, you would do the following:

test-and-push:
  WAIT
    BUILD +test
  END
  BUILD +my-image
my-image:
  ...
  SAVE IMAGE --push my-org/my-image:latest

This type of behavior is useful in order to have better control over the order of push operations. For example, you may want to push an image to a registry, followed by a deployment that uses the newly pushed image. Here is how this might look like:

push-and-deploy:
  ...
  WAIT
    BUILD +my-image
  END
  RUN --push ./deploy.sh my-org/my-image:latest
my-image:
  ...
  SAVE IMAGE --push my-org/my-image:latest

Where ./deploy.sh is custom deployment script instructing a production environment to start using the image that was just pushed.

Promoting experimental features

This version promotes a number of features that have been previously in Experimental and Beta status. To make use of
the features in this version you need to declare VERSION 0.7 at the top of your Earthfile.

Declaring VERSION 0.7 is equivalent to

VERSION \
  --check-duplicate-images \
  --earthly-git-author-args \
  --earthly-locally-arg \
  --earthly-version-arg \
  --explicit-global \
  --new-platform \
  --no-tar-build-output \
  --save-artifact-keep-own \
  --shell-out-anywhere \
  --use-cache-command \
  --use-chmod \
  --use-copy-link \
  --use-host-command \
  --use-no-manifest-list \
  --use-pipelines \
  --use-project-secrets \
  --wait-block \
  0.6

For more information on the individual Earthfile feature flags see the Earthfile version-specific features page.

Changed

• The behavior of the --push mode has changed in a backwards incompatible manner. Previously, --push commands would only execute if all other commands had succeeded. This precondition is no longer enforced, allowing push commands to execute in the middle of the build now. Previously under VERSION --wait-block 0.6.
• ARGs declared in the base target do not automatically become global unless explicitly declared as such via ARG --global. Previously under VERSION --explicit-global 0.6.
• The Cloud-based secrets model is now project-based; it is not compatible with the older global secrets model. Earthfiles which are defined as VERSION 0.5 or VERSION 0.6 will continue to use the old global secrets namespace; however
  the earthly command line no longer supports accessing or modifying the global secrets. A new earthly secrets migrate command has been added to help transition the global-based secrets to the new project-based secrets. If you need to manage secrets from Earthly 0.6 without migrating to the new 0.7 secrets, please use an older Earthly binary.
• All COPY and SAVE ARTIFACT operations now use union filesystem merging for performing the COPY. This is similar to COPY --link in Dockerfiles, however in Earthly it is automatically enabled for all such operations. Previously under VERSION --use-copy-link 0.6.
• The platform logic has been improved to allow overriding the platform in situations where previously it was not possible. Additionally, the default platform is now the native platform of the runner, and not of the host running Earthly. This makes platforms work better in remote runner settings. Previously under VERSION --new-platform 0.6.
• Earthly will automatically shellout to determine the $HOME value when referenced #2469
• Improved error message when invalid shell variable name is configured for a secret. #2478
• earthly ls has been promoted from experimental to beta status.
• Setting a VERSION feature flag boolean to false (or any other value) will now raise an error; previously it was syntactically valid but had no effect.
• SAVE ARTIFACT <path> AS LOCAL ... when used under a TRY / FINALLY can fail to be fully transferred to the host when the TRY command fails (resulting in an partially transferred file); an underflow can still occur, and is now detected and will not export the partial file. 2452
• The --keep-own flag for SAVE ARTIFACT is now applied by default; note that COPY --keep-own must still be used in order to keep ownership
• Values from the .env file will no longer be propigated to Earthfile ARGs or RUN --secret=... commands; instead values must be placed in .arg or .secret files respectively. Note that this is a backwards incompatible change and will apply to all Earthfiles (regardless of the defined VERSION value). #1736
• Some particularly obtuse syntax errors now have hints added to help clarify what the expected syntax might be. #2656

Added

• The commands PIPELINE and TRIGGER have been introduced for defining Earthly CI pipelines. Previously under VERSION --use-pipelines 0.6.
• The clause WAIT is now generally available. The WAIT clause allows controlling of build order for operations that require it. This allows use-cases such as pushing images to a registry, followed by infrastructure changes that use the newly pushed images. Previously under VERSION --wait-block 0.6.
• The command CACHE is now generally available. The CACHE command allows declaring a cache mount that can be used by any RUN command in the target, and also persists in the final image of the target (contents available when used via FROM). Previously under VERSION --use-cache-command 0.6.
• The command HOST is now generally available. The HOST command allows declaring an /etc/hosts entry. Previously under VERSION --use-host-command 0.6.
• New ARG EARTHLY_GIT_COMMIT_AUTHOR_TIMESTAMP will contain the author timestamp of the current git commit. #2462
• New ARGs EARTHLY_VERSION and EARTHLY_BUILD_SHA contain the version of Earthly and the git sha of Earthly itself, respectively.
• It is now possible to execute shell commands as part of any command that allows using variables. For example VOLUME $(cat /volume-name.txt). Previously under VERSION --shell-out-anywhere 0.6.
• Allow custom image to be used for git operations. #2027
• Earthly now checks for duplicate image names when performing image outputs. Previously under VERSION --check-duplicate-images 0.6.
• SAVE IMAGE --no-manifest-list allows outputting images of a different platform than the default one, but without the manifest list. This is useful for outputting images for platforms that do not support manifest lists, such as AWS Lambda. Previously under VERSION --use-no-manifest-list 0.6.
• COPY --chmod <mode> allows setting the permissions of the copied files. Previously under VERSION --use-chmod 0.6.
• The new ARG EARTHLY_LOCALLY indicates whether the current target is executed in a LOCALLY context. Previously under VERSION --earthly-locally-arg 0.6.
• The new ARGs EARTHLY_GIT_AUTHOR and EARTHLY_GIT_CO_AUTHORS contain the author and co-authors of the current git commit, respectively. Previously under VERSION --earthly-git-author-args 0.6.
• earthly doc [projectRef[+targetRef]] is a new subcommand in beta status. It will parse and output documentation comments on targets.
• Ability to store docker registry credentials in cloud secrets and corresponding earthly registry setup|list|remove commands; credentials can be associated with either your user or pro...

v0.7.0-rc2

The documentation for this version is available at the Earthly 0.7 documentation page.

Earthly CI

Earthly 0.7 is the first version compatible with Earthly CI.

Earthly 0.7 introduces the new keywords PIPELINE and TRIGGER to help define Earthly CI pipelines.

my-pipeline:
    PIPELINE --push
    TRIGGER push main
    TRIGGER pr main
    BUILD +my-target

For more information on how to use PIPELINE and TRIGGER, please see the reference documentation.

Podman support

Podman support has now been promoted out of beta status and is generally available in 0.7. Earthly will automatically detect the container frontend, whether that's docker or podman and use it automatically for running Buildkit locally, or for outputting images locally resulting from the build.

Please note that rootful podman is required. Rootless podman is not supported.

VERSION is now mandatory

The VERSION command
is now required for all Earthfiles, and an error will occur if it is missing. If you are not ready to update your
Earthfiles to use 0.7 (or 0.6), you can declare VERSION 0.5 to continue to use your Earthfiles.

Pushing no longer requires everything else to succeed

The behavior of the --push mode has changed in a backwards incompatible manner. Previously, --push commands would only execute if all other commands had succeeded. This precondition is no longer enforced, to allow for more flexible push ordering via the new WAIT clause. To achieve the behavior of the previous --push mode, you need to wrap any pre-required commands in a WAIT clause. For example, to push an image only if tests have passed, you would do the following:

test-and-push:
  WAIT
    BUILD +test
  END
  BUILD +my-image
my-image:
  ...
  SAVE IMAGE --push my-org/my-image:latest

This type of behavior is useful in order to have better control over the order of push operations. For example, you may want to push an image to a registry, followed by a deployment that uses the newly pushed image. Here is how this might look like:

push-and-deploy:
  ...
  WAIT
    BUILD +my-image
  END
  RUN --push ./deploy.sh my-org/my-image:latest
my-image:
  ...
  SAVE IMAGE --push my-org/my-image:latest

Where ./deploy.sh is custom deployment script instructing a production environment to start using the image that was just pushed.

Promoting experimental features

This version promotes a number of features that have been previously in Experimental and Beta status. To make use of
the features in this version you need to declare VERSION 0.7 at the top of your Earthfile.

Declaring VERSION 0.7 is equivalent to

VERSION \
  --explicit-global \
  --check-duplicate-images \
  --earthly-version-arg \
  --use-cache-command \
  --use-host-command \
  --use-copy-link \
  --new-platform \
  --no-tar-build-output \
  --use-no-manifest-list \
  --use-chmod \
  --shell-out-anywhere \
  --earthly-locally-arg \
  --use-project-secrets \
  --use-pipelines \
  --earthly-git-author-args \
  --wait-block \
  0.6

For more information on the individual Earthfile feature flags see the Earthfile version-specific features page.

Changed

• The behavior of the --push mode has changed in a backwards incompatible manner. Previously, --push commands would only execute if all other commands had succeeded. This precondition is no longer enforced, allowing push commands to execute in the middle of the build now. Previously under VERSION --wait-block 0.6.
• ARGs declared in the base target do not automatically become global unless explicitly declared as such via ARG --global. Previously under VERSION --explicit-global 0.6.
• The Cloud-based secrets model is now project-based; it is not compatible with the older global secrets model. Earthfiles which are defined as VERSION 0.5 or VERSION 0.6 will continue to use the old global secrets namespace; however
  the earthly command line no longer supports accessing or modifying the global secrets. A new earthly secrets migrate command has been added to help transition the global-based secrets to the new project-based secrets. If you need to manage secrets from Earthly 0.6 without migrating to the new 0.7 secrets, please use an older Earthly binary.
• All COPY and SAVE ARTIFACT operations now use union filesystem merging for performing the COPY. This is similar to COPY --link in Dockerfiles, however in Earthly it is automatically enabled for all such operations. Previously under VERSION --use-copy-link 0.6.
• The platform logic has been improved to allow overriding the platform in situations where previously it was not possible. Additionally, the default platform is now the native platform of the runner, and not of the host running Earthly. This makes platforms work better in remote runner settings. Previously under VERSION --new-platform 0.6.
• Earthly will automatically shellout to determine the $HOME value when referenced #2469
• Improved error message when invalid shell variable name is configured for a secret. #2478
• earthly ls has been promoted from experimental to beta status.
• Setting a VERSION feature flag boolean to false (or any other value) will now raise an error; previously it was syntactically valid but had no effect.
• SAVE ARTIFACT <path> AS LOCAL ... when used under a TRY / FINALLY can fail to be fully transferred to the host when the TRY command fails (resulting in an partially transferred file); an underflow can still occur, and is now detected and will not export the partial file. 2452

Added

• The commands PIPELINE and TRIGGER have been introduced for defining Earthly CI pipelines. Previously under VERSION --use-pipelines 0.6.
• The clause WAIT is now generally available. The WAIT clause allows controlling of build order for operations that require it. This allows use-cases such as pushing images to a registry, followed by infrastructure changes that use the newly pushed images. Previously under VERSION --wait-block 0.6.
• The command CACHE is now generally available. The CACHE command allows declaring a cache mount that can be used by any RUN command in the target, and also persists in the final image of the target (contents available when used via FROM). Previously under VERSION --use-cache-command 0.6.
• The command HOST is now generally available. The HOST command allows declaring an /etc/hosts entry. Previously under VERSION --use-host-command 0.6.
• New ARG EARTHLY_GIT_COMMIT_AUTHOR_TIMESTAMP will contain the author timestamp of the current git commit. #2462
• New ARGs EARTHLY_VERSION and EARTHLY_BUILD_SHA contain the version of Earthly and the git sha of Earthly itself, respectively.
• It is now possible to execute shell commands as part of any command that allows using variables. For example VOLUME $(cat /volume-name.txt). Previously under VERSION --shell-out-anywhere 0.6.
• Allow custom image to be used for git operations. #2027
• Earthly now checks for duplicate image names when performing image outputs. Previously under VERSION --check-duplicate-images 0.6.
• SAVE IMAGE --no-manifest-list allows outputting images of a different platform than the default one, but without the manifest list. This is useful for outputting images for platforms that do not support manifest lists, such as AWS Lambda. Previously under VERSION --use-no-manifest-list 0.6.
• COPY --chmod <mode> allows setting the permissions of the copied files. Previously under VERSION --use-chmod 0.6.
• The new ARG EARTHLY_LOCALLY indicates whether the current target is executed in a LOCALLY context. Previously under VERSION --earthly-locally-arg 0.6.
• The new ARGs EARTHLY_GIT_AUTHOR and EARTHLY_GIT_CO_AUTHORS contain the author and co-authors of the current git commit, respectively. Previously under VERSION --earthly-git-author-args 0.6.
• earthly doc [projectRef[+targetRef]] is a new subcommand in beta status. It will parse and output documentation comments on targets.
• Ability to store docker registry credentials in cloud secrets and corresponding earthly registry login|list|logout commands; credentials can be associated with either your user or project.
• New satellite commands for enabling auto-upgrades and forcing a manual upgrade.

Fixed

• Support for saving files larger than 64kB on failure within a TRY/FINALLY block. #2452
• Fixed race condition where SAVE IMAGE or SAVE ARTIFACT AS LOCAL commands were not always performed when contained in a target that was referenced by both a FROM (or COPY) and a BUILD command within the context of a WAIT/END block. #2237
• WORKDIR is lost when --use-copy-link feature is enabled with GIT CLONE or COPY --keep-own commands. Note that --use-copy-link is enabled by default in VERSION 0.7. #2544
• The CACHE command did not work when used inside a WITH DOCKER block. #2549
• The --platform argument is no longer passed to docker or podman, which caused podman to always pull the buildkit image even when it already existed locally. #2511, #2566
• Fi...

v0.7.0-rc1

This version promotes a number of features that have been previously in Experimental and Beta status. To make use of
the features in this version you need to declare VERSION 0.7 at the top of your Earthfile. The VERSION command
is now required for all Earthfiles, and an error will occur if it is missing. If you are not ready to update your
Earthfiles to use 0.7 (or 0.6), you can declare VERSION 0.5 to continue to use your Earthfiles.

Declaring VERSION 0.7 is equivalent to

VERSION \
  --explicit-global \
  --check-duplicate-images \
  --earthly-version-arg \
  --use-cache-command \
  --use-host-command \
  --use-copy-link \
  --new-platform \
  --no-tar-build-output \
  --use-no-manifest-list \
  --use-chmod \
  --shell-out-anywhere \
  --earthly-locally-arg \
  --use-project-secrets \
  --use-pipelines \
  --earthly-git-author-args \
  0.6

For more information on the individual Earthfile feature flags see the Earthfile version-specific features page.

Changed

• The Cloud-based secrets model is now project-based; it is not compatable with the older global secrets model. Earthfiles which are defined as VERSION 0.5 or VERSION 0.6 will continue to use the old global secrets namespace; however
  the earthly command line no longer supports accessing or modifying the global secrets. A new earthly secrets migrate command has been added to help transition the global-based secrets to the new project-based secrets.
• Earthly will automatically shellout to determine the $HOME value when referenced; this requires the --shell-out-anywhere feature flag. #2469
• Improved error message when invalid shell variable name is configured for a secret. #2478

Added

• New ARG EARTHLY_GIT_COMMIT_AUTHOR_TIMESTAMP will contain the author timestamp of the current git commit, this ARG must be enabled with the VERSION --git-commit-author-timestamp feature flag. #2462
• Allow custom image to be used for git opperations. #2027

Fixed

• Support for saving files larger than 64kB on failure within a TRY/FINALLY block. #2452
• Fixed race condition where SAVE IMAGE or SAVE ARTIFACT AS LOCAL commands were not always performed when contained in a target that was referenced by both a FROM (or COPY) and a BUILD command within the context of a WAIT/END block. #2237
• WORKDIR is lost when --use-copy-link feature is enabled with GIT CLONE or COPY --keep-own commands. #2544
• The CACHE command did not work when used inside a WITH DOCKER block. #2549
• The --platform argument is no longer passed to docker or podman, which caused podman to always pull the buildkit image even when it already existed locally. #2511, #2566

v0.6.30

Added

• Added support for a custom .netrc file path using the standard NETRC environmental variable. #2426
• Ability to run multiple Earthly installations at a time via EARTHLY_INSTALLATION_NAME environment variable, or the --installation-name CLI flag. The installation name defaults to earthly if not specified. Different installations use different configurations, different buildkit Daemons, different cache volumes, and different ports.
• New EARTHLY_CI builtin arg, which is set to true when earthly is run with the --ci flag, this ARG must be enabled with the VERSION --ci-arg feature flag. #2398

Changed

• Updated buildkit to include changes up to a5263dd0f990a3fe17b67e0002b76bfd1f5b433d, which includes a change to speed-up buildkit startup time.
• The Earthly Docker image works better for cases where a buildkit instance is not needed. The image now works without --privileged when using NO_BUILDKIT=1, and additionally, the image can also use /var/run/docker.sock or DOCKER_HOST for the buildkit daemon.

Fixed

• Fixed Earthly on Mac would randomly hang on 1. Init if Earthly was installed from Homebrew or the Earthly homebrew tap. #2247
• Only referenced ARGs from .env are displayed on failures, this prevents secrets contained in .env from being displayed. #1736
• Earthly now correctly detects if Podman is running but is under the disguise of the Docker CLI.
• Improved performance when copying files. Fully-cached builds are now dramatically faster as a result. #2049
• Fixed --shell-out-anywhere bug where inner quotes were incorrectly removed. #2340

v0.6.29

Added

• Cache mounts sharing mode can now be specified via RUN --mount type=cache,sharing=shared via CACHE --sharing=shared. Allowed values are locked (default - lock concurrent acccess to the cache), shared (allow concurrent access) and private (create a new empty cache on concurrent access).

Changed

• Increases the cache limit for local and git sources from 10% to 50% to support copying large files (e.g. binary assets).
• The default cache mount sharing mode is now locked instead of shared. This means that if you have multiple builds running concurrently, they will block on each other to gain access to the cache mount. If you want to share the cache as it was shared in previous version of Earthly, you can use RUN --mount type=cache,sharing=shared or CACHE --sharing=shared.

Fixed

• CACHE command was not being correctly used in IF, FOR, ARG and other commands. #2330
• Fixed buildkit gckeepstorage config value which was was set to 1000 times larger than the cache size, now it is set to the cache size.
• Fixed Earthly not detecting the correct image digest for some images loaded in WITH DOCKER --load and causing cache not to be bust correctly. #2337 and #2288

v0.6.28

Added

• A summary of context file transfers is now displayed every 15 seconds.
• Satellite wake command, which can force a satellite to wake up (useful for calling inspect or other non-build related commands).

Changed

• WITH DOCKER merging of user specific /etc/docker/daemon.json settings data now applies to arrays (previously only dictionaries were supported).
• A final warning will be displayed if earthly is terminated due to a interrupt signal (ctrl-c).

v0.6.27

Changed

• Support for all ssh-based key types (e.g. ssh-ed25519), and not only ssh-rsa. #1783

Fixed

• Unable to specify public key to add via the command-line, e.g. running earthly account add-key <key> ignored the key and fellback to an interactive prompt.
• GIT CLONE command was ignoring the WORK DIR command when --use-copy-link feature was set.

v0.6.26

Added

• Build failures now show the file and line number of the failing command
• Introduced EARTHLY_GIT_AUTHOR and EARTHLY_GIT_CO_AUTHORS ARGS

Fixed

• Some network operations were being incorrectly executed with a timeout of 0.
• Upon earthly ls failure it will display the failure reason

Changed

• Loading Docker images as part of WITH DOCKER is now faster through the use of an embedded registry in Buildkit. This functionality was previously hidden (VERSION --use-registry-for-with-docker) and was only auto-enabled for Earthly Satellite users. It is now enabled by default for all builds. #1268

v0.6.25

Fixed

• Fixed outputting images with long names #2053
• Fixed buildkit connection timing out occasionally #2229

v0.6.24

Added

• The new earthly org invite command now has the ability to invite multiple email addresses at once.
• Experimental support for TRY/FINALLY, which allows saving artifacts upon failure. #988, #587.
  Not that this is only a partial implementation, and only accepts a single RUN command in the TRY, and only SAVE ARTIFACT commands in the FINALLY block.
• Ability to enable specific satellite features via cli flags, e.g. the new experimental sleep feature can be enabled with
  earthly satellite launch --feature-flags satellite-sleep my-satellite.

Changed

• Bootstraping zsh autocompletion will first attempt to install under /usr/local/share/zsh/site-functions, and will now
  fallback to /usr/share/zsh/site-functions.
• The earthly preview org command has been promoted to GA, and is now available under earthly org.
• earthly sat select with no arguments now prints the current satellite and the usage text.
• The interactive debugger now connects over the buildkit session connection rather than an unencrypted tcp connection; this makes it possible
  to use the interactive debugger with remote buildkit instances.

Fixed

• Fixed Earthly failing when using a remote docker host from a machine with an incompatible architecture. #1895
• Earthly will no longer race with itself when starting up buildkit. #2194
• The error reported when failing to initiate a connection to buildkit has been reworded to account for the remote buildkit/satellite case too.
• Errors related to parsing VERSION feature flags will no longer be displayed during auto-completion.