Proposal: Support for running commands on the host system

[vladaionescu]

Background

Earthfiles are meant to be the entire build definition for a project and not be wrapped in anything else. The reason is that we want no additional layers between the CI and Earthly which could create complexity / confusion / difficulty reproducing CI failures.

However, sometimes developers need to perform operations outside of containers, for development purposes:

• Running the app they are working on
• Running docker containers with host volumes mounted (eg for live reloads)
• Starting a docker-compose stack
• Resetting the dev database state
• Hooking up a debugger
• Etc etc etc

This kind of operations need to be executed on the host environment (or using the host docker daemon).

Traditionally people have used a combination of Makefile recipes or bash scripts to store long commands that they use regularly. This is a proposal for features in Earthly to satisfy this use case.

Proposal

The proposal is to have special Earthly targets that execute the commands on the host system directly, rather than in containers.

Wrap common dev commands

Here is an example use-case: wrapping common dev commands.

unsafe host compose:
  ENV COMPOSE_FILES="-f file1.yml -f file2.yml -f file3.yml -f file4.yml"

unsafe host up:
  FROM +compose
  RUN docker-compose $COMPOSE_FILES up -d "$@"

unsafe host down:
  FROM +compose
  RUN docker-compose $COMPOSE_FILES down "$@"

In this example, the Earthly is used to wrap a complex docker-compose command into simpler forms that can be invoked with just earth +up, earth +down.

Build + start app in one go

Another example is using Earthly to build a series of containers (or making sure that they are up-to-date) and also starting (or restarting) them in a docker compose stack.

all-images:
  BUILD +image1
  BUILD +image2
  BUILD +image3

unsafe host up:
  BUILD +all-images
  RUN docker-compose up -d

How does it work

Notice first of all, the use of unsafe host in the declaration of a target. This is what informs Earthly to run the instructions on the host.

What commands do:

• BUILD +some-target - if the target is of type unsafe host, then simply execute the instructions within (calling targets from other dirs is supported - system cd's into the right dir automatically - but calling a remote target is not supported here). If the target is not unsafe host, run earth +some-target.
• RUN ... - executes the command on the host, in its own shell (eg exporting env vars or cd to different dirs is not propagated from one RUN to another).
• ARG ... - declares an arg, as usual. Variable args are allowed.
• ENV ... - declares an env, as usual.
• FROM ... - executes the commands in the mentioned target before continuing. The referenced target has to be of type local. The key difference from BUILD is that any ENV is propagated in the case of FROM, whereas for BUILD it is not.
• No other commands are supported.

Referencing a local target from a regular target would not be allowed.

Possible extension

Being able to reuse a set of host commands in the context of a containerized build. Example use-case: the same commands to bring up a docker-compose stack could be used both as a local dev environment and also as a setup step for an integration test that runs within Earthly.

In this case, we could use the word "snippet" to declare such special targets. These snippets can then be exectued within regular Earthly targets. The word unsafe can be used in the execution of the earth command itself, as syntactic salt: earth --unsafe +my-snippet.

Possible issues

Executing commands on the host is exactly the kind of thing Earthly stands against :-) . If misused, the user goes back to builds that cannot be easily reproduced. The different commands may do wildly different things on different systems (linux vs mac vs windows) and also caching and idempotency are lost. This proposal could be mistaken for some kind of April fools joke :-)

Also, the language becomes a little more complicated and possibly confusing for new users. It's easy to miss the local keyword and be confused as to how the build actually works.

However, there may be very valid use-cases for this.

Ask

Please upvote / downvote this issue to signify interest. Also, if you have additional example use-cases for such a feature, please leave a comment.

[idelvall]

-1. As you pointed out this would jeopardize reproducibility, also increase complexity and blur earthly responsability. Most important, the gains are weak from my perspective. You could always have a script that runs earthly, and host commands right?

However, there may be very valid use-cases for this

Can you elaborate more on this?

[vladaionescu]

You could always have a script that runs earthly, and host commands right?

Yes... although what we want to do is have Earthly be the only script a CI needs and not have to wrap it in anything.

Can you elaborate more on this?

Mainly the use-cases mentioned in the background section.

[idelvall]

What about a command for running webhooks? This shouldn't compromise reproducibility and would allow for integration with external resources

[ihcsim]

@vladaionescu I see this as a good "break the glass"/"know what you are doing" feature. Most projects would have some form of existing build scripts to handle local development and CI already. It will be a great adoption experience if earthly can work with existing stuff.

Instead of new local <command> targets, how about just a new LOCAL_EXEC command in the recipe: E.g.,

build:
   LOCAL_EXEC my_build_script.sh # you chose this life

[vladaionescu]

One issue we might have with a single command is the interaction with anything around it. eg

RUN ...
COPY ...
LOCAL_EXEC ...
RUN ...

Would that LOCAL_EXEC interact with the others? If not... then it's very confusing that it's doing two completely unrelated things. Also, another concern is how that would work with the layer caching. Eg if cache is bust at the COPY, would it run LOCAL_EXEC again?

I think it's much cleaner if such operations are isolated in their own special target type.

[ihcsim]

My initial thought is that LOCAL_EXEC will be used for running self-contained scripts that have their own commands to build, run, test artifacts. Won't you have the same issue with using targets? E.g.,

local up:
  FROM +compose
  RUN docker-compose $COMPOSE_FILES up -d "$@"
  RUN ...
  COPY ...
  RUN ...

If the goal is to allow users to perform operations outside of containers, cache support will be very limited to begin with, right?

[vladaionescu]

If the goal is to allow users to perform operations outside of containers, cache support will be very limited to begin with, right?

Yes, there would be no caching for local targets at all.

[adamgordonbell]

+1 for this feature. The wrapping docker-compose up and down would be for starting integration testing dependencies in CI as well.

[vladaionescu]

Closing in favor of #167 EDIT: Closing in favor of #580

[JensRantil]

Closing in favor of #167

Recurse :-)

[paynecodes]

Closing in favor of #167

Anyone know which issue is the correct issue regarding these use cases?

[vladaionescu]

LoL - my bad. It's #580