implement WAIT block to allow finer control of push operations

[alexcb]

Earthly is implementing a new WAIT / END block, which will be used to control the order of commands that result in pushes.

A quick example of the new syntax:

VERSION --wait-block 0.6

img:
    FROM alpine
    RUN ...
    SAVE IMAGE --push user/img:tag

deploy:
    WAIT
        BUILD +img
        BUILD +some-other-target-in-parallel
    END
    RUN --push run ./deploy user/img:tag

This syntax came about from discussions surrounding #1836

Current status:

• should now be functionally complete; however, it is still experimental.
• WAIT/END doesn't export to remote cache #2178

Remaining:

• docs
• community testing

[JensRantil]

👋 SRE here - and a little late to the party - but I feel like the background reasoning is a little flawed around deploys. How do you know that a deploy has succeeded just because the docker push succeeded? Majority of deploy systems are not synchronous; For example, K8s load balancers and AWS load balancers are async in bringing in new services. I have a sense that we might be adding another feature to Earthly here that doesn't actually solve the root issue.

IMO the real solution is to not make atomic changes to multiple services. There be so many dragons there. Make many smaller deploys and make sure you have monitoring in place to know if it failed.

Finally, I'd actually expect my microservices to start even through its dependency is down. It should be resilient to that kind of hickup. The service will fail, for sure, but I expect my system to auto-heal. See also this.

Edit: Please ignore this message. I reread #1836 and misunderstood things!

[alexcb]

Thanks for the note @JensRantil,

How do you know that a deploy has succeeded just because the docker push succeeded?

We don't -- all we now is a push has been received by a registry; if the registry returns with a success, earhly will now proceed to a following RUN command (previously all SAVE IMAGE --push commands were collected and executed after all RUN commands had finished).

There be so many dragons

I agree -- software is hard!

This change is to make it possible for users to mix the two operations -- ultimately leaving it up to them for how they wish to define their targets to perform builds, tests, and maybe even deploys (via their own custom scripts).

I'd actually expect my microservices to start even through its dependency is down

I agree that it's a good practice to have services deal with failures (e.g. re-attempting to connect to a database if its down); however, I'm confused how this comment pertains to earthly (and this new WAIT feature).

[alexcb]

This has been released under https://github.com/earthly/earthly/releases/tag/v0.6.23

It should be functionally complete; however, it should still be considered experimental and must be enabled with VERSION --wait-block 0.6

[JensRantil]

👋 I reread #1836 and now I understand my reasoning was wrong. Moonwalking out of this... 😅