Idea: allow interactive "on-demand" access to debugging session

[rrjjvv]

What existing functionality needs improvement?

The earthly --interactive feature is one of my favorite and most heavily used features, especially in tandem with WITH DOCKER RUN. This works great in tandem with forced/targeted failures (i.e., inserting an && exit 1 within the RUN). But sometimes you spot an issue earlier in the chain, or you don't have already have an injected failure. As we know, WITH DOCKER RUN has considerable overhead. It would be really nice to not incur that overhead for the sake of finding the exact spot to inject the exit 1.

I commonly work with kubernetes, and in particular, running tests using kind via WITH DOCKER RUN. In my current project, I have nine --load args that come to around 3.4GB. With a fully cached build, it takes 'only' about 20 seconds to get the images loaded into the internal registry (it was much longer prior to the recent change that parallelizes the loads). With how kind works, those images are 'pulled' (from the internal registry), only to be tarred and untarred a few times to load them into the 'nodes'. That takes about 70 seconds. All-in-all, for a fully cached build, there's a fixed cost of 90-100 seconds before the actual tests execute (which take substantially longer).

If I see something wrong that needs investigating, I need to either:

1. wait for my manufactured exit 1 to get hit (if I have one) or,
2. wait for the build to naturally fail on its own, or
3. abort the build immediately (thus hitting Interrupting WITH DOCKER RUN accumulates d-in-d cruft  #3696) in order to inject an exit 1 for a follow-up build.

Each one of those entails incurring a time penalty (and trying to anticipate which is the least bad). Being able to Ctrl-C and drop into a debugging shell immediately would remove that penalty altogether.

This feature request and my recent bug report (#3696) are very related. As I noted, this particular project generates a whole lot of docker data (for beyond the 3.4GB of images). This feature would not technically be a fix for that bug, but it would effectively fix it for me.

Expected Behavior

The expected/desired behavior is that (when running with earthly --interactive) when I send Ctrl-C when executing a RUN instruction, I drop into a debugging shell (the same experience had the RUN exited non-zero).

Misc:
My primary interest is for a RUN within a WITH DOCKER context; if there's a technical reason why that would be easier than a general RUN, I would still be thrilled with that limitation. On the other hand, having it available for only standalone RUN wouldn't be of much value.

I proposed SIGINT (Ctrl-C) since that seemed 'least surprising'. I would be perfectly fine with a different signal (e.g. SIGUSR1) if there was a technical or aesthetic reason that SIGINT shouldn't be used. Or even something like a posting to a message bus... as long as it's interactive.