Cache failed builds

[BuildStream-Migration-Bot]

See original issue on GitLab
In GitLab by [Gitlab user @sstriker] on Aug 25, 2017, 11:01

When inputs don't change, there is no reason to expect a build to succeed a second time. Currently when executing bst build on a bst that previously failed will retry the build.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Aug 25, 2017, 19:07

Dear user, your project produces the artifact gtk+.bd5k8sd3fs49fse55..... which is known to fail. If you want to try building it anyway, try again with bst build --force ...

Interesting indeed.

We could generate artifacts for failed builds and share those failures with peers too. A user trying to build something which failed could just end up downloading a log file and seeing what went wrong.

And of course I guess you have to be allowed to bypass it and build it anyway in case you are interested in debugging it in a shell ?

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @sstriker] on Aug 25, 2017, 21:07

Thanks for expanding.

Yes indeed, it should be overridable as you sketch.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @hashpling] on Aug 29, 2017, 10:12

Even running in a sandbox, there are events where you might expect the a build to succeed when run a second time. One example, which has happened more than once, is running a build in Docker with the maximum allocated memory set too low. Re-running the identical build with sufficient memory can be expected to succeed.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @sstriker] on Aug 29, 2017, 18:06

I would still be ok with having the explicitly retry those builds than to re-run them by default. Whether we have a global override switch which always tries to build is another question.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Mar 30, 2018, 05:50

Note that [Gitlab user @hashpling]'s concern might be addressable with some additional load balancing or configuration, this was raised separately in #185 and should probably be discussed there instead, maybe there are some creative solutions we can come up with.

An additional note I have to add to the subject of caching failed builds, is related to how we handle local build failures.

I think that if we add functionality to:

• Cache build directories in artifacts for later use (a feature which has been developed and should land as a part of Opening a workspace with a cached build #311)
• Cache failed build artifacts, which should include the build directory in it's failed state along with anything which might have been produced in %{install-root}
• Allow sharing of failed build artifacts
• Allow debugging of a failed build state, as a feature of bst shell, and as a possibility for force retrying in bst build

Then we should have a single story for retrying a failed build and not separate ones depending on whether it failed locally or remotely.

Essentially I think that when we cache a failed build artifact, we should immediately delete the local build directory, as it will be in the cached artifact anyway. This will solve the issue of unmanaged, ever growing build directories in ~/.cache/buildstream/build at the same time.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @richardmaw-codethink] on Apr 23, 2018, 17:06

The caches currently lack any interface to get metadata
independent of the contents of the artifact.

Currently the first we know about an artifact's contents
is when we are staging it to assemble a dependent.

If we were to just add failed builds to the cache
and build result metadata to the artifact
then we would have to fail the first time the artifact is extracted.

This would likely be an improvement in responsiveness
compared with attempting a build that will fail,
since extracting would be less work than building,
but if we knew sooner we could opt to fail before starting any builds
or would know to enqueue a rebuild.

If we were to make the queue more dynamic
we could solve the early-failing or rebuilding issue.

We could enqueue extract jobs before build jobs,
staging from the pre-extracted artifacts,
and if it's a failed build artifact then enqueue a rebuild
before dependent builds of the failed build artifact.

Alternatively we could during the staging stage of a build
if the artifact being extracted was a failed build
queue a rebuild job of both the failed build and the current build
and terminate the current build job.

Instead of making the queue more dynamic
we could make the cache able to distinguish successful from failed builds,
and keep roughly the same architecture.

If we care about cache compatibility between versions of buildstream
we could bodge the fact that it's a failed build into the cache key
(either by extending the name with a token or adding it to the computation)
and when looking in the cache for an artifact, check both keys.

Unless it's an obvious and parseable part of the name
this wouldn't be accessible to cache cleanup policy logic.

An alternative would be to maintain a separate cache for failures,
though this would require changing a lot of code paths to expect more caches
or to have a wrapper cache that operates on both,
though this could require changes to the
this wouldn't be accessible to cache cleanup policy logic.

An alternative would be to maintain a separate cache for failures,
though this would require changing a lot of code paths to expect more caches
or to have a wrapper cache that operates on both,
though this could require changes to the
this wouldn't be accessible to cache cleanup policy logic.

An alternative would be to maintain a separate cache for failures,
though this would require changing a lot of code paths to expect more caches
or to have a wrapper cache that operates on both,
though this could require changes to the API.

A more long-term alternative would be to include metadata alongside content,
but while we could probably do that fairly easy in the tar cache
by adding a nice machine parseable file alongside,
I've not yet any suggestions for how best to handle it with ostree yet.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @richardmaw-codethink] on Apr 23, 2018, 17:48

I'm going to have a deeper look at options for metadata in the caches and put together a proposal for what to do, then barring objections, begin implementing it.
I just thought my write-up would be useful if anyone has time to comment.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @richardmaw-codethink] on Apr 24, 2018, 17:51

OSTree has a couple of ways of handling metadata.

There's "detached metadata" for metadata describing a commit
that may change over time (which could be useful for tracking usage),
or the commit may include as much additional metadata as wanted,
though affects the OSTree cache ID this shouldn't be an issue
since the Buildstream Element cache ID wouldn't change.

By extending the Artifact Cache API to include a method for getting the metadata
the Pipeline planner could, in the same way that it's told to ignore the cache,
be told to ignore cached failures and rebuild, or decide it's not buildable.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @richardmaw-codethink] on Apr 26, 2018, 15:40

The Google Content-Addressable Store itself operates on a lower level than artifact caches.
It doesn't have a native mechanism for metadata, presumably because you could instead build your own mechanism with a top-level directory containing a subdirectory of the contents and a separate one for the metadata.

Given the goal is to be able to fetch metadata to make a decision on whether a build was a failure it's unfortunate that we can't use the GetTree method of the Remote Execution API for this, since from my understanding it will download all the tree data in an arbitrary order rather than just a requested subtree.

In terms of the inputs and outputs, the Artifact Cache seems closer in concept to the Remote Action Cache, the difference being that our caches are provided with the key rather than computing it from the build description.
A Remote Action Cache ActionResult includes the exit_code metadata which could encode whether a build was a success or a failure.

As a result I don't think we can so comfortably include arbitrary metadata in the artifact cache, but we could include a result, which could either be an enum from wrapping commands we send to a Remote Action Cache service, or an arbitrary exit code.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Apr 27, 2018, 07:20

Hmmm, ok so at first sight of this, I find the description is misleading.

The goal of caching failed builds is indeed partly to avoid building things which we can safely predict will fail, but I feel that the bigger motivation here, is to allow sharing of the failed build artifacts through a cache server.

I.e.: When a build fails on a CI server, currently all you have is the logs; instead, we should be able to download the related artifacts such that we can run a bst shell on them and debug what went on remotely, locally.

[Gitlab user @richardmaw]-codethink: I think that you are overthinking things by desiring to get the ArtifactCache to perform partial extractions.

Rather, I think that we might benefit from coupling the artifact extraction with the artifact creation:

• If we built an artifact, there is a very high likelyhood that we will later need to extract it
  • Currently when a build completes and separate builds blocking on that build begin simultaneously, we have separate extractions which occur in parallel, and the winner replaces the extract dir while the loser's work in extraction is discarded. This is an overhead which we would reduce by forcing the extraction to occur while caching the artifact
• If we downloaded an artifact, the same cost is likely also incurred when racing to extract the artifacts at build time
• If it is a failed artifact which we build
  • The likelyhood that we want to shell into it is high, this will anyway require an extraction
• If it is a failed artifact which we downloaded
  • Here is one place where we might want a bit more optimization, the likelyhood that we want to extract it is a bit lower

However, I should further note that:

• With OSTree, checkouts of individual full artifacts should be very quick, and not require additional storage, they will be done with hardlinks.
• With the upcoming CAS, I expect checkouts to be similarly optimized

In this light, I think that things will be faster overall if we extract at cache time, and that the overhead for the very few artifacts which we did not need to really extract is sufficiently negligent to not merit overengineering the artifact caches for partial extractions.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Apr 27, 2018, 08:25

[Gitlab user @richardmaw]-codethink I think I have an idea about this.

From what I understand, a tricky part of implementing this is that we assume:

• We are running bst build target.bst
• We have downloaded a failed dependency.bst artifact

In this case, we probably want to handle things at the scheduler Queue level.

• Elements pass through queues, such that they go through PullQueue before ending up in BuildQueue
• When an element arrives in the BuildQueue, but it is already cached, we currently skip the element
• Instead of skipping the element, the BuildQueue should mark the element as failed, and go through the regular process of handling a failed build.
• This also requires that we have a new Element._failed() state which can be observed from the Queues, in addition to other exposed states like Element._cached(), this will be fairly symmetric with other existing code, and should be resolved withing Element._update_state().

As I mentioned in https://gitlab.com/BuildStream/buildstream/issues/76#note_65830031, with this we should have a uniform way of debugging failed builds, and this should probably be the default mode of operation.

I would expect that if I have downloaded a failed build, I will be presented with an interactive prompt, and handle things in exactly the same way as if I had failed to build the artifact myself.

Note that reconstructing a build environment from a failed build artifact also depends on the caching of the build tree, which is being worked on in #311 and is in turn, also blocking on #376.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @richardmaw-codethink] on Apr 27, 2018, 18:00

A lot of my ramblings are irrelevant given an artifact's contents is extracted and retained.

I'd assumed that extract happened directly into the staging area, not noticing that splitting is handled after being extracted from the cache rather than before they were put in.

This simplifies things for me, though it feels weird that we're effectively keeping two caches.

From my reading of the job planner logic it could also omit including the element in the queue in the first place, though I'd have to double check.

While a cached failure would let us skip the build of the failed element itself, we could also skip all the other builds before we start them if it was our policy to treat a cached failure as a build fail.