Improve the liveness probe for the Helm Chart's Worker StatefulSet.

[topherbullock]

By default the workers will retire (and then be pruned) if the following are logged by the worker:

guardian.api.garden-server.create.failed

baggageclaim.api.volume-server.create-volume-async.failed-to-create

This means that any failure to create a container will retire the worker. Creating a container can fail for any old reason, even a poorly written task, so this is less than ideal.

[ralekseenkov]

we filed this a while ago, with one specific example
#2656

[william-tran]

lifecycle related: #2757

Should this be changed? a worker coming back up after not having cleaned itself up yields this error: guardian.attach.lookup-failed

What about btrfs errors? disk out of space errors? We really need to disambiguate any old error from ones that require a worker restart to fix.

[topherbullock]

@ralekseenkov Thanks for linking that! Definitely sums up the exact case we figured would cause the liveness probe to retire the workers.

@william-tran

We really need to disambiguate any old error from ones that require a worker restart to fix.

I'm thinking we'll likely want a health check of some sort on the worker itself rather than rely on logs. We don't have much control over Garden's logs, so differentiating the root cause behind a failure to create purely by reading logs would be a pain.

[topherbullock]

@cirocosta and I are spiking on a worker component to ping Garden (via the /ping endpoint) and Baggageclaim (no /ping .. but we'll figure something out.. ).

Will add more details on what we discover.

[cirocosta]

Hey @topherbullock ,

I just submitted to concourse/charts a branch that, like concourse/dex, keeps a bunch of PRs / branches merged into a single view:

• https://github.com/concourse/charts/blob/maintenance/run.sh

That allows us to reference concourse/charts#merged and move forward with the work.

Do you think that is enough to consider this issue not blocked?

Thx!

[cirocosta]

Hey,

On the last Friday, I started working on making those health checks a bit more in-depth, creating a minimum load in the worker to inspect whether it's still capable of serving what its purpose (creating containers and volumes) - https://github.com/cirocosta/concourse-worker-health-checker.

Here's a quick walk through the code:

main.go

	aggregate := &checkers.Aggregate{
		Checkers: []checkers.Checker{
			&checkers.Baggageclaim{Address: *baggageclaimUrl},
			&checkers.Garden{Address: *gardenUrl},
		},
	}

where Checker is an interface:

checkers/checker.go

type Checker interface {
	// Check performs a health check with its execution time
	// limited by a context that might be canceled at any
	// time.
	Check(ctx context.Context) (err error)

So that a Baggageclaim checker looks like:

checkers/baggageclaim.go

func (b *Baggageclaim) Check(ctx context.Context) (err error) {
	handle := mustCreatedHandle()

	err = b.createVolume(ctx, handle)
	if err != nil {
		err = errors.Wrapf(err,
			"failed to create volume %s", handle)
		return
	}

	err = b.destroyVolume(ctx, handle)
	if err != nil {
		err = errors.Wrapf(err,
			"failed to delete volume %s", handle)
		return
	}

	return
}

and Aggregate provides a concurrent execution step which can be timed-out by a parent context:

checkers/aggregate.go

func (h *Aggregate) Check(ctx context.Context) (err error) {
	var group *errgroup.Group

	group, ctx = errgroup.WithContext(ctx)
	for _, checker := range h.Checkers {
		checker := checker // goroutine closure

		group.Go(func() error {
			return checker.Check(ctx)
		})
	}

	err = group.Wait()
	return
}

This way we can, in the future, increase this with either even more in-depth checkers or some others that could look into things like btrfs metadata usage or whatever we see fit.

I started that repo as something separate that gets deployed as a sidecar to the worker as a way of detecting any problems that might arise with it, but the intention is to get it upstream with the code being executed on every readinessProbe.

Wdyt?

thx!

[cirocosta]

Hey,

Just a quick update; #3025 adds the functionality I described above.

Thanks!

[cirocosta]

Hey, I'll consider this one as closed as in master (see a7e13ce) we already have the healtcheck endpoint there which is used by the release candidate chart.

Work goes on in #3025 👍