Traffic Vault: Process <0.325.0> with 0 neighbours crashed with reason: no match of right hand value {error,closed}

[zrhoffman]

I'm submitting a ...

• bug report

Traffic Control components affected ...

• CDN in a Box
• Traffic Vault
• CI tests

Current behavior:

In CDN in a Box, Traffic Vault sometimes crashes when starting up with this message:

[error] <0.325.0> CRASH REPORT Process <0.325.0> with 0 neighbours crashed with reason: no match of right hand value {error,closed} in mochiweb_http:loop/2 line 56

See Anything else for context in the logs.

This affects our CI because the CDN in a Box Readiness check will sometimes fail due to this (after timing out after 12 minutes of attempts to contact a Delivery Service).

Expected behavior:

Traffic Vault should start successfully. The Traffic Vault image should include a health check that attempts to query the ssl buck and fails if it cannot. If we do this, we can

• Make Traffic Vault exit when the health check fails
• Make Traffic Vault restart when it exits

which would fix this issue with an additional cost of 1-2 minutes of startup time when the issue occurs.

Minimal reproduction of the problem with instructions:

1. Start CDN-in-a-Box and watch for that error in the Traffic Vault logs

2. If Traffic Vault gets all the way to

   <0.2189.0>@yz_fuse:create:73 Creating fuse for search index sslkeys

   but the error is not there (the sslkeys being created does not mean the crash did not occur), docker-compose restart trafficvault and try again

This error is uncommon to reproduce, so don't feel like you need to reproduce it to fix this issue.

Anything else:

The log is in a gist (https://gist.github.com/zrhoffman/58850e41d124b0a49acd3203699661d9) because it's 951 lines

[zrhoffman]

Closing #5182 because the Riak Traffic Vault is deprecated