Health check for shared configuration correctness

[callumdmay]

We operate a DSS pool shared amongst multiple different organizations, and a recurring challenge we face is effectively verifying the correct configuration of newly deployed DSS instances, particularly ensuring they are connected to the same underlying data source as existing instance.

The current InterUSS DSS implementation offers a valuable /healthy endpoint for general service availability. Building on this, we were thinking it would be really helpful to have a similar endpoint that reads a unique, hard-to-guess value (like a UUID) from a special health check table in the database. This would give us a simple way to confirm that all DSS instances are pointing to the same data source by checking if they all return the same value.

Ideally, for our use case, we'd love to not only verify new setups but also continuously monitor the health of our pool. Perhaps this could be extended to allow the unique value to change periodically (either through another endpoint or based on a configurable time interval). This would provide a simple and effective way to ensure all DSS instances remain correctly configured over time.

Happy to answer questions or provide additional clarification!

[BenjaminPelletier]

/healthy is nice because it's extremely lightweight, but I agree it does not provide a great deal of information -- it doesn't even verify connectivity to the underlying database, let alone successful pooling with other DSS instances. It seems like there should be a large amount of value in a DSS operator being able to answer the question "Is the DSS pool configured and working the way I expect it to be?" and it seems like this would essentially do that. Especially with increasing levels of operationalization, it seems like this kind of feature is becoming more and more important and urgent.

Outline 1

Brainstorming, what do you think of this implementation outline?

1. Add a pool_participants table in the database with "DSS instance" and "Last heartbeat" columns
2. At startup, a DSS instance upserts its DSS instance ID and the current time into that table
3. DSS exposes an aux endpoint to query pool participants, which basically returns the contents of the pool_participants table
4. A USS verifies successful initial pooling by verifying that the expected participants are present in that response
5. DSS also exposes another aux endpoint/method that updates its "Last heartbeat" value to the current timestamp, and the DSS instance operator has its automation call that endpoint as frequently as desired/agreed by other pool participants
6. A USS verifies ongoing pool integrity by verifying that all expected participants have a "Last heartbeat"

Variants to Outline 1

There are many more specific decisions if that overall outline seems good to everyone, but I think we would want to settle on a high-level outline like Outline 1 before diving into specifics like these:

1. :
   a. pool_participants table could be an entirely new category of tables (sibling of rid and scd), or there could be a table like this for each service (one for rid, a different one for scd)
   b. Maybe instead of a separate table, an actual operational intent with a magic property (indicating that it's not a real operational intent) could be added to the op intents table -- this would provide additional verification coverage since the presence of the pool_participants table doesn't necessarily indicate the presence of the scd schema tables (hypothetically, the pool could be initialized for RID only and not SCD and pool_participants would still suggest success)
2. DSS instance ID could be its locality string, or perhaps some form of audience identities, or something else
3. This endpoint could be protected by access tokens with a custom scope, or something else
4. :
   a. A USS could also query other instances' endpoints to verify they all show the same thing, or just rely on their own instance's endpoint (probably sufficient)
   b. The table could just contain the plain information, or could also include some kind of proof that the heartbeat was from the expected recipient (for instance, perhaps an additional column containing an access token that only the participant could have retrieved from the auth server)
5. The DSS instance itself could automatically periodically call this function instead of relying on the automation of its owning USS (but that would add more moving parts that could fail), or the USS could be relied upon to call the endpoint which would incidentally verify that a bit more of the USS's system was also functioning properly (in addition to its DSS instance)
6. ...or by verifying that polling from all DSS instances yields consistent results, or some other criteria

Ops generalization

It seems like a pool_participants table could have utility beyond just this use case; by adding a few more columns, it seems like most of the read aspects of #1140 could be implemented almost for free (though this wouldn't be compatible with variant 1b).

@mickmis @barroco @the-glu do you have any thoughts? It seems like this would be a valuable addition in the relatively short-term.

[barroco]

Overall outline 1 looks good to me and seems to achieve the goal to verify correct configuration and pooling of DSS instances end to end. It would be interesting to outline some example of misconfiguration you have observed.

[BenjaminPelletier]

One important application is simple but sophisticated production deployment. We have manual calls, the legacy prober, and automated testing suites which can be used to verify the configuration and functionality of a production deployment, but they do so in a fairly invasive way -- specifically, they actually emplace flight data into the real production system. This could potentially be limited to an area on earth that is guaranteed not to have real flights (like in the middle of an ocean near a pole), but if that attempt to quarantine the test/verification data failed, the consequences could be significant. So, presumably the operation/execution of the automated test tools would need to have a very high level of productionization (low-latency on-call support, specific playbooks for all foreseen problems along with training on those procedures, etc). That imposes either a high burden on the verifying USS, or else a relatively high-impact risk on production operations (that they could be disrupted by the less sophisticated verification operations).

Furthermore, verification in a separate geographic area in production requires the system to behave differently than "real life" -- for instance, the system might need to be told to assume that the small test parcel was under US jurisdiction (subject to US flight rules) even though standard jurisdictional maps would disagree with that assumption. So, even automated testing in production does not necessarily exercise the system in as full of a way as true, real operations.

One benefit of this approach is that the need to ensure high-quality verification ops in prod is avoided by building deployment verification into the core DSS product rather than relying on a separate product. This wouldn't replace software and system design verification which would continue to be performed in a qualification environment, but it would be a strong alternative for verifying that the production deployment was actually deployed correctly and equivalently to the system that was more thoroughly tested in the qualification environment.

[callumdmay]

The proposed approach looks solid. If we wanted something simpler that wouldn't require considerations around instance identifiers or changing startup functionality we could just use a single row in the table with a timestamp and a uuid. Then provide an endpoint that does a simple RMW transaction:

start db transaction

read row

if row.expiration is stale:
  write new uuid with timestamp = now

return row.uuid

end db transaction

Doing this atomically within a transaction means that every DSS instance can be polled simultaneously and will return the same uuid value. No statefulness required.

This solution lacks the ability to verify another DSS instances connectivity unilaterally through their own endpoint (it doesn't return USS-specific data) but this is by design - We are hoping for a solution that ensures connectivity down the entire stack from http ingress to the database, which requires that each DSS instance be verified through their public API.

[the-glu]

A few thoughts:

• Shouldn't the DSS update their own "Last heartbeat" them self, at regular intervals? It feels strange that something external, must be used/relied on to do so, if the 'external' thing fail, the DSS would be marked as down when he isn't. That also mean all DSS must be regularly called, when you may have only one DSS operator that his pooling/checking his own DSS. If the DSS is self-updating himself, that would make the feature self-contains and without external dependencies.

• The DSS could also expose in various endpoints/DB columns his expected heartbeat interval, meaning others DSS / external callers can just compare last heartbeat / expected internal / current time to know if a DSS is unhealthy, and allow, if needed more flexibility with multiple HB values.

• In term of available endpoints, if you want easy monitoring, you may want to add a specifc one on the DSS that is doing the validation and/or add a validation 'sugestion' with all data, allowing a simple monitoring system to just check e.g. the http return code, and a smart monitoring system using all data to do his own computations

[mickmis]

I think this raises the question of who has the authoritative list of DSS instances. Is this new table supposed to be that authoritative list?
If yes, it should be maintained and notably retired instances removed from that table. But whose responsibility is that? However doing so would indeed address a lot of #1140.
If not, how clients are supposed to get the knowledge of the different instances in the pool? Which IIUC is required with that outline to know the health of the pool as a whole.

Other than that, outline 1 sounds reasonable.

Also, it may be worth distinguishing heartbeats (DSS instance-initiated periodic alive signal) from some health check / self-tests (DSS instance or operator -initiated self testing of basic functions).

[the-glu]

About removed instance, an automatic removal after 'x hours' without an HB would be an option without a specific responsibility but a shared one

[mickmis]

About removed instance, an automatic removal after 'x hours' without an HB would be an option without a specific responsibility but a shared one

But then when an instance is offline for long enough, while still part of the pool, the entry will get removed and that will make the pool appear healthy. Which would not be true.

[BenjaminPelletier]

Shouldn't the DSS update their own "Last heartbeat" them self, at regular intervals?

IMO, maybe. From a practicality standpoint, background tasks are more moving parts that can fail so external is slightly better, though we do already have a garbage collector running as a background task so this may not be too hard to add. But, the conceptual argument for external IMO is that a DSS instance is just a USS component and shouldn't be thought of as a stand-alone service. If a USS's oversight functionality (monitoring & alerting) isn't functioning properly, their DSS instance may currently still be able to serve traffic correctly, but their overall DSS subsystem is not functioning fully correctly -- in that case, it seems valuable to detect the subsystem failure even though the DSS as its own service is still functioning properly.

In any case, even if we had a background task automatically initiating heartbeats from within the core-service container, I would expect us to make that a configurable option so that users could adopt a monitoring & alerting strategy that best fits their needs.

The DSS could also expose in various endpoints/DB columns his expected heartbeat interval, meaning others DSS / external callers can just compare last heartbeat / expected internal / current time to know if a DSS is unhealthy, and allow, if needed more flexibility with multiple HB values.

Adding a next_heartbeat_expected_before field for the current heartbeat seems like a good idea to me.

In term of available endpoints, if you want easy monitoring, you may want to add a specifc one on the DSS that is doing the validation and/or add a validation 'sugestion' with all data, allowing a simple monitoring system to just check e.g. the http return code, and a smart monitoring system using all data to do his own computations

That does sound desirable, but I'm not sure we can capture all the checks that may be relevant. For instance, a DSS operator likely expects their pool to contain a certain set of participants (e.g., USS1, USS2, USS3) and I'm not sure how this endpoint would accept specification of that expectation, especially during pool additions and removals. I certainly think this is a good idea where practical though, like for instance evaluating whether the heartbeats of all registered participants are timely (per previous note).

I think this raises the question of who has the authoritative list of DSS instances. Is this new table supposed to be that authoritative list?

Ultimately, "authoritative" is the set of CRDB nodes constituting the CRDB cluster (or equivalent concept in YugabyteDB), but I would expect this list to attempt to communicate that authoritative information as accurately as practical.

If yes, it should be maintained and notably retired instances removed from that table. But whose responsibility is that?

I would expect cleanup of retired DSS instances would generally be the responsibility of the operator of those instances or, failing that, the responsibility of the other pool participants who want to use this information to monitor the pool status. Ideally, this table would be perfectly tied to the actual deployment and operational status of all instances, but I don't think there is a feasible way to accomplish that goal in the first iteration of the feature. Instead, MVP can rely on clients who care about the information to maintain the information in exceptional circumstances.

Also, it may be worth distinguishing heartbeats (DSS instance-initiated periodic alive signal) from some health check / self-tests (DSS instance or operator -initiated self testing of basic functions).

I agree we shouldn't let the heartbeat become a light (or medium) weight automated test through scope creep. I'm hoping to have the heartbeat limited to the purpose of establishing the current liveness/functionality of a DSS instance, including its ability to interact with the DAR. In terms of initiation of the heartbeat, I am less concerned with core-service initiating the heartbeat versus the rest of the USS as a client initiating the heartbeat because the DSS is merely a component of the USS and I don't think monitoring and alerting outside the core-service container is fundamentally different from monitoring and alerting inside the core-service container.

About removed instance, an automatic removal after 'x hours' without an HB would be an option without a specific responsibility but a shared one

But then when an instance is offline for long enough, while still part of the pool, the entry will get removed and that will make the pool appear healthy. Which would not be true.

Yes, this is the challenge with assuming the table of instances is "authoritative". It would be great to have it more guaranteed to be reflective of the actual deployment and functionality status of all instances, but I do not think this is achievable for MVP. Instead, it can provide a more narrow set of information (the last time DSS instance X confirmed it was working was Y) that should achieve the MVP goal (determine if the DSS pool is configured and working the way its operators expect it to be). Then, we can improve on that functionality in the future.