Healthcheck service #24817

vinzscam · 2024-05-17T09:42:57Z

Hey, I just made a Pull Request!

This PR adds a new healthcheck service which adds a bunch of endpoints to the root HTTP Router

✔️ Checklist

A changeset describing the change and affected packages. (more info)
Added or updated documentation
Tests for new functionality and regression tests for bug fixes
Screenshots attached (for UI changes)
All your commits have a Signed-off-by line in the message. (more info)

backstage-goalie · 2024-05-17T09:43:21Z

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/backend-app-api	packages/backend-app-api	patch	`v0.7.6-next.2`
@backstage/backend-defaults	packages/backend-defaults	patch	`v0.3.0-next.2`
@backstage/backend-plugin-api	packages/backend-plugin-api	patch	`v0.6.19-next.2`
@backstage/backend-test-utils	packages/backend-test-utils	patch	`v0.4.0-next.2`

Rugvip

Nice! 🎉

Couple of initial high level bits

Rugvip · 2024-05-17T09:53:08Z

packages/backend-app-api/src/services/implementations/httpRouter/createHealthcheck.ts

+  const router = Router();
+  let handler = () => Promise.resolve({ status: 'ok' });
+
+  router.get('/healthcheck', async (_request: Request, response: Response) => {


Thinking we should have this separated from the main routes, probably something like /.backstage/health, but potentially grouped a bit

Rugvip · 2024-05-17T09:55:56Z

packages/backend-plugin-api/src/services/definitions/HttpRouterService.ts

+export interface HttpRouterHealthCheckConfig {
+  handler: () => Promise<any>;
+}
+
 /**
 * @public
 */
 export interface HttpRouterService {


What do you think about keeping health checks at the root level instead? Something like /.backstage/health/v1/readiness for checking whether the entire instance is up, and /.backstage/health/v1/<plugin-id>/readiness for individual plugins?

but the other way around, /.backstage/health/v1/readiness/<plugin-id>

Rugvip · 2024-05-17T09:58:21Z

packages/backend-plugin-api/api-report.md

 // @public (undocumented)
 export interface HttpRouterService {
  // (undocumented)
  addAuthPolicy(policy: HttpRouterServiceAuthPolicy): void;
  // (undocumented)
+  healthCheckConfig(healthCheckOptions: HttpRouterHealthCheckConfig): void;


Not sure this should live on the http router tbh. Or framed a bit differently: if we can make this a separate service we probably should. I'm thinking this should probably be additive as well so that it's possible to add health checks from modules etc.

Rugvip · 2024-05-17T09:59:56Z

packages/backend-plugin-api/api-report.md

+// @public (undocumented)
+export interface HttpRouterHealthCheckConfig {
+  // (undocumented)
+  handler: () => Promise<any>;


I think it's important to consider a liveness + readiness split. It can help avoid situations where instances are shut down because they timed out at startup, while still being able to be pretty aggressive about timeouts when it comes to whether the service is functional at all.

do you have any suggestion for a possible root level readiness implementation?

camilaibs

Thank you 🎉, left minor comments. Could we also deprecate the legacy handler from the backend-common package as well?

packages/backend-app-api/src/services/implementations/httpRouter/createHealthcheck.ts

packages/backend-app-api/src/services/implementations/httpRouter/httpRouterServiceFactory.ts

packages/backend-plugin-api/api-report.md

packages/backend-plugin-api/src/services/definitions/HttpRouterService.ts

camilaibs

Looks great, left minor comments 🙌🏻

camilaibs · 2024-05-29T13:17:24Z

packages/backend-defaults/src/CreateBackend.ts

@@ -58,6 +59,7 @@ export const defaultServiceFactories = [
  userInfoServiceFactory(),
  urlReaderServiceFactory(),
  eventsServiceFactory(),
+  healthServiceFactory(),


I believe we have also to provide a mock implementation here

camilaibs · 2024-05-29T13:18:45Z

packages/backend-app-api/src/services/implementations/rootHttpRouter/createHealthcheck.ts

I guess this is not used anymore, right?

camilaibs · 2024-05-29T13:19:19Z

packages/backend-app-api/src/services/implementations/rootHttpRouter/createHealthcheck.test.ts

I guess this file can be deleted, right?

camilaibs · 2024-05-29T13:21:56Z

.changeset/bright-panthers-leave.md

+'@backstage/backend-app-api': patch
+---
+
+Added a new health service which adds new endpoints for health checks.


It would be nice add this service to the core services docs:
https://backstage.io/docs/backend-system/core-services/index

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

Rugvip

Nice! 👍

Couple of nits

Rugvip · 2024-06-03T13:06:17Z

packages/backend-plugin-api/src/services/definitions/coreServices.ts

+   * The service reference for the plugin scoped {@link HealthService}.
+   */
+  export const health = createServiceRef<
+    import('./HealthService').HealthService


I think it's prolly best to keep this as void instead, since now you can pass anything instead. We can still add methods in the future even if it starts out as void.

Rugvip · 2024-06-03T13:14:52Z

packages/backend-app-api/src/services/implementations/health/createHealthRouter.ts

+  router.get('/v1/readiness', async (_request: Request, response: Response) => {
+    if (!isRunning) {
+      throw new Error('Backend has not started yet');
+    }
+    response.json({ status: 'ok' });
+  });
+
+  router.get('/v1/liveness', async (_request: Request, response: Response) => {
+    response.json({ status: 'ok' });
+  });


Could you add a quick description to each of these with what their purpose is?

Rugvip · 2024-06-03T13:15:10Z

packages/backend-app-api/src/services/implementations/health/createHealthRouter.ts

+
+  router.get('/v1/readiness', async (_request: Request, response: Response) => {
+    if (!isRunning) {
+      throw new Error('Backend has not started yet');


Thinking this should be a 503 ServiceUnavailableError. Perhaps we shouldn't throw an error in this case either, so that we can have a more consistent response? Instead setting an explicit status code and returning whatever the opposite of status: 'ok' is

freben · 2024-06-05T12:36:07Z

packages/backend-plugin-api/src/services/definitions/coreServices.ts

+   * The service reference for the plugin scoped {@link HealthService}.
+   */
+  export const health = createServiceRef<
+    import('./HealthService').HealthService
+  >({ id: 'core.health', scope: 'root' });


Suggested change

* The service reference for the plugin scoped {@link HealthService}.

*/

export const health = createServiceRef<

import('./HealthService').HealthService

>({ id: 'core.health', scope: 'root' });

* The service reference for the root scoped {@link RootHealthService}.

*/

export const rootHealth = createServiceRef<

import('./RootHealthService').RootHealthService

>({ id: 'core.rootHealth', scope: 'root' });

Every single other root scoped thing is named like this - we should probably follow suit here. Make sure to change to that naming pattern everywhere else too.

I'd also be interested in what the future might be in terms of per-plugin liveness/health. That's sort of where the most interesting developments might be in the future. This root scoped one could at most do simple lifecycle matching like it does today, or possibly offer some form of rollup of the states of all installed plugins (not even sure that's a good idea though). I mean, how often do we really desire to look at the liveness of effectively a machine that hosts many plugins? What does that even mean?

vinzscam requested review from a team as code owners May 17, 2024 09:42

vinzscam requested review from Rugvip and camilaibs May 17, 2024 09:42

Rugvip requested changes May 17, 2024

View reviewed changes

camilaibs requested changes May 17, 2024

View reviewed changes

vinzscam changed the title ~~Healthcheck endpoint~~ Healthcheck service May 17, 2024

vinzscam force-pushed the healthcheck-router branch 3 times, most recently from 3ede1f0 to 74a6836 Compare May 24, 2024 11:54

camilaibs approved these changes May 29, 2024

View reviewed changes

github-actions bot added the documentation Improvements or additions to documentation label Jun 3, 2024

vinzscam added 2 commits June 3, 2024 14:04

backend-app-api: httpRouter add healthCheckConfig

8dfdaee

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-app-api: healthcheck middleware

91dc14d

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

vinzscam force-pushed the healthcheck-router branch from cf48e42 to 8c5ca35 Compare June 3, 2024 12:11

github-actions bot added the area:catalog Related to the Catalog Project Area label Jun 3, 2024

vinzscam added 11 commits June 3, 2024 14:14

backend-plugin-api: export healthCheckConfig fn

3a6da81

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

healthcheck changeset

ed4da53

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-app-api: move healthchecks to healthService

92e4538

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-defaults: add healthservice

96c1364

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

tweak changeset

351fb58

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-app-api: remove test

968c2de

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-app-api: mark healthServiceFactory as public

204bc94

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-test-utils: mock health service

779616b

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-app-api: clean up

566ba2f

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

docs: add health service

5ec007f

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

health service api report

926d6c9

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

vinzscam force-pushed the healthcheck-router branch from 8c5ca35 to 926d6c9 Compare June 3, 2024 12:14

github-actions bot removed the area:catalog Related to the Catalog Project Area label Jun 3, 2024

backend-test-utils: health service mock changeset

cdc62c9

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

Rugvip requested changes Jun 3, 2024

View reviewed changes

vinzscam requested a review from backstage-service as a code owner June 5, 2024 11:28

vinzscam force-pushed the healthcheck-router branch from 4fd88dc to cdc62c9 Compare June 5, 2024 11:29

freben reviewed Jun 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Healthcheck service #24817

Healthcheck service #24817

vinzscam commented May 17, 2024 •

edited

backstage-goalie bot commented May 17, 2024 •

edited

Rugvip left a comment

Rugvip May 17, 2024

Rugvip May 17, 2024

vinzscam May 17, 2024

freben May 17, 2024

Rugvip May 17, 2024

Rugvip May 17, 2024

vinzscam May 17, 2024

camilaibs left a comment

camilaibs left a comment •

edited

camilaibs May 29, 2024

camilaibs May 29, 2024

camilaibs May 29, 2024

camilaibs May 29, 2024

Rugvip left a comment

Rugvip Jun 3, 2024

Rugvip Jun 3, 2024

Rugvip Jun 3, 2024

freben Jun 5, 2024

Healthcheck service #24817

Are you sure you want to change the base?

Healthcheck service #24817

Conversation

vinzscam commented May 17, 2024 • edited

Hey, I just made a Pull Request!

✔️ Checklist

backstage-goalie bot commented May 17, 2024 • edited

Changed Packages

Rugvip left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camilaibs left a comment

Choose a reason for hiding this comment

camilaibs left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rugvip left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinzscam commented May 17, 2024 •

edited

backstage-goalie bot commented May 17, 2024 •

edited

camilaibs left a comment •

edited