Healthcheck service #24817

vinzscam · 2024-05-17T09:42:57Z

Hey, I just made a Pull Request!

This PR adds a new healthcheck service which adds a bunch of endpoints to the root HTTP Router

✔️ Checklist

A changeset describing the change and affected packages. (more info)
Added or updated documentation
Tests for new functionality and regression tests for bug fixes
Screenshots attached (for UI changes)
All your commits have a Signed-off-by line in the message. (more info)

backstage-goalie · 2024-05-17T09:43:21Z

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/backend-defaults	packages/backend-defaults	patch	`v0.3.0`
@backstage/backend-plugin-api	packages/backend-plugin-api	patch	`v0.6.19`
@backstage/backend-test-utils	packages/backend-test-utils	patch	`v0.4.0`

Rugvip

Nice! 🎉

Couple of initial high level bits

Rugvip · 2024-05-17T09:53:08Z

packages/backend-app-api/src/services/implementations/httpRouter/createHealthcheck.ts

+  const router = Router();
+  let handler = () => Promise.resolve({ status: 'ok' });
+
+  router.get('/healthcheck', async (_request: Request, response: Response) => {


Thinking we should have this separated from the main routes, probably something like /.backstage/health, but potentially grouped a bit

Rugvip · 2024-05-17T09:55:56Z

packages/backend-plugin-api/src/services/definitions/HttpRouterService.ts

+export interface HttpRouterHealthCheckConfig {
+  handler: () => Promise<any>;
+}
+
 /**
 * @public
 */
 export interface HttpRouterService {


What do you think about keeping health checks at the root level instead? Something like /.backstage/health/v1/readiness for checking whether the entire instance is up, and /.backstage/health/v1/<plugin-id>/readiness for individual plugins?

but the other way around, /.backstage/health/v1/readiness/<plugin-id>

Rugvip · 2024-05-17T09:58:21Z

packages/backend-plugin-api/api-report.md

 // @public (undocumented)
 export interface HttpRouterService {
  // (undocumented)
  addAuthPolicy(policy: HttpRouterServiceAuthPolicy): void;
  // (undocumented)
+  healthCheckConfig(healthCheckOptions: HttpRouterHealthCheckConfig): void;


Not sure this should live on the http router tbh. Or framed a bit differently: if we can make this a separate service we probably should. I'm thinking this should probably be additive as well so that it's possible to add health checks from modules etc.

Rugvip · 2024-05-17T09:59:56Z

packages/backend-plugin-api/api-report.md

+// @public (undocumented)
+export interface HttpRouterHealthCheckConfig {
+  // (undocumented)
+  handler: () => Promise<any>;


I think it's important to consider a liveness + readiness split. It can help avoid situations where instances are shut down because they timed out at startup, while still being able to be pretty aggressive about timeouts when it comes to whether the service is functional at all.

do you have any suggestion for a possible root level readiness implementation?

camilaibs

Thank you 🎉, left minor comments. Could we also deprecate the legacy handler from the backend-common package as well?

packages/backend-app-api/src/services/implementations/httpRouter/createHealthcheck.ts

packages/backend-app-api/src/services/implementations/httpRouter/httpRouterServiceFactory.ts

packages/backend-plugin-api/api-report.md

packages/backend-plugin-api/src/services/definitions/HttpRouterService.ts

camilaibs

Looks great, left minor comments 🙌🏻

camilaibs · 2024-05-29T13:17:24Z

packages/backend-defaults/src/CreateBackend.ts

@@ -58,6 +59,7 @@ export const defaultServiceFactories = [
  userInfoServiceFactory(),
  urlReaderServiceFactory(),
  eventsServiceFactory(),
+  healthServiceFactory(),


I believe we have also to provide a mock implementation here

camilaibs · 2024-05-29T13:18:45Z

packages/backend-app-api/src/services/implementations/rootHttpRouter/createHealthcheck.ts

I guess this is not used anymore, right?

camilaibs · 2024-05-29T13:19:19Z

packages/backend-app-api/src/services/implementations/rootHttpRouter/createHealthcheck.test.ts

I guess this file can be deleted, right?

camilaibs · 2024-05-29T13:21:56Z

.changeset/bright-panthers-leave.md

+'@backstage/backend-app-api': patch
+---
+
+Added a new health service which adds new endpoints for health checks.


It would be nice add this service to the core services docs:
https://backstage.io/docs/backend-system/core-services/index

Rugvip

Nice! 👍

Couple of nits

Rugvip · 2024-06-03T13:06:17Z

packages/backend-plugin-api/src/services/definitions/coreServices.ts

+   * The service reference for the plugin scoped {@link HealthService}.
+   */
+  export const health = createServiceRef<
+    import('./HealthService').HealthService


I think it's prolly best to keep this as void instead, since now you can pass anything instead. We can still add methods in the future even if it starts out as void.

Rugvip · 2024-06-03T13:14:52Z

packages/backend-app-api/src/services/implementations/health/createHealthRouter.ts

+  router.get('/v1/readiness', async (_request: Request, response: Response) => {
+    if (!isRunning) {
+      throw new Error('Backend has not started yet');
+    }
+    response.json({ status: 'ok' });
+  });
+
+  router.get('/v1/liveness', async (_request: Request, response: Response) => {
+    response.json({ status: 'ok' });
+  });


Could you add a quick description to each of these with what their purpose is?

Rugvip · 2024-06-03T13:15:10Z

packages/backend-app-api/src/services/implementations/health/createHealthRouter.ts

+
+  router.get('/v1/readiness', async (_request: Request, response: Response) => {
+    if (!isRunning) {
+      throw new Error('Backend has not started yet');


Thinking this should be a 503 ServiceUnavailableError. Perhaps we shouldn't throw an error in this case either, so that we can have a more consistent response? Instead setting an explicit status code and returning whatever the opposite of status: 'ok' is

freben · 2024-06-05T12:36:07Z

packages/backend-plugin-api/src/services/definitions/coreServices.ts

+   * The service reference for the plugin scoped {@link HealthService}.
+   */
+  export const health = createServiceRef<
+    import('./HealthService').HealthService
+  >({ id: 'core.health', scope: 'root' });


Suggested change

* The service reference for the plugin scoped {@link HealthService}.

*/

export const health = createServiceRef<

import('./HealthService').HealthService

>({ id: 'core.health', scope: 'root' });

* The service reference for the root scoped {@link RootHealthService}.

*/

export const rootHealth = createServiceRef<

import('./RootHealthService').RootHealthService

>({ id: 'core.rootHealth', scope: 'root' });

Every single other root scoped thing is named like this - we should probably follow suit here. Make sure to change to that naming pattern everywhere else too.

I'd also be interested in what the future might be in terms of per-plugin liveness/health. That's sort of where the most interesting developments might be in the future. This root scoped one could at most do simple lifecycle matching like it does today, or possibly offer some form of rollup of the states of all installed plugins (not even sure that's a good idea though). I mean, how often do we really desire to look at the liveness of effectively a machine that hosts many plugins? What does that even mean?

camilaibs · 2024-06-18T11:39:16Z

.changeset/bright-panthers-leave.md

I beleive should deprecate these functions and types as well in backend common:

backstage/packages/backend-common/api-report.md

Lines 188 to 192 in 6fa100a

export function createStatusCheckRouter(options: {

logger: LoggerService;

path?: string;

statusCheck?: StatusCheck;

}): Promise<express.Router>;

backstage/packages/backend-common/api-report.md

Line 641 in 6fa100a

export type StatusCheck = () => Promise<any>;

backstage/packages/backend-common/api-report.md

Lines 644 to 646 in 6fa100a

export function statusCheckHandler(

options?: StatusCheckHandlerOptions,

): Promise<RequestHandler>;

backstage/packages/backend-common/api-report.md

Lines 649 to 651 in 6fa100a

export interface StatusCheckHandlerOptions {

statusCheck?: StatusCheck;

}

good catch, I'm gonna do it on a following PR

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

.changeset/bright-panthers-leave.md

.changeset/serious-kings-trade.md

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

.changeset/bright-panthers-leave.md

docs/backend-system/core-services/root-health.md

freben · 2024-06-20T11:18:10Z

docs/backend-system/core-services/root-health.md

+const backend = createBackend();
+
+class MyRootHealthService implements RootHealthService {
+  async getLiveness(): Promise<{ status: number; payload?: any }> {


the payloads could have been JsonValue

packages/backend-defaults/src/entrypoints/rootHealth/rootHealthServiceFactory.test.ts

freben · 2024-06-20T11:21:17Z

packages/backend-defaults/src/entrypoints/rootHealth/rootHealthServiceFactory.test.ts

+    });
+
+    it(`should return a 500 response if the server has stopped`, async () => {
+      const lifecycle = mockServices.rootLifecycle.mock();


you can also do

const lifecycle = mockServices.rootLifecycle.mock({ addStartupHook: jest.fn(() => { ... },

freben · 2024-06-20T11:29:18Z

packages/backend-defaults/src/entrypoints/rootHttpRouter/rootHttpRouterServiceFactory.ts

      const { indexPath, configure = defaultConfigure } = options ?? {};
      const logger = rootLogger.child({ service: 'rootHttpRouter' });
      const app = express();

      const router = DefaultRootHttpRouter.create({ indexPath });
      const middleware = MiddlewareFactory.create({ config, logger });
      const routes = router.handler();
+
+      const healthRouter = createHealthRouter({ health });


I don't think that this file should have any changes at all.

The rootHealthServiceFactory should have a dependency on coreServices.rootHttpRouter and just call .use on it with the right path.

Your DefaultRootHealthService could have a create method that accepts the root http router service as an argument and does that work. You could also make your rootHealthServiceFactory be of the options callback form, where you can pass in custom liveness and readiness callbacks as an adopter, and then the DefaultRootHealthService uses those instead of the default ones if they were given.

I believe this approach simplifies matters in the event that the adopter wishes to provide their own implementation of the health service. With your approach, they would essentially have to replicate what the default implementation of the healthServiceFactory is doing, remembering the path where the middleware should be added. This could potentially lead to problems if they attach a middleware to a non-standard path.

This alternative approach simplifies things: they only need to provide their own implementation of the getLiveness and getReadiness methods. No raw operations on the root router are necessary.

What I meant was,

backend.add( rootHealthServiceFactory({ getLiveness: async () => ..., }) )

as the first level of adaptation, and if you need more,

backend.add( createServiceFactory({ service: coreServices.rootHealth, deps: { rootHttpRouter: coreServices.rootHttpRouter, }, async factory({ rootHttpRouter }) { // performs all registration etc inside create return DefaultRootHealthService.create({ rootHttpRouter, getLiveness: async () => ..., }); } }) )

of course, this is all moot once we have service modules :)

But yeah I see that we did the same type of thing for the auth and lifecycle middlewares in httpRouterServiceFactory. So maybe this is where we were going with all this anyway.

I think this approach simplifies the customization process for adopters and provides a more opinionated method for exposing the metrics

freben · 2024-06-20T11:29:50Z

packages/backend-plugin-api/src/services/definitions/RootHealthService.ts

+  /**
+   * Get the liveness status of the backend.
+   */
+  getLiveness(): Promise<{ status: number; payload?: any }>;


yeah again, making these payloads JsonObject conveys the limitation that they are being serialized as a response body.

Co-authored-by: Fredrik Adelöw <freben@gmail.com> Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

github-actions · 2024-06-24T11:40:26Z

Thank you for contributing to Backstage! The changes in this pull request will be part of the 1.29.0 release, scheduled for Tue, 16 Jul 2024.

vinzscam requested review from a team as code owners May 17, 2024 09:42

vinzscam requested review from Rugvip and camilaibs May 17, 2024 09:42

Rugvip requested changes May 17, 2024

View reviewed changes

camilaibs requested changes May 17, 2024

View reviewed changes

vinzscam changed the title ~~Healthcheck endpoint~~ Healthcheck service May 17, 2024

vinzscam force-pushed the healthcheck-router branch 3 times, most recently from 3ede1f0 to 74a6836 Compare May 24, 2024 11:54

camilaibs approved these changes May 29, 2024

View reviewed changes

github-actions bot added the area:documentation Improvements or additions to documentation label Jun 3, 2024

vinzscam force-pushed the healthcheck-router branch from cf48e42 to 8c5ca35 Compare June 3, 2024 12:11

github-actions bot added the area:catalog Related to the Catalog Project Area label Jun 3, 2024

vinzscam force-pushed the healthcheck-router branch from 8c5ca35 to 926d6c9 Compare June 3, 2024 12:14

github-actions bot removed the area:catalog Related to the Catalog Project Area label Jun 3, 2024

Rugvip requested changes Jun 3, 2024

View reviewed changes

vinzscam requested a review from backstage-service as a code owner June 5, 2024 11:28

vinzscam force-pushed the healthcheck-router branch from 4fd88dc to cdc62c9 Compare June 5, 2024 11:29

freben reviewed Jun 5, 2024

View reviewed changes

Rugvip added the workflow:after-release This is a bit too scary to merge until after the next release label Jun 14, 2024

camilaibs reviewed Jun 18, 2024

View reviewed changes

vinzscam added 7 commits June 20, 2024 10:28

backend-app-api: httpRouter add healthCheckConfig

8dc15aa

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-app-api: healthcheck middleware

8bbbc31

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-plugin-api: export healthCheckConfig fn

14e982e

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

healthcheck changeset

53ced70

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-app-api: move healthchecks to healthService

8d47357

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-defaults: add healthservice

b26877f

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

tweak changeset

1ce0bb7

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

vinzscam added 12 commits June 20, 2024 10:32

backend-app-api: remove test

0717aae

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-app-api: mark healthServiceFactory as public

49b4890

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-test-utils: mock health service

a38ffb2

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-app-api: clean up

972896d

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

docs: add health service

652da2e

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

health service api report

78fdc5a

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-test-utils: health service mock changeset

fce7887

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-defaults: move over health service

918b397

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-app-api: flip logic of health service

44f97c1

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-defaults: fix notice

506fc23

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

Rename HealthService to RootHealthService

0c62151

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

health api reports

5c4e876

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

vinzscam force-pushed the healthcheck-router branch from cb2689f to 2c97234 Compare June 20, 2024 10:11

vinzscam added 2 commits June 20, 2024 12:20

api reports

73a4565

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

docs: update root health docs

e36e507

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

vinzscam force-pushed the healthcheck-router branch from 2c97234 to e36e507 Compare June 20, 2024 10:20

vinzscam commented Jun 20, 2024

View reviewed changes

.changeset/bright-panthers-leave.md Outdated Show resolved Hide resolved

vinzscam commented Jun 20, 2024

View reviewed changes

.changeset/serious-kings-trade.md Outdated Show resolved Hide resolved

Apply suggestions from code review

4e8455f

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

freben reviewed Jun 20, 2024

View reviewed changes

vinzscam and others added 4 commits June 20, 2024 15:22

Update .changeset/bright-panthers-leave.md

73b4e72

Co-authored-by: Fredrik Adelöw <freben@gmail.com> Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

Apply suggestions from code review

cd22b40

Co-authored-by: Fredrik Adelöw <freben@gmail.com> Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-defaults: test tweaks

7df5c88

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

backend-plugin-api: fix typings in health service

878f2ef

Signed-off-by: Vincenzo Scamporlino <vincenzos@spotify.com>

freben approved these changes Jun 24, 2024

View reviewed changes

benjdlambert approved these changes Jun 24, 2024

View reviewed changes

freben merged commit d776d36 into master Jun 24, 2024
28 checks passed

freben deleted the healthcheck-router branch June 24, 2024 11:39

camilaibs mentioned this pull request Jun 24, 2024

[NBS 1.0] Deprecate status check handler #25372

Merged

5 tasks

	export function createStatusCheckRouter(options: {
	logger: LoggerService;
	path?: string;
	statusCheck?: StatusCheck;
	}): Promise<express.Router>;

	export function statusCheckHandler(
	options?: StatusCheckHandlerOptions,
	): Promise<RequestHandler>;

	export interface StatusCheckHandlerOptions {
	statusCheck?: StatusCheck;
	}

Healthcheck service #24817

Healthcheck service #24817

Uh oh!

Conversation

vinzscam commented May 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Hey, I just made a Pull Request!

✔️ Checklist

Uh oh!

backstage-goalie bot commented May 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changed Packages

Uh oh!

Rugvip left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

camilaibs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

camilaibs left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rugvip left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinzscam commented May 17, 2024 •

edited

Loading

backstage-goalie bot commented May 17, 2024 •

edited

Loading

camilaibs left a comment •

edited

Loading

vinzscam Jun 20, 2024 •

edited

Loading