[Security Solution] Extract OpenAPI codegen to a package

[xcrzx]

Related to: https://github.com/elastic/security-team/issues/7583

Summary

Extract the OpenAPI codegen into a separate package @kbn/openapi-generator. The code generator script is now configurable.

README

This code generator could be used to generate runtime types, documentation, server stub implementations, clients, and much more given OpenAPI specification.

Getting started

To start with code generation you should have OpenAPI specification describing your API endpoint request and response schemas along with common types used in your API. The code generation script supports OpenAPI 3.1.0, refer to https://swagger.io/specification/ for more details.

OpenAPI specification should be in YAML format and have .schema.yaml extension. Here's a simple example of OpenAPI specification:

openapi: 3.0.0
info:
  title: Install Prebuilt Rules API endpoint
  version: 2023-10-31
paths:
  /api/detection_engine/rules/prepackaged:
    put:
      operationId: InstallPrebuiltRules
      x-codegen-enabled: true
      summary: Installs all Elastic prebuilt rules and timelines
      tags:
        - Prebuilt Rules API
      responses:
        200:
          description: Indicates a successful call
          content:
            application/json:
              schema:
                type: object
                properties:
                  rules_installed:
                    type: integer
                    description: The number of rules installed
                    minimum: 0
                  rules_updated:
                    type: integer
                    description: The number of rules updated
                    minimum: 0
                  timelines_installed:
                    type: integer
                    description: The number of timelines installed
                    minimum: 0
                  timelines_updated:
                    type: integer
                    description: The number of timelines updated
                    minimum: 0
                required:
                  - rules_installed
                  - rules_updated
                  - timelines_installed
                  - timelines_updated

Put it anywhere in your plugin, the code generation script will traverse the whole plugin directory and find all .schema.yaml files.

Then to generate code run the following command:

node scripts/generate_openapi --rootDir ./x-pack/plugins/security_solution

By default it uses the zod_operation_schema template which produces runtime types for request and response schemas described in OpenAPI specification. The generated code will be placed adjacent to the .schema.yaml file and will have .gen.ts extension.

Example of generated code:

import { z } from 'zod';

/*
 * NOTICE: Do not edit this file manually.
 * This file is automatically generated by the OpenAPI Generator, @kbn/openapi-generator.
 */

export type InstallPrebuiltRulesResponse = z.infer<typeof InstallPrebuiltRulesResponse>;
export const InstallPrebuiltRulesResponse = z.object({
  /**
   * The number of rules installed
   */
  rules_installed: z.number().int().min(0),
  /**
   * The number of rules updated
   */
  rules_updated: z.number().int().min(0),
  /**
   * The number of timelines installed
   */
  timelines_installed: z.number().int().min(0),
  /**
   * The number of timelines updated
   */
  timelines_updated: z.number().int().min(0),
});

Programmatic API

Alternatively, you can use the code generator programmatically. You can create a script file and run it with node command. This could be useful if you want to set up code generation in your CI pipeline. Here's an example of such script:

require('../../../../../src/setup_node_env');
const { generate } = require('@kbn/openapi-generator');
const { resolve } = require('path');

const SECURITY_SOLUTION_ROOT = resolve(__dirname, '../..');

generate({
  rootDir: SECURITY_SOLUTION_ROOT, // Path to the plugin root directory
  sourceGlob: './**/*.schema.yaml', // Glob pattern to find OpenAPI specification files
  templateName: 'zod_operation_schema', // Name of the template to use
});

CI integration

To make sure that generated code is always in sync with its OpenAPI specification it is recommended to add a command to your CI pipeline that will run code generation on every pull request and commit the changes if there are any.

First, create a script that will run code generation and commit the changes. See .buildkite/scripts/steps/code_generation/security_solution_codegen.sh for an example:

#!/usr/bin/env bash

set -euo pipefail

source .buildkite/scripts/common/util.sh

.buildkite/scripts/bootstrap.sh

echo --- Security Solution OpenAPI Code Generation

(cd x-pack/plugins/security_solution && yarn openapi:generate)
check_for_changed_files "yarn openapi:generate" true

This script sets up the minimal environment required fro code generation and runs the code generation script. Then it checks if there are any changes and commits them if there are any using the check_for_changed_files function.

Then add the code generation script to your plugin build pipeline. Open your plugin build pipeline, for example .buildkite/pipelines/pull_request/security_solution.yml, and add the following command to the steps list adjusting the path to your code generation script:

  - command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh
    label: 'Security Solution OpenAPI codegen'
    agents:
      queue: n2-2-spot
    timeout_in_minutes: 60
    parallelism: 1

Now on every pull request the code generation script will run and commit the changes if there are any.

OpenAPI Schema

The code generator supports the OpenAPI definitions described in the request, response, and component sections of the document.

For every API operation (GET, POST, etc) it is required to specify the operationId field. This field is used to generate the name of the generated types. For example, if the operationId is InstallPrebuiltRules then the generated types will be named InstallPrebuiltRulesResponse and InstallPrebuiltRulesRequest. If the operationId is not specified then the code generation will throw an error.

The x-codegen-enabled field is used to enable or disable code generation for the operation. If it is not specified then code generation is disabled by default. This field could be also used to disable code generation of common components described in the components section of the OpenAPI specification.

Keep in mind that disabling code generation for common components that are referenced by external OpenAPI specifications could lead to errors during code generation.

Schema files organization

It is recommended to limit the number of operations and components described in a single OpenAPI specification file. Having one HTTP operation in a single file will make it easier to maintain and will keep the generated artifacts granular for ease of reuse and better tree shaking. You can have as many OpenAPI specification files as you want.

Common components

It is common to have shared types that are used in multiple API operations. To avoid code duplication you can define common components in the components section of the OpenAPI specification and put them in a separate file. Then you can reference these components in the parameters and responses sections of the API operations.

Here's an example of the schema that references common components:

openapi: 3.0.0
info:
  title: Delete Rule API endpoint
  version: 2023-10-31
paths:
  /api/detection_engine/rules:
    delete:
      operationId: DeleteRule
      description: Deletes a single rule using the `rule_id` or `id` field.
      parameters:
        - name: id
          in: query
          required: false
          description: The rule's `id` value.
          schema:
            $ref: '../../../model/rule_schema/common_attributes.schema.yaml#/components/schemas/RuleSignatureId'
        - name: rule_id
          in: query
          required: false
          description: The rule's `rule_id` value.
          schema:
            $ref: '../../../model/rule_schema/common_attributes.schema.yaml#/components/schemas/RuleObjectId'
      responses:
        200:
          description: Indicates a successful call.
          content:
            application/json:
              schema:
                $ref: '../../../model/rule_schema/rule_schemas.schema.yaml#/components/schemas/RuleResponse'

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[spong]

Haven't had a chance to take a look at this yet @xcrzx, but hoping to give it a test run by integrating w/ the elastic_assistant plugin to see how generation goes. Will provide some feedback here in the next couple of days 👍

[spong]

Thank you for including the detailed description here in the README as well!

WRT wider adoption and use, should we surface as a documentation set on docs.elastic.dev, or have an entry in the Kibana Developer Guide?

[xcrzx]

Yes, that's a good idea 👍 I've also been thinking about making a broader announcement, like sending an email about the codegen features to attract more early adopters. But starting with documentation is definitely a solid first step.

[spong]

I didn't read close enough the first pass through and created an assistant.schema.yml and it wasn't getting picked up since it wasn't .yaml. After some searching it seems yaml is the defacto/preferred extension even though we have a lot of .yml files floating around the Kibana repo. nit here would be maybe also support .yml, or print an info/warning message around 🕵️‍♀️ Found 0 schemas, parsing, but this is entirely user error, so feel free to do nothing here 🙂

[xcrzx]

Adding .yml support should be pretty straightforward. I can tackle that in subsequent PRs, since work on the package is still ongoing.

[spong]

Not 100% on usage here, but there is the ToolingLog package for logger usage within Kibana tooling, e.g. from the resolver generator script (which also uses console for some output, so IDK...):

kibana/x-pack/plugins/security_solution/scripts/endpoint/resolver_generator_script.ts

Lines 328 to 332 in a449481

	const logger = new ToolingLog({
	level: 'info',
	writeTo: process.stdout,
	});
	const toolingLogOptions = { log: logger };

edit: Probably because of the logger prefix vs prettyprint of console would be my first guess here.

[xcrzx]

Oh, I didn't know that we have a package for logging. Thanks for bringing it up 👍 I'll look into replacing the console.logs with it.

[spong]

What change ended up causing these int()'s to be added here and in the endpoint files generated below? Not seeing a change to a schema.yaml, so looks like it was the {{~#*inline "type_integer"~}} change over in zod_schema_item.handlebars?

[xcrzx]

While moving templates around, I noticed that we were missing .int() for integers, so I decided to add it right away. I also made some other minor clean-ups in the templates but they didn't affect generated artifacts.

[spong]

Checked out, tested locally, and LGTM!

Thank you for all your efforts here in not only setting up our OpenAPI code generation, but taking the time to generalize it so others can start leveraging all this goodness as well! Left a few nits, but everything worked 👍 when testing locally, and code changes look good as well, so no changes necessary 🙂

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 816c51d

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/openapi-generator	-	7	+7

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	12.8MB	12.8MB	+168.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/openapi-generator	-	7	+7

ESLint disabled in files

id	before	after	diff
@kbn/openapi-generator	-	1	+1
securitySolution	67	66	-1
total			-0

ESLint disabled line counts

id	before	after	diff
@kbn/openapi-generator	-	2	+2
securitySolution	455	454	-1
total			+1

Total ESLint disabled count

id	before	after	diff
@kbn/openapi-generator	-	3	+3
securitySolution	522	520	-2
total			+1

History

• 💔 Build #161261 failed 3b90994
• 💔 Build #161243 failed 9aa31c6
• 💔 Build #160320 failed 1c208d1
• 💔 Build #158791 failed 299b894
• 💔 Build #158447 failed 1214307

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @xcrzx

[tomsonpl]

DW lgtm 👍 thanks