-
Notifications
You must be signed in to change notification settings - Fork 8.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution] Extract OpenAPI codegen to a package #166269
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,201 @@ | ||
# OpenAPI Code Generator for Kibana | ||
|
||
This code generator could be used to generate runtime types, documentation, server stub implementations, clients, and much more given OpenAPI specification. | ||
|
||
## Getting started | ||
|
||
To start with code generation you should have OpenAPI specification describing your API endpoint request and response schemas along with common types used in your API. The code generation script supports OpenAPI 3.1.0, refer to https://swagger.io/specification/ for more details. | ||
|
||
OpenAPI specification should be in YAML format and have `.schema.yaml` extension. Here's a simple example of OpenAPI specification: | ||
|
||
```yaml | ||
openapi: 3.0.0 | ||
info: | ||
title: Install Prebuilt Rules API endpoint | ||
version: 2023-10-31 | ||
paths: | ||
/api/detection_engine/rules/prepackaged: | ||
put: | ||
operationId: InstallPrebuiltRules | ||
x-codegen-enabled: true | ||
summary: Installs all Elastic prebuilt rules and timelines | ||
tags: | ||
- Prebuilt Rules API | ||
responses: | ||
200: | ||
description: Indicates a successful call | ||
content: | ||
application/json: | ||
schema: | ||
type: object | ||
properties: | ||
rules_installed: | ||
type: integer | ||
description: The number of rules installed | ||
minimum: 0 | ||
rules_updated: | ||
type: integer | ||
description: The number of rules updated | ||
minimum: 0 | ||
timelines_installed: | ||
type: integer | ||
description: The number of timelines installed | ||
minimum: 0 | ||
timelines_updated: | ||
type: integer | ||
description: The number of timelines updated | ||
minimum: 0 | ||
required: | ||
- rules_installed | ||
- rules_updated | ||
- timelines_installed | ||
- timelines_updated | ||
``` | ||
|
||
Put it anywhere in your plugin, the code generation script will traverse the whole plugin directory and find all `.schema.yaml` files. | ||
|
||
Then to generate code run the following command: | ||
|
||
```bash | ||
node scripts/generate_openapi --rootDir ./x-pack/plugins/security_solution | ||
``` | ||
|
||
![Generator command output](image.png) | ||
|
||
By default it uses the `zod_operation_schema` template which produces runtime types for request and response schemas described in OpenAPI specification. The generated code will be placed adjacent to the `.schema.yaml` file and will have `.gen.ts` extension. | ||
|
||
Example of generated code: | ||
|
||
```ts | ||
import { z } from 'zod'; | ||
|
||
/* | ||
* NOTICE: Do not edit this file manually. | ||
* This file is automatically generated by the OpenAPI Generator, @kbn/openapi-generator. | ||
*/ | ||
|
||
export type InstallPrebuiltRulesResponse = z.infer<typeof InstallPrebuiltRulesResponse>; | ||
export const InstallPrebuiltRulesResponse = z.object({ | ||
/** | ||
* The number of rules installed | ||
*/ | ||
rules_installed: z.number().int().min(0), | ||
/** | ||
* The number of rules updated | ||
*/ | ||
rules_updated: z.number().int().min(0), | ||
/** | ||
* The number of timelines installed | ||
*/ | ||
timelines_installed: z.number().int().min(0), | ||
/** | ||
* The number of timelines updated | ||
*/ | ||
timelines_updated: z.number().int().min(0), | ||
}); | ||
``` | ||
## Programmatic API | ||
|
||
Alternatively, you can use the code generator programmatically. You can create a script file and run it with `node` command. This could be useful if you want to set up code generation in your CI pipeline. Here's an example of such script: | ||
|
||
```ts | ||
require('../../../../../src/setup_node_env'); | ||
const { generate } = require('@kbn/openapi-generator'); | ||
const { resolve } = require('path'); | ||
|
||
const SECURITY_SOLUTION_ROOT = resolve(__dirname, '../..'); | ||
|
||
generate({ | ||
rootDir: SECURITY_SOLUTION_ROOT, // Path to the plugin root directory | ||
sourceGlob: './**/*.schema.yaml', // Glob pattern to find OpenAPI specification files | ||
templateName: 'zod_operation_schema', // Name of the template to use | ||
}); | ||
``` | ||
|
||
## CI integration | ||
|
||
To make sure that generated code is always in sync with its OpenAPI specification it is recommended to add a command to your CI pipeline that will run code generation on every pull request and commit the changes if there are any. | ||
|
||
First, create a script that will run code generation and commit the changes. See `.buildkite/scripts/steps/code_generation/security_solution_codegen.sh` for an example: | ||
|
||
```bash | ||
#!/usr/bin/env bash | ||
|
||
set -euo pipefail | ||
|
||
source .buildkite/scripts/common/util.sh | ||
|
||
.buildkite/scripts/bootstrap.sh | ||
|
||
echo --- Security Solution OpenAPI Code Generation | ||
|
||
(cd x-pack/plugins/security_solution && yarn openapi:generate) | ||
check_for_changed_files "yarn openapi:generate" true | ||
``` | ||
|
||
This scripts sets up the minimal environment required fro code generation and runs the code generation script. Then it checks if there are any changes and commits them if there are any using the `check_for_changed_files` function. | ||
|
||
Then add the code generation script to your plugin build pipeline. Open your plugin build pipeline, for example `.buildkite/pipelines/pull_request/security_solution.yml`, and add the following command to the steps list adjusting the path to your code generation script: | ||
|
||
```yaml | ||
- command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh | ||
label: 'Security Solution OpenAPI codegen' | ||
agents: | ||
queue: n2-2-spot | ||
timeout_in_minutes: 60 | ||
parallelism: 1 | ||
``` | ||
|
||
Now on every pull request the code generation script will run and commit the changes if there are any. | ||
|
||
## OpenAPI Schema | ||
|
||
The code generator supports the OpenAPI definitions described in the request, response, and component sections of the document. | ||
|
||
For every API operation (GET, POST, etc) it is required to specify the `operationId` field. This field is used to generate the name of the generated types. For example, if the `operationId` is `InstallPrebuiltRules` then the generated types will be named `InstallPrebuiltRulesResponse` and `InstallPrebuiltRulesRequest`. If the `operationId` is not specified then the code generation will throw an error. | ||
|
||
The `x-codegen-enabled` field is used to enable or disable code generation for the operation. If it is not specified then code generation is disabled by default. This field could be also used to disable code generation of common components described in the `components` section of the OpenAPI specification. | ||
|
||
Keep in mind that disabling code generation for common components that are referenced by external OpenAPI specifications could lead to errors during code generation. | ||
|
||
### Schema files organization | ||
|
||
It is recommended to limit the number of operations and components described in a single OpenAPI specification file. Having one HTTP operation in a single file will make it easier to maintain and will keep the generated artifacts granular for ease of reuse and better tree shaking. You can have as many OpenAPI specification files as you want. | ||
|
||
### Common components | ||
|
||
It is common to have shared types that are used in multiple API operations. To avoid code duplication you can define common components in the `components` section of the OpenAPI specification and put them in a separate file. Then you can reference these components in the `parameters` and `responses` sections of the API operations. | ||
|
||
Here's an example of the schema that references common components: | ||
|
||
```yaml | ||
openapi: 3.0.0 | ||
info: | ||
title: Delete Rule API endpoint | ||
version: 2023-10-31 | ||
paths: | ||
/api/detection_engine/rules: | ||
delete: | ||
operationId: DeleteRule | ||
description: Deletes a single rule using the `rule_id` or `id` field. | ||
parameters: | ||
- name: id | ||
in: query | ||
required: false | ||
description: The rule's `id` value. | ||
schema: | ||
$ref: '../../../model/rule_schema/common_attributes.schema.yaml#/components/schemas/RuleSignatureId' | ||
- name: rule_id | ||
in: query | ||
required: false | ||
description: The rule's `rule_id` value. | ||
schema: | ||
$ref: '../../../model/rule_schema/common_attributes.schema.yaml#/components/schemas/RuleObjectId' | ||
responses: | ||
200: | ||
description: Indicates a successful call. | ||
content: | ||
application/json: | ||
schema: | ||
$ref: '../../../model/rule_schema/rule_schemas.schema.yaml#/components/schemas/RuleResponse' | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License | ||
* 2.0 and the Server Side Public License, v 1; you may not use this file except | ||
* in compliance with, at your election, the Elastic License 2.0 or the Server | ||
* Side Public License, v 1. | ||
*/ | ||
|
||
export * from './src/openapi_generator'; | ||
export * from './src/cli'; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License | ||
* 2.0 and the Server Side Public License, v 1; you may not use this file except | ||
* in compliance with, at your election, the Elastic License 2.0 or the Server | ||
* Side Public License, v 1. | ||
*/ | ||
|
||
module.exports = { | ||
preset: '@kbn/test', | ||
rootDir: '../..', | ||
roots: ['<rootDir>/packages/kbn-openapi-generator'], | ||
}; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
{ | ||
"devOnly": true, | ||
"id": "@kbn/openapi-generator", | ||
"owner": "@elastic/security-detection-engine", | ||
"type": "shared-common" | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
{ | ||
"bin": { | ||
"openapi-generator": "./bin/openapi-generator.js" | ||
}, | ||
"description": "OpenAPI code generator for Kibana", | ||
"license": "SSPL-1.0 OR Elastic License 2.0", | ||
"name": "@kbn/openapi-generator", | ||
"private": true, | ||
"version": "1.0.0" | ||
} |
Original file line number | Diff line number | Diff line change | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,44 @@ | ||||||||||||
/* | ||||||||||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||||||||||||
* or more contributor license agreements. Licensed under the Elastic License | ||||||||||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except | ||||||||||||
* in compliance with, at your election, the Elastic License 2.0 or the Server | ||||||||||||
* Side Public License, v 1. | ||||||||||||
*/ | ||||||||||||
import yargs from 'yargs/yargs'; | ||||||||||||
import { generate } from './openapi_generator'; | ||||||||||||
import { AVAILABLE_TEMPLATES } from './template_service/template_service'; | ||||||||||||
|
||||||||||||
export function runCli() { | ||||||||||||
yargs(process.argv.slice(2)) | ||||||||||||
.command( | ||||||||||||
'*', | ||||||||||||
'Generate code artifacts from OpenAPI specifications', | ||||||||||||
(y) => | ||||||||||||
y | ||||||||||||
.option('rootDir', { | ||||||||||||
describe: 'Root directory to search for OpenAPI specs', | ||||||||||||
demandOption: true, | ||||||||||||
string: true, | ||||||||||||
}) | ||||||||||||
.option('sourceGlob', { | ||||||||||||
describe: 'Elasticsearch target', | ||||||||||||
default: './**/*.schema.yaml', | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't read close enough the first pass through and created an There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Adding |
||||||||||||
string: true, | ||||||||||||
}) | ||||||||||||
.option('templateName', { | ||||||||||||
describe: 'Template to use for code generation', | ||||||||||||
default: 'zod_operation_schema' as const, | ||||||||||||
choices: AVAILABLE_TEMPLATES, | ||||||||||||
}) | ||||||||||||
.showHelpOnFail(false), | ||||||||||||
(argv) => { | ||||||||||||
generate(argv).catch((err) => { | ||||||||||||
// eslint-disable-next-line no-console | ||||||||||||
console.error(err); | ||||||||||||
Comment on lines
+37
to
+38
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not 100% on usage here, but there is the kibana/x-pack/plugins/security_solution/scripts/endpoint/resolver_generator_script.ts Lines 328 to 332 in a449481
edit: Probably because of the logger prefix vs prettyprint of console would be my first guess here. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, I didn't know that we have a package for logging. Thanks for bringing it up 👍 I'll look into replacing the |
||||||||||||
process.exit(1); | ||||||||||||
}); | ||||||||||||
} | ||||||||||||
) | ||||||||||||
.parse(); | ||||||||||||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,19 +1,18 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License | ||
* 2.0; you may not use this file except in compliance with the Elastic License | ||
* 2.0. | ||
* 2.0 and the Server Side Public License, v 1; you may not use this file except | ||
* in compliance with, at your election, the Elastic License 2.0 or the Server | ||
* Side Public License, v 1. | ||
*/ | ||
spong marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
import execa from 'execa'; | ||
import { resolve } from 'path'; | ||
|
||
const KIBANA_ROOT = resolve(__dirname, '../../../../../../'); | ||
import { REPO_ROOT } from '@kbn/repo-info'; | ||
|
||
export async function fixEslint(path: string) { | ||
await execa('npx', ['eslint', '--fix', path], { | ||
// Need to run eslint from the Kibana root directory, otherwise it will not | ||
// be able to pick up the right config | ||
cwd: KIBANA_ROOT, | ||
cwd: REPO_ROOT, | ||
}); | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License | ||
* 2.0 and the Server Side Public License, v 1; you may not use this file except | ||
* in compliance with, at your election, the Elastic License 2.0 or the Server | ||
* Side Public License, v 1. | ||
*/ | ||
|
||
export function getGeneratedFilePath(sourcePath: string) { | ||
return sourcePath.replace(/\..+$/, '.gen.ts'); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for including the detailed description here in the README as well!
WRT wider adoption and use, should we surface as a documentation set on docs.elastic.dev, or have an entry in the Kibana Developer Guide?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that's a good idea 👍 I've also been thinking about making a broader announcement, like sending an email about the codegen features to attract more early adopters. But starting with documentation is definitely a solid first step.