Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] - Add async serdes.deserialize support & req/res as context #749

Open
wants to merge 25 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
2954777
[feat] - Async deserialization and req/res as
macyabbey-okta Jul 17, 2022
669f03b
Capture knowledge gained on how the package works, thoughts and struc…
macyabbey-okta Jul 17, 2022
dcf40fb
Self code review. Cleaning up comments. Single module async-utils. Mo…
macyabbey-okta Jul 17, 2022
273069d
Impl and test coverage for what is expected when async deserialize th…
macyabbey-okta Jul 17, 2022
f6ba222
Fix mermaid
macyabbey-okta Jul 17, 2022
e50c9d6
Add npm run test:base to make it easier to target specific tests.
macyabbey-okta Jul 17, 2022
d5a1bc2
Remove not-needed change to non-async serdes error handling which als…
macyabbey-okta Jul 17, 2022
c84bbc7
Let sync deserialize/serialize influence error messages too.
macyabbey-okta Jul 18, 2022
50fea82
Fixing gaps in schema traversal (oneOf/allOf/anyOf/items)
macyabbey-okta Jul 18, 2022
da67f24
More coverage for updating schema with in case of XOF/array/items.
macyabbey-okta Jul 19, 2022
9e9b302
Fixing bug with async and context support when XOF has an async sub-s…
macyabbey-okta Jul 19, 2022
c609e86
Fix bug where instancePath of schemas in a request body where top lev…
macyabbey-okta Jul 19, 2022
e5980a9
Convenience for trigger debug
macyabbey-okta Jul 19, 2022
445f165
Remove spread operators.
macyabbey-okta Jul 21, 2022
6be17fa
Squash validation messages from oneOf sub-schemas that do not match t…
macyabbey-okta Jul 28, 2022
94bde72
Allowing new option 'filterOneOf' to pair down errors thrown by oneOf…
macyabbey-okta Aug 1, 2022
b5058be
Fix handling of items in a schema, better typing.
macyabbey-okta Aug 1, 2022
5e1c573
Handling no items case properly
macyabbey-okta Aug 1, 2022
3d25af8
Use most specific oneOf subschema
macyabbey-okta Aug 1, 2022
8125da9
De-duping errors for same keyword/path combinations.
macyabbey-okta Aug 10, 2022
d99cd04
If schema has discriminator and a default, apply default values to th…
macyabbey-okta Aug 12, 2022
9bc3885
Fix slice to get ref to component schema
macyabbey-okta Aug 12, 2022
d03c2a6
Let HttpErrors thrown from serdes surface with error name as keyword …
macyabbey-okta Aug 18, 2022
fb72ff1
check for object in discriminator validation
colinwang-okta Oct 12, 2022
349a0b9
check if oneOf entry is actually type object
colinwang-okta Oct 12, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 52 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,15 @@ Click the Gitpod badge to setup a ready to code dev env in the cloud.
2. Install the dependencies

```shell
#
# Note was only able to develop effectively in this repo using:
# - npm v6.x, as its still on package lock version 1
# - node 14.x, ran into issues when trying to develop on node 16/18
#
# From the project directory, run
npm i
```

Be [Create a Pull Request](#create-a-pull-request) once you've written you code.

## Run the tests
Expand All @@ -52,7 +57,7 @@ Be [Create a Pull Request](#create-a-pull-request) once you've written you code.
3. Make a new branch
4. Make your changes
5. Push it back to your repo
6. From the Github UI, Click the Compare & pull request button
6. From the Github UI, Click the Compare & pull request button

NOTE: this button will be present for some period of time after 5. If the button no longer there, Create pull request and select the branches manually)
6. From the Github UI, Click Create pull request to open a new pull request
Expand All @@ -63,6 +68,50 @@ Be [Create a Pull Request](#create-a-pull-request) once you've written you code.
`src` contains the source code
`test` contains the tests

### How it works

This middleware uses [ajv](https://ajv.js.org/) to validate
request and responses according to an OpenAPI document.

You can think of it in the following steps:

1. Parse/normalize configuration options.
2. Load the spec file
3. Setup internal data structures (Open API document, Ajv schemas).
4. Create ajv dynamic validation functions for use in the middleware pipeline.
5. Establish a [install middleware pipeline](./src/openapi.validator.ts)
6. Throw errors from the pipeline if any aspect of the request or response deviate from the spec. These errors have appropriate HTTP status codes and rich data based on OpenApi specification.

#### Setup internal data structures

This is a complex part of the project, and this library has a few categories of data structures.

1) A typed Open API document, with some amount of processing that still conforms to the Open API specification and closely aligns to the configured Open API document.
2) A subset of the Open API document scoped down to a particular route. Can observe this in [./src/framework/openapi.context.ts](./src/framework/openapi.context.ts)
3) Be-spoke data structures for request / response validation that pick out parts of the Open API document, then run ajv validation on them, using JSON schema definitions lifted from the Open API specification. This can be seen in `bodySchema`, `generalSchema` of [./src/openapi.request.validator.ts](./src/openapi.request.validator.ts), as well as some of the `ajv` initialization in [./src/framework/ajv/index.ts](./src/framework/ajv/index.ts)

See code notes below.

#### Middleware pipeline

```mermaid
graph LR;
A["Parse path parameters"]-->B["Attach req.openapi"];
B-->C["Multi-part parsing / validation"];
C-->D["Security enforcement"];
D-->E["Request validation"];
E-->F["Response validation"];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems off; how can a response be validated before a handler has an opportunity to generate it?

F-->G["Operation handlers"];
```

Code notes

- [src/middlewares/parsers](./src/middlewares/parsers) - Parsing aspects of a request based on the openapi schema. For example, parsing request parameters into easy to consume object structures based on a parameter's definition in the [OpenAPI document](https://swagger.io/docs/specification/describing-parameters/).
- [src/middlewares/parsers/schema.preprocessor.ts](./src/middlewares/parsers/schema.preprocessor.ts) - Complex pre-processing of an openapi document to enable custom features like `serdes` functionality and `intuitive handling of readOnly/writeOnly`. The custom JSON schema keywords defined in [./src/framework/ajv/index.ts](./src/framework/ajv/index.ts) only work based on assumed decoration of the Open API document schemas done by this preprocessor. This dependence w/ low cohesion could be an improvement point in the future to make this package easier to follow.
- [src/framework/ajv](./src/framework/ajv) Ajv configuration / initialization
- The library uses entirely runtime `ajv compilation`. This is sub-optimal. Ideally it would provide a means for downstream consumers to generate necessary validation functions at build time, which are then referenced at runtime. See [ajv standalone validation code](https://ajv.js.org/standalone.html)
- This library is big on stateful classes with side-effects. These are hard to understand. Especially the "tricks" used to share memory between the request and response Open API documents, the be-spoke schemas, and the various different ajv runtime compile functions.

## Need help?

Reach out on [gitter](https://gitter.im/cdimascio-oss/community).
Expand All @@ -75,7 +124,7 @@ We're happy to help!
**A:** You cannot directly create a branch in this repo. Instead [Create a Pull Request](#create-a-pull-request)


## Misc
## Misc
If you are not a project, you may ignore this section

### Generate Change Log
Expand Down
18 changes: 14 additions & 4 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 5 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@
"scripts": {
"compile": "rm -rf dist/ && tsc",
"compile:windows": "rmdir dist /s /q & tsc",
"test": "TS_NODE_FILES=true mocha -r source-map-support/register -r ts-node/register --files --recursive -R spec test/**/*.spec.ts",
"test:debug": "TS_NODE_FILES=true mocha -r source-map-support/register -r ts-node/register --inspect-brk --files --recursive test/**/*.spec.ts",
"test:base": "TS_NODE_FILES=true mocha -r source-map-support/register -r ts-node/register --files --recursive -R spec",
"test": "npm run test:base test/**/*.spec.ts",
"test:debug:base": "TS_NODE_FILES=true mocha -r source-map-support/register -r ts-node/register --inspect-brk --files --recursive",
"test:debug": " npm run test:debug:base test/**/*.spec.ts",
"test:windows": "set TS_NODE_FILES=true & mocha -r source-map-support/register -r ts-node/register --files --recursive test/**/*.spec.ts",
"test:coverage": "TS_NODE_FILES=true nyc mocha -r source-map-support/register -r ts-node/register --recursive test/**/*.spec.ts",
"test:coverage:windows": "set TS_NODE_FILES=true & nyc mocha -r source-map-support/register -r ts-node/register --recursive test/**/*.spec.ts",
Expand All @@ -32,7 +34,6 @@
"author": "Carmine DiMascio <cdimascio@gmail.com>",
"license": "MIT",
"dependencies": {
"@types/multer": "^1.4.7",
"ajv": "^8.6.2",
"ajv-draft-04": "^1.0.0",
"ajv-formats": "^2.1.1",
Expand All @@ -52,6 +53,7 @@
"@types/express": "4.17.13",
"@types/mocha": "^9.1.0",
"@types/morgan": "^1.9.3",
"@types/multer": "^1.4.7",
"@types/node": "^17.0.21",
"@types/supertest": "^2.0.11",
"body-parser": "^1.19.2",
Expand Down
Loading