feat: unify error responses (#3607)

This PR implements the first version of a suggested unification (and
documentation) of the errors that we return from the API today.

The goal is for this to be the first step towards the error type defined
in this internal [linear
task](https://linear.app/unleash/issue/1-629/define-the-error-type
'Define the new API error type').

## The state of things today

As things stand, we currently have no (or **very** little) documentation
of the errors that are returned from the API. We mention error codes,
but never what the errors may contain.

Second, there is no specified format for errors, so what they return is
arbitrary, and based on ... Who knows? As a result, we have multiple
different errors returned by the API depending on what operation you're
trying to do. What's more, with OpenAPI validation in the mix, it's
absolutely possible for you to get two completely different error
objects for operations to the same endpoint.

Third, the errors we do return are usually pretty vague and don't really
provide any real help to the user. "You don't have the right
permissions". Great. Well what permissions do I need? And how would I
know? "BadDataError". Sick. Why is it bad?

... You get it.

## What we want to achieve

The ultimate goal is for error messages to serve both humans and
machines. When the user provides bad data, we should tell them what
parts of the data are bad and what they can do to fix it. When they
don't have the right permissions, we should tell them what permissions
they need.

Additionally, it would be nice if we could provide an ID for each error
instance, so that you (or an admin) can look through the logs and locate
he incident.

## What's included in **this** PR?

This PR does not aim to implement everything above. It's not intended to
magically fix everything. Its goal is to implement the necessary
**breaking** changes, so that they can be included in v5. Changing error
messages is a slightly grayer area than changing APIs directly, but
changing the format is definitely something I'd consider breaking.

So this PR:

- defines a minimal version of the error type defined in the [API error
definition linear
task](https://linear.app/unleash/issue/1-629/define-the-error-type).
- aims to catch all errors we return today and wrap them in the error
type
-   updates tests to match the new expectations.

An important point: because we are cutting v5 very soon and because work
for this wasn't started until last week, the code here isn't necessarily
very polished. But it doesn't need to be. The internals can be as messy
as we want, as long as the API surface is stable.

That said, I'm very open to feedback about design and code completeness,
etc, but this has intentionally been done quickly.

Please also see my inline comments on the changes for more specific
details.

### Proposed follow-ups

As mentioned, this is the first step to implementing the error type. The
public API error type only exposes `id`, `name`, and `message`. This is
barely any more than most of the previous messages, but they are now all
using the same format. Any additional properties, such as `suggestion`,
`help`, `documentationLink` etc can be added as features without
breaking the current format. This is an intentional limitation of this
PR.

Regarding additional properties: there are some error responses that
must contain extra properties. Some of these are documented in the types
of the new error constructor, but not all. This includes `path` and
`type` properties on 401 errors, `details` on validation errors, and
more.

Also, because it was put together quickly, I don't yet know exactly how
we (as developers) would **prefer** to use these new error messages
within the code, so the internal API (the new type, name, etc), is just
a suggestion. This can evolve naturally over time if (based on feedback
and experience) without changing the public API.

## Returning multiple errors

Most of the time when we return errors today, we only return a single
error (even if many things are wrong). AJV, the OpenAPI integration we
use does have a setting that allows it to return all errors in a request
instead of a single one. I suggest we turn that on, but that we do it in
a separate PR (because it updates a number of other snapshots).

When returning errors that point to `details`, the objects in the
`details` now contain a new `description` property. This "deprecates"
the `message` property. Due to our general deprecation policy, this
should be kept around for another full major and can be removed in v6.

```json
{
  "name": "BadDataError",
  "message": "Something went wrong. Check the `details` property for more information."
  "details": [{
    "message": "The .params property must be an object. You provided an array.",
    "description": "The .params property must be an object. You provided an array.",
  }]
}
```

src/lib/app.ts

@@ -28,6 +28,7 @@ import { Knex } from 'knex';
 import maintenanceMiddleware from './middleware/maintenance-middleware';
 import { unless } from './middleware/unless-middleware';
 import { catchAllErrorHandler } from './middleware/catch-all-error-handler';
+import { UnleashError } from './error/api-error';
 
 export default async function getApp(
     config: IUnleashConfig,
@@ -183,12 +184,17 @@ export default async function getApp(
         res.send(indexHTML);
     });
 
-    app.get(`${baseUriPath}/*`, (req, res) => {
-        if (req.path.startsWith(`${baseUriPath}/api`)) {
-            res.status(404).send({ message: 'Not found' });
-            return;
-        }
+    // handle all API 404s
+    app.use(`${baseUriPath}/api`, (req, res) => {
+        const error = new UnleashError({
+            name: 'NotFoundError',
+            message: `The path you were looking for (${baseUriPath}/api${req.path}) is not available.`,
+        });
+        res.status(error.statusCode).send(error);
+        return;
+    });
 
+    app.get(`${baseUriPath}/*`, (req, res) => {
         res.send(indexHTML);
     });
 

src/lib/error/api-error.test.ts

@@ -0,0 +1,219 @@
+import { ErrorObject } from 'ajv';
+import {
+    ApiErrorSchema,
+    fromOpenApiValidationError,
+    fromOpenApiValidationErrors,
+} from './api-error';
+
+describe('OpenAPI error conversion', () => {
+    it('Gives useful error messages for missing properties', () => {
+        const error = {
+            keyword: 'required',
+            instancePath: '',
+            dataPath: '.body',
+            schemaPath: '#/components/schemas/addonCreateUpdateSchema/required',
+            params: {
+                missingProperty: 'enabled',
+            },
+            message: "should have required property 'enabled'",
+        };
+
+        const { description } = fromOpenApiValidationError({})(error);
+
+        // it tells the user that the property is required
+        expect(description.includes('required'));
+        // it tells the user the name of the missing property
+        expect(description.includes(error.params.missingProperty));
+    });
+
+    it('Gives useful error messages for type errors', () => {
+        const error = {
+            keyword: 'type',
+            instancePath: '',
+            dataPath: '.body.parameters',
+            schemaPath:
+                '#/components/schemas/addonCreateUpdateSchema/properties/parameters/type',
+            params: {
+                type: 'object',
+            },
+            message: 'should be object',
+        };
+
+        const parameterValue = [];
+        const { description } = fromOpenApiValidationError({
+            parameters: parameterValue,
+        })(error);
+
+        // it provides the message
+        expect(description.includes(error.message));
+        // it tells the user what they provided
+        expect(description.includes(JSON.stringify(parameterValue)));
+    });
+
+    it('Gives useful pattern error messages', () => {
+        const error = {
+            instancePath: '',
+            keyword: 'pattern',
+            dataPath: '.body.description',
+            schemaPath:
+                '#/components/schemas/addonCreateUpdateSchema/properties/description/pattern',
+            params: {
+                pattern: '^this is',
+            },
+            message: 'should match pattern "^this is"',
+        };
+
+        const requestDescription = 'A pattern that does not match.';
+        const { description } = fromOpenApiValidationError({
+            description: requestDescription,
+        })(error);
+
+        // it tells the user what the pattern it should match is
+        expect(description.includes(error.params.pattern));
+        // it tells the user which property it pertains to
+        expect(description.includes('description'));
+        // it tells the user what they provided
+        expect(description.includes(requestDescription));
+    });
+
+    it('Gives useful min/maxlength error messages', () => {
+        const error = {
+            instancePath: '',
+            keyword: 'maxLength',
+            dataPath: '.body.description',
+            schemaPath:
+                '#/components/schemas/addonCreateUpdateSchema/properties/description/maxLength',
+            params: {
+                limit: 5,
+            },
+            message: 'should NOT be longer than 5 characters',
+        };
+
+        const requestDescription = 'Longer than the max length';
+        const { description } = fromOpenApiValidationError({
+            description: requestDescription,
+        })(error);
+
+        // it tells the user what the pattern it should match is
+        expect(description.includes(error.params.limit.toString()));
+        // it tells the user which property it pertains to
+        expect(description.includes('description'));
+        // it tells the user what they provided
+        expect(description.includes(requestDescription));
+    });
+
+    it('Handles numerical min/max errors', () => {
+        const error = {
+            keyword: 'maximum',
+            instancePath: '',
+            dataPath: '.body.newprop',
+            schemaPath:
+                '#/components/schemas/addonCreateUpdateSchema/properties/newprop/maximum',
+            params: {
+                comparison: '<=',
+                limit: 5,
+                exclusive: false,
+            },
+            message: 'should be <= 5',
+        };
+
+        const propertyValue = 6;
+        const { description } = fromOpenApiValidationError({
+            newprop: propertyValue,
+        })(error);
+
+        // it tells the user what the limit is
+        expect(description.includes(error.params.limit.toString()));
+        // it tells the user what kind of comparison it performed
+        expect(description.includes(error.params.comparison));
+        // it tells the user which property it pertains to
+        expect(description.includes('newprop'));
+        // it tells the user what they provided
+        expect(description.includes(propertyValue.toString()));
+    });
+
+    it('Handles multiple errors', () => {
+        const errors: [ErrorObject, ...ErrorObject[]] = [
+            {
+                keyword: 'maximum',
+                instancePath: '',
+                // @ts-expect-error
+                dataPath: '.body.newprop',
+                schemaPath:
+                    '#/components/schemas/addonCreateUpdateSchema/properties/newprop/maximum',
+                params: {
+                    comparison: '<=',
+                    limit: 5,
+                    exclusive: false,
+                },
+                message: 'should be <= 5',
+            },
+            {
+                keyword: 'required',
+                instancePath: '',
+                dataPath: '.body',
+                schemaPath:
+                    '#/components/schemas/addonCreateUpdateSchema/required',
+                params: {
+                    missingProperty: 'enabled',
+                },
+                message: "should have required property 'enabled'",
+            },
+        ];
+
+        // create an error and serialize it as it would be shown to the end user.
+        const serializedUnleashError: ApiErrorSchema =
+            fromOpenApiValidationErrors({ newprop: 7 }, errors).toJSON();
+
+        expect(serializedUnleashError.name).toBe('ValidationError');
+        expect(serializedUnleashError.message).toContain('`details`');
+        expect(
+            serializedUnleashError.details!![0].description.includes('newprop'),
+        );
+        expect(
+            serializedUnleashError.details!![1].description.includes('enabled'),
+        );
+    });
+
+    it('Handles deeply nested properties gracefully', () => {
+        const error = {
+            keyword: 'type',
+            dataPath: '.body.nestedObject.a.b',
+            schemaPath:
+                '#/components/schemas/addonCreateUpdateSchema/properties/nestedObject/properties/a/properties/b/type',
+            params: { type: 'string' },
+            message: 'should be string',
+            instancePath: '',
+        };
+
+        const { description } = fromOpenApiValidationError({
+            nestedObject: { a: { b: [] } },
+        })(error);
+
+        // it should hold the full path to the error
+        expect(description.includes('nestedObject.a.b'));
+        // it should include the value that the user sent
+        expect(description.includes('[]'));
+    });
+
+    it('Handles deeply nested properties on referenced schemas', () => {
+        const error = {
+            keyword: 'type',
+            dataPath: '.body.nestedObject.a.b',
+            schemaPath: '#/components/schemas/parametersSchema/type',
+            params: { type: 'object' },
+            message: 'should be object',
+            instancePath: '',
+        };
+
+        const illegalValue = 'illegal string';
+        const { description } = fromOpenApiValidationError({
+            nestedObject: { a: { b: illegalValue } },
+        })(error);
+
+        // it should hold the full path to the error
+        expect(description.includes('nestedObject.a.b'));
+        // it should include the value that the user sent
+        expect(description.includes(illegalValue));
+    });
+});