Skip to content

Commit

Permalink
Add output object type inference based upon Schema definition (#697)
Browse files Browse the repository at this point in the history
This PR enhances the capabilities of MVOM for type inference based on
the schema definition. These changes allow, given a schema, to infer the
output types of either a `Document` or a `Model` instance.

The following is a summary of the changes:

### `Schema` class
The `Schema` class has been changed so that there is now a generic
`TSchemaDefinition` on the class which will be inferred based on the
provided `definition` parameter to the `Schema` constructor. This
generic is the basis for all other changes included in this PR.

Several new utility types have been added to the `schema.ts` module to
allow for output object type inference:

The exported `InferDocumentObject` type accepts a `Schema` as a generic
and will output a type which aligns with the structure of a `Document`
that was instantiated based upon that schema's definition.

The exported `InferModelObject` type accepts a `Schema` as a generic and
will output a type which aligns with the structure of a `Model` that was
instantiated based upon that schema's definition. The primary difference
between this type and the `InferDocumentObject` type is that a `Model`
instance will include the `_id` and `__v` properties, so this is
effectively an extension of the `InferDocumentObject` type.

An internal `InferSchemaType` type provides most of the work regarding
the inference. It will accept a schema's type definition (that which is
assigned to a schema's property value) and recursively process it to
determine the output type of a property in the schema's definition. It
will handle each scalar type as well as embedded definitions, embedded
schemas, scalar arrays, nested scalar arrays, and document arrays.

An internal `InferStringType` type provides additional utility for
string schema type definitions. Since those type definitions may have an
enumeration constraint, this type will check if there is a defined
enumeration in the definition. If there is, and it is a readonly array,
then the output of the string type will be a union of the enumerated
strings.

An internal `InferRequiredType` type will detect whether a scalar type
is required or not. If it is not required then the resulting output will
be unioned with `null`.

#### Usage
Consumers of MVOM can use the `InferModelObject` and
`InferDocumentObject` types to generate a type which will provide the
shape of the output for a `Model` or `Document` as follows:

```ts
import { Schema } from 'mvom';
import type { InferDocumentObject, InferModelObject } from 'mvom';

const schema = new Schema({ stringProp: { type: 'string', path: '1' } });

type DocumentOutput = InferDocumentObject<typeof schema>; // { stringProp: string | null }
type ModelOutput = InferModelObject<typeof schema>; // { _id: string; __v: string; stringProp: string | null }
```

#### Testing of the utility types
The `Schema.test.ts` suite was updated to have several new tests to
confirm that the utility types emit the expected output type. A new
utility type named `Assert` was created which will compare two types and
return `true` if they match and an error output if they do not. This
`Assert` type was used to compare various schemas to their expected
output.

This test suite can also provide a lot of insight into the expected
output types for various schema definitions.

Note: A failed type assertion would not fail any unit tests, but it
would trigger typechecking errors. They have the same end result which
is that the CI suite would fail.

### compileModel.ts & dynamic `Model` class
The compileModel function has been augmented to take advantage of the
`InferModelObject` inference. The `schema` provided to the function will
have its output inferred. This output is used to form a new
`ModelCompositeValue` type which is the intersection of the `Model`
instance and the inferred output. The various static and instance
methods on the `Model` class which return instances of the `Model` will
now be of this composite object. Effectively, those methods will now
return a type which has all of the properties strongly typed as defined
by the schema.

Additionally, when instantiating a new `Model`, the `data` property
supplied to the constructor must now comply with the inferred object
shape. That is, there is type safety applied to the data that would
construct a `Model` instance.

### `Document` class
Similar to the changes made to the `Model` class, the `Document` class
now outputs a `DocumentCompositeValue` from the static methods that can
instantiate a new `Document` instance. This composite type will be the
intersection of the `Document` instance and the inferred output object.

### Schema type validator changes
Prior to this PR, the validators for a schema type would accept not only
the value being validated, but also the `Document` instance they were
being validated for. The passing of the `Document` instance was never
used by any validators. Because the `Document` is now generic, this
complicated the validation as all schema types would have needed to be
provided the type of the `Schema` which they were a member of. Since
this validation was never used anywhere, it was far simpler to simply
remove the document parameter from the validators.

### Query class changes
The `Query` class accept a `Model` constructor previous to this PR.
However, TypeScript was complaining about this constructor due to the
new composite output type of the `Model` methods. Instead, the `Query`
class was modified to individually accept as parameters the things it
was using from the `Model` constructor -- the connection, schema, and
file. The `Model` method were adjusted to provide this information
directly instead of providing its own constructor. This should have no
impact to anything with the `Query` class as the end result is
identical.
  • Loading branch information
shawnmcknight committed Jun 18, 2024
1 parent 23e9b48 commit d5cf1ca
Show file tree
Hide file tree
Showing 30 changed files with 1,894 additions and 442 deletions.
3 changes: 2 additions & 1 deletion jest.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ module.exports = {
'!**/index.d.ts', // do not test typescript declaration files
'!**/src/**/constants/**', // do not test shared constants folder
'!dist/**', // do not test compiled output
'!**/scripts/**', // do not test scripts
'!**/scripts/**', // do not test scripts,
'!**/website/**', // do not test docs website
],
coverageThreshold: {
'**/*.?([cm])[jt]s?(x)': {
Expand Down
12 changes: 6 additions & 6 deletions src/Connection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import {
import type { Logger } from './LogHandler';
import LogHandler from './LogHandler';
import type Schema from './Schema';
import type { SchemaDefinition } from './Schema';
import type {
DbServerDelimiters,
DbServerLimits,
Expand All @@ -42,7 +43,6 @@ import type {
DbSubroutineResponseTypes,
DbSubroutineResponseTypesMap,
DbSubroutineSetupOptions,
GenericObject,
} from './types';

// #region Types
Expand Down Expand Up @@ -344,18 +344,18 @@ class Connection {
}

/** Define a new model */
public model<TSchema extends GenericObject>(
schema: Schema | null,
file: string,
): ModelConstructor {
public model<
TSchema extends Schema<TSchemaDefinition> | null,
TSchemaDefinition extends SchemaDefinition,
>(schema: TSchema, file: string): ModelConstructor<TSchema, TSchemaDefinition> {
if (this.status !== ConnectionStatus.connected || this.dbServerInfo == null) {
this.logHandler.error('Cannot create model until database connection has been established');
throw new Error('Cannot create model until database connection has been established');
}

const { delimiters } = this.dbServerInfo;

return compileModel<TSchema>(this, schema, file, delimiters, this.logHandler);
return compileModel(this, schema, file, delimiters, this.logHandler);
}

/** Get the db server information (date, time, etc.) */
Expand Down
127 changes: 91 additions & 36 deletions src/Document.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,21 @@ import { assignIn, cloneDeep, get as getIn, set as setIn } from 'lodash';
import { TransformDataError } from './errors';
import ForeignKeyDbTransformer from './ForeignKeyDbTransformer';
import type Schema from './Schema';
import type { DbServerDelimiters, GenericObject, MvRecord } from './types';
import type { InferDocumentObject, SchemaDefinition } from './Schema';
import type { DbServerDelimiters, MvRecord } from './types';

// #region Types
export interface DocumentConstructorOptions {
data?: GenericObject;
/** Type of data property for constructing a document dependent upon the schema */
export type DocumentData<
TSchema extends Schema<TSchemaDefinition> | null,
TSchemaDefinition extends SchemaDefinition,
> = TSchema extends Schema<TSchemaDefinition> ? InferDocumentObject<TSchema> : never;

export interface DocumentConstructorOptions<
TSchema extends Schema<TSchemaDefinition> | null,
TSchemaDefinition extends SchemaDefinition,
> {
data?: DocumentData<TSchema, TSchemaDefinition>;
record?: MvRecord;
isSubdocument?: boolean;
}
Expand All @@ -16,27 +26,45 @@ export interface BuildForeignKeyDefinitionsResult {
entityName: string;
entityIds: string[];
}

/**
* An intersection type that combines the `Document` class instance with the
* inferred shape of the document object based on the schema definition.
*/
type DocumentCompositeValue<
TSchema extends Schema<TSchemaDefinition> | null,
TSchemaDefinition extends SchemaDefinition,
> =
TSchema extends Schema<TSchemaDefinition>
? Document<TSchema, TSchemaDefinition> & InferDocumentObject<TSchema>
: Document<TSchema, TSchemaDefinition>;
// #endregion

/** A document object */
class Document {
class Document<
TSchema extends Schema<TSchemaDefinition> | null,
TSchemaDefinition extends SchemaDefinition,
> {
[key: string]: unknown;

public _raw?: MvRecord;
public _raw: TSchema extends Schema<TSchemaDefinition> ? undefined : MvRecord;

/** Array of any errors which occurred during transformation from the database */
public _transformationErrors: TransformDataError[];

/** Schema instance which defined this document */
readonly #schema: Schema | null;
readonly #schema: TSchema;

/** Record array of multivalue data */
#record: MvRecord;

/** Indicates whether this document is a subdocument of a composing parent */
readonly #isSubdocument: boolean;

protected constructor(schema: Schema | null, options: DocumentConstructorOptions) {
protected constructor(
schema: TSchema,
options: DocumentConstructorOptions<TSchema, TSchemaDefinition>,
) {
const { data = {}, record, isSubdocument = false } = options;

this.#schema = schema;
Expand All @@ -48,31 +76,53 @@ class Document {
_transformationErrors: { configurable: false, enumerable: false, writable: false },
});

this._raw = (
schema == null ? this.#record : undefined
) as TSchema extends Schema<TSchemaDefinition> ? undefined : MvRecord;

this.#transformRecordToDocument();

// load the data passed to constructor into document instance
assignIn(this, data);
}

/** Create a new Subdocument instance from a record array */
public static createSubdocumentFromRecord(schema: Schema, record: MvRecord): Document {
return new Document(schema, { record, isSubdocument: true });
public static createSubdocumentFromRecord<
TSchema extends Schema<TSchemaDefinition> | null,
TSchemaDefinition extends SchemaDefinition,
>(schema: TSchema, record: MvRecord): DocumentCompositeValue<TSchema, TSchemaDefinition> {
return new Document(schema, { record, isSubdocument: true }) as DocumentCompositeValue<
TSchema,
TSchemaDefinition
>;
}

/** Create a new Subdocument instance from data */
public static createSubdocumentFromData(schema: Schema, data: GenericObject): Document {
return new Document(schema, { data, isSubdocument: true });
public static createSubdocumentFromData<
TSchema extends Schema<TSchemaDefinition>,
TSchemaDefinition extends SchemaDefinition,
>(
schema: TSchema,
data: DocumentData<TSchema, TSchemaDefinition>,
): DocumentCompositeValue<TSchema, TSchemaDefinition> {
return new Document(schema, { data, isSubdocument: true }) as DocumentCompositeValue<
TSchema,
TSchemaDefinition
>;
}

/** Create a new Document instance from a record string */
public static createDocumentFromRecordString(
schema: Schema,
public static createDocumentFromRecordString<
TSchema extends Schema<TSchemaDefinition> | null,
TSchemaDefinition extends SchemaDefinition,
>(
schema: TSchema,
recordString: string,
dbServerDelimiters: DbServerDelimiters,
): Document {
): DocumentCompositeValue<TSchema, TSchemaDefinition> {
const record = Document.convertMvStringToArray(recordString, dbServerDelimiters);

return new Document(schema, { record });
return new Document(schema, { record }) as DocumentCompositeValue<TSchema, TSchemaDefinition>;
}

/** Convert a multivalue string to an array */
Expand Down Expand Up @@ -198,7 +248,7 @@ class Document {
value = schemaType.cast(value);
setIn(this, keyPath, value);

const validationResult = await schemaType.validate(value, this);
const validationResult = await schemaType.validate(value);
if (validationResult instanceof Map) {
validationResult.forEach((errors, key) => {
if (errors.length > 0) {
Expand All @@ -220,26 +270,31 @@ class Document {

/** Apply schema structure using record to document instance */
#transformRecordToDocument() {
const plainDocument =
this.#schema === null
? { _raw: this.#record }
: Array.from(this.#schema.paths).reduce((document, [keyPath, schemaType]) => {
let setValue;
try {
setValue = schemaType.get(this.#record);
} catch (err) {
if (err instanceof TransformDataError) {
// if this was an error in data transformation, set the value to null and add to transformationErrors list
setValue = null;
this._transformationErrors.push(err);
} else {
// otherwise rethrow any other type of error
throw err;
}
}
setIn(document, keyPath, setValue);
return document;
}, {});
if (this.#schema == null) {
// if this is a document without a schema, there is nothing to transform
return;
}

const plainDocument = Array.from(this.#schema.paths).reduce(
(document, [keyPath, schemaType]) => {
let setValue;
try {
setValue = schemaType.get(this.#record);
} catch (err) {
if (err instanceof TransformDataError) {
// if this was an error in data transformation, set the value to null and add to transformationErrors list
setValue = null;
this._transformationErrors.push(err);
} else {
// otherwise rethrow any other type of error
throw err;
}
}
setIn(document, keyPath, setValue);
return document;
},
{},
);

assignIn(this, plainDocument);
}
Expand Down
42 changes: 25 additions & 17 deletions src/Query.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import type { ModelConstructor } from './compileModel';
import type Connection from './Connection';
import { InvalidParameterError, QueryLimitError } from './errors';
import type LogHandler from './LogHandler';
import type { SchemaDefinition } from './Schema';
import type Schema from './Schema';
import type {
DbDocument,
DbSubroutineInputFind,
Expand Down Expand Up @@ -71,8 +73,14 @@ export interface QueryExecutionResult {

/** A query object */
class Query<TSchema extends GenericObject = GenericObject> {
/** Model constructor to use with query */
private readonly Model: ModelConstructor;
/** Connection instance to run query on */
private readonly connection: Connection;

/** Schema used for query */
private readonly schema: Schema<SchemaDefinition> | null;

/** File to run query against */
private readonly file: string;

/** Log handler instance used for diagnostic logging */
private readonly logHandler: LogHandler;
Expand All @@ -99,14 +107,18 @@ class Query<TSchema extends GenericObject = GenericObject> {
private conditionCount = 0;

public constructor(
Model: ModelConstructor,
connection: Connection,
schema: Schema<SchemaDefinition> | null,
file: string,
logHandler: LogHandler,
selectionCriteria: Filter<TSchema>,
options: QueryConstructorOptions = {},
) {
const { sort, limit, skip, projection } = options;

this.Model = Model;
this.connection = connection;
this.schema = schema;
this.file = file;
this.logHandler = logHandler;
this.limit = limit;
this.skip = skip;
Expand All @@ -120,7 +132,7 @@ class Query<TSchema extends GenericObject = GenericObject> {
/** Execute query */
public async exec(options: QueryExecutionOptions = {}): Promise<QueryExecutionResult> {
const { maxReturnPayloadSize, requestId, userDefined } = options;
let queryCommand = `select ${this.Model.file}`;
let queryCommand = `select ${this.file}`;
if (this.selection != null) {
queryCommand = `${queryCommand} with ${this.selection}`;
}
Expand All @@ -131,12 +143,12 @@ class Query<TSchema extends GenericObject = GenericObject> {
await this.validateQuery(queryCommand, requestId);

const projection =
this.projection != null && this.Model.schema != null
? this.Model.schema.transformPathsToDbPositions(this.projection)
this.projection != null && this.schema != null
? this.schema.transformPathsToDbPositions(this.projection)
: null;

const executionOptions: DbSubroutineInputFind = {
filename: this.Model.file,
filename: this.file,
queryCommand,
...(this.skip != null && { skip: this.skip }),
...(this.limit != null && { limit: this.limit }),
Expand All @@ -150,11 +162,7 @@ class Query<TSchema extends GenericObject = GenericObject> {
};

this.logHandler.debug(`executing query "${queryCommand}"`);
const data = await this.Model.connection.executeDbSubroutine(
'find',
executionOptions,
setupOptions,
);
const data = await this.connection.executeDbSubroutine('find', executionOptions, setupOptions);

return {
count: data.count,
Expand Down Expand Up @@ -317,7 +325,7 @@ class Query<TSchema extends GenericObject = GenericObject> {
* @throws {link InvalidParameterError} Nonexistent schema property or property does not have a dictionary specified
*/
private getDictionaryId(property: string): string {
const dictionaryTypeDetail = this.Model.schema?.dictPaths.get(property);
const dictionaryTypeDetail = this.schema?.dictPaths.get(property);
if (dictionaryTypeDetail == null) {
throw new InvalidParameterError({
message: 'Nonexistent schema property or property does not have a dictionary specified',
Expand All @@ -330,7 +338,7 @@ class Query<TSchema extends GenericObject = GenericObject> {

/** Transform query constant to internal u2 format (if applicable) */
private transformToQuery(property: string, constant: unknown): unknown {
const dictionaryTypeDetail = this.Model.schema?.dictPaths.get(property);
const dictionaryTypeDetail = this.schema?.dictPaths.get(property);
if (dictionaryTypeDetail == null) {
throw new InvalidParameterError({
message: 'Nonexistent schema property or property does not have a dictionary specified',
Expand All @@ -343,7 +351,7 @@ class Query<TSchema extends GenericObject = GenericObject> {

/** Validate the query before execution */
private async validateQuery(query: string, requestId?: string): Promise<void> {
const { maxSort, maxWith, maxSentenceLength } = await this.Model.connection.getDbLimits({
const { maxSort, maxWith, maxSentenceLength } = await this.connection.getDbLimits({
requestId,
});

Expand Down
Loading

0 comments on commit d5cf1ca

Please sign in to comment.