feat(core): rework serialization rules to always respect populate hint (

#4203)

Implicit serialization, so calling `toObject()` or `toJSON()` on the
entity, as opposed to explicitly using the `serialize()` helper, now
works entirely based on `populate` hints. This means that, unless you
explicitly marked some entity as populated via
`wrap(entity).populated()`, it will be part of the serialized form only
if it was part of the `populate` hint:

```ts
// let's say both Author and Book entity has a m:1 relation to Publisher entity
// we only populate the publisher relation of the Book entity
const user = await em.findOneOrFail(Author, 1, {
  populate: ['books.publisher'],
});

const dto = wrap(user).toObject();
console.log(dto.publisher); // only the FK, e.g. `123`
console.log(dto.books[0].publisher); // populated, e.g. `{ id: 123, name: '...' }`
```

Moreover, the implicit serialization now respects the partial loading
hints too. Previously, all loaded properties were serialized, partial
loading worked only on the database query level. Since v6, we also prune
the data on runtime. This means that unless the property is part of the
partial loading hint (`fields` option), it won't be part of the DTO -
only exception is the primary key, you can optionally hide it via
`hidden: true` in the property options. Main difference here will be the
foreign keys, those are often automatically selected as they are needed
to build the entity graph, but will no longer be part of the DTO.

```ts
const user = await em.findOneOrFail(Author, 1, {
  fields: ['books.publisher.name'],
});

const dto = wrap(user).toObject();
// only the publisher's name will be available, previously there would be also `book.author`
// `{ id: 1, books: [{ id: 2, publisher: { id: 3, name: '...' } }] }`
```

**This also works for embeddables, including nesting and object mode.**

This method used to return a single object conditionally based on its
inputs, but the solution broke intellisense for the `populate` option.
The method signature still accepts single object or an array of objects,
but always returns an array.

To serialize single entity, you can use array destructing, or use
`wrap(entity).serialize()` which handles a single entity only.

```ts
const dtos = serialize([user1, user, ...], { exclude: ['id', 'email'], forceObject: true });
const [dto1] = serialize(user, { exclude: ['id', 'email'], forceObject: true });
const dto2 = wrap(user).serialize({ exclude: ['id', 'email'], forceObject: true });
```

Closes #4138
Closes #4199

docs/docs/serializing.md

@@ -92,29 +92,71 @@ const book = new Book(author);
 console.log(wrap(book).toJSON().authorName); // 'God'
 ```
 
+## Implicit serialization
+
+Implicit serialization means calling `toObject()` or `toJSON()` on the entity, as opposed to explicitly using the `serialize()` helper. Since v6, it works entirely based on `populate` hints. This means that, unless you explicitly marked some entity as populated via `wrap(entity).populated()`, it will be part of the serialized form only if it was part of the `populate` hint:
+
+```ts
+// let's say both Author and Book entity has a m:1 relation to Publisher entity
+// we only populate the publisher relation of the Book entity
+const user = await em.findOneOrFail(Author, 1, {
+  populate: ['books.publisher'],
+});
+
+const dto = wrap(user).toObject();
+console.log(dto.publisher); // only the FK, e.g. `123`
+console.log(dto.books[0].publisher); // populated, e.g. `{ id: 123, name: '...' }`
+```
+
+Moreover, the implicit serialization now respects the partial loading hints too. Previously, all loaded properties were serialized, partial loading worked only on the database query level. Since v6, we also prune the data on runtime. This means that unless the property is part of the partial loading hint (`fields` option), it won't be part of the DTO. Main difference here is the primary and foreign keys, that are often automatically selected as they are needed to build the entity graph, but will no longer be part of the DTO.
+
+```ts
+const user = await em.findOneOrFail(Author, 1, {
+  fields: ['books.publisher.name'],
+});
+
+const dto = wrap(user).toObject();
+// only the publisher's name will be available + primary keys
+// `{ id: 1, books: [{ id: 2, publisher: { id: 3, name: '...' } }] }`
+```
+
+Primary keys are automatically included. If you want to hide them, you have two options:
+
+- use `hidden: true` in the property options
+- use `serialization: { includePrimaryKeys: false }` in the ORM config
+
+**This also works for embeddables, including nesting and object mode.**
+
 ## Explicit serialization
 
 The serialization process is normally driven by the `populate` hints. If you want to take control over this, you can use the `serialize()` helper:
 
 ```ts
 import { serialize } from '@mikro-orm/core';
 
-const dto = serialize(user); // serialize single entity
+const dtos = serialize([user1, user2]);
+// [
+//   { name: '...', books: [1, 2, 3], identity: 123 },
+//   { name: '...', ... },
+// ]
+
+const [dto] = serialize(user1); // always returns an array
 // { name: '...', books: [1, 2, 3], identity: 123 }
 
-const dtos = serialize(users); // supports arrays as well
-// [{ name: '...', books: [1, 2, 3], identity: 123 }, ...]
+// for a single entity instance we can as well use `wrap(e).serialize()`
+const dto2 = wrap(user1).serialize();
+// { name: '...', books: [1, 2, 3], identity: 123 }
 ```
 
 By default, every relation is considered as not populated - this will result in the foreign key values to be present. Loaded collections will be represented as arrays of the foreign keys. To control the shape of the serialized response we can use the second `options` parameter:
 
 ```ts
-export interface SerializeOptions<T extends object, P extends string = never> {
+export interface SerializeOptions<T extends object, P extends string = never, E extends string = never> {
   /** Specify which relation should be serialized as populated and which as a FK. */
   populate?: AutoPath<T, P>[] | boolean;
 
   /** Specify which properties should be omitted. */
-  exclude?: AutoPath<T, P>[];
+  exclude?: AutoPath<T, E>[];
 
   /** Enforce unpopulated references to be returned as objects, e.g. `{ author: { id: 1 } }` instead of `{ author: 1 }`. */
   forceObject?: boolean;
@@ -130,9 +172,9 @@ export interface SerializeOptions<T extends object, P extends string = never> {
 Here is a more complex example:
 
 ```ts
-import { serialize } from '@mikro-orm/core';
+import { wrap } from '@mikro-orm/core';
 
-const dto = serialize(author, {
+const dto = wrap(author).serialize({
   populate: ['books.author', 'books.publisher', 'favouriteBook'], // populate some relations
   exclude: ['books.author.email'], // skip property of some relation
   forceObject: true, // not populated or not initialized relations will result in object, e.g. `{ author: { id: 1 } }`

docs/docs/upgrading-v5-to-v6.md

@@ -234,3 +234,45 @@ console.log(ref.age); // real value is available after flush
 ## Metadata CacheAdapter requires sync API
 
 To allow working with cache inside `MikroORM.initSync`, the metadata cache now enforces sync API. You should usually depend on the file-based cache for the metadata, which now uses sync methods to work with the file system.
+
+## Implicit serialization changes
+
+Implicit serialization, so calling `toObject()` or `toJSON()` on the entity, as opposed to explicitly using the `serialize()` helper, now works entirely based on `populate` hints. This means that, unless you explicitly marked some entity as populated via `wrap(entity).populated()`, it will be part of the serialized form only if it was part of the `populate` hint:
+
+```ts
+// let's say both Author and Book entity has a m:1 relation to Publisher entity
+// we only populate the publisher relation of the Book entity
+const user = await em.findOneOrFail(Author, 1, {
+  populate: ['books.publisher'],
+});
+
+const dto = wrap(user).toObject();
+console.log(dto.publisher); // only the FK, e.g. `123`
+console.log(dto.books[0].publisher); // populated, e.g. `{ id: 123, name: '...' }`
+```
+
+Moreover, the implicit serialization now respects the partial loading hints too. Previously, all loaded properties were serialized, partial loading worked only on the database query level. Since v6, we also prune the data on runtime. This means that unless the property is part of the partial loading hint (`fields` option), it won't be part of the DTO - only exception is the primary key, you can optionally hide it via `hidden: true` in the property options. Main difference here will be the foreign keys, those are often automatically selected as they are needed to build the entity graph, but will no longer be part of the DTO.
+
+```ts
+const user = await em.findOneOrFail(Author, 1, {
+  fields: ['books.publisher.name'],
+});
+
+const dto = wrap(user).toObject();
+// only the publisher's name will be available, previously there would be also `book.author`
+// `{ id: 1, books: [{ id: 2, publisher: { id: 3, name: '...' } }] }`
+```
+
+**This also works for embeddables, including nesting and object mode.**
+
+## `serialize` helper always returns array
+
+This method used to return a single object conditionally based on its inputs, but the solution broke intellisense for the `populate` option. The method signature still accepts single object or an array of objects, but always returns an array.
+
+To serialize single entity, you can use array destructing, or use `wrap(entity).serialize()` which handles a single entity only.
+
+```ts
+const dtos = serialize([user1, user, ...], { exclude: ['id', 'email'], forceObject: true });
+const [dto1] = serialize(user, { exclude: ['id', 'email'], forceObject: true });
+const dto2 = wrap(user).serialize({ exclude: ['id', 'email'], forceObject: true });
+```

packages/core/src/EntityManager.ts

@@ -40,6 +40,7 @@ import type {
   RequiredEntityData,
   Ref,
   EntityKey,
+  AnyString,
 } from './typings';
 import type { TransactionOptions } from './enums';
 import { FlushMode, LoadStrategy, LockMode, PopulateHint, ReferenceKind, SCALAR_TYPES } from './enums';
@@ -146,7 +147,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
   async find<
     Entity extends object,
     Hint extends string = never,
-    Fields extends string = '*',
+    Fields extends string = never,
   >(entityName: EntityName<Entity>, where: FilterQuery<Entity>, options: FindOptions<Entity, Hint, Fields> = {}): Promise<Loaded<Entity, Hint, Fields>[]> {
     if (options.disableIdentityMap) {
       const em = this.getContext(false);
@@ -294,17 +295,17 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
   protected async processWhere<
     Entity extends object,
     Hint extends string = never,
-    Fields extends string = '*',
+    Fields extends string = never,
   >(entityName: string, where: FilterQuery<Entity>, options: FindOptions<Entity, Hint, Fields> | FindOneOptions<Entity, Hint, Fields>, type: 'read' | 'update' | 'delete'): Promise<FilterQuery<Entity>> {
     where = QueryHelper.processWhere({
-      where: where as FilterQuery<Entity>,
+      where,
       entityName,
       metadata: this.metadata,
       platform: this.driver.getPlatform(),
       convertCustomTypes: options.convertCustomTypes,
       aliased: type === 'read',
     });
-    where = await this.applyFilters(entityName, where, options.filters ?? {}, type);
+    where = (await this.applyFilters(entityName, where, options.filters ?? {}, type))!;
     where = await this.applyDiscriminatorCondition(entityName, where);
 
     return where;
@@ -335,7 +336,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
   /**
    * @internal
    */
-  async applyFilters<Entity extends object>(entityName: string, where: FilterQuery<Entity>, options: Dictionary<boolean | Dictionary> | string[] | boolean, type: 'read' | 'update' | 'delete'): Promise<FilterQuery<Entity>> {
+  async applyFilters<Entity extends object>(entityName: string, where: FilterQuery<Entity> | undefined, options: Dictionary<boolean | Dictionary> | string[] | boolean, type: 'read' | 'update' | 'delete'): Promise<FilterQuery<Entity> | undefined> {
     const meta = this.metadata.find<Entity>(entityName);
     const filters: FilterDef[] = [];
     const ret: Dictionary[] = [];
@@ -397,7 +398,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
   async findAndCount<
     Entity extends object,
     Hint extends string = never,
-    Fields extends string = '*',
+    Fields extends string = never,
   >(entityName: EntityName<Entity>, where: FilterQuery<Entity>, options: FindOptions<Entity, Hint, Fields> = {}): Promise<[Loaded<Entity, Hint, Fields>[], number]> {
     const em = this.getContext(false);
     const [entities, count] = await Promise.all([
@@ -463,7 +464,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
   async findByCursor<
     Entity extends object,
     Hint extends string = never,
-    Fields extends string = '*',
+    Fields extends string = never,
   >(entityName: EntityName<Entity>, where: FilterQuery<Entity>, options: FindByCursorOptions<Entity, Hint, Fields> = {}): Promise<Cursor<Entity, Hint, Fields>> {
     const em = this.getContext(false);
     entityName = Utils.className(entityName);
@@ -484,7 +485,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
   async refresh<
     Entity extends object,
     Hint extends string = never,
-    Fields extends string = '*',
+    Fields extends string = never,
   >(entity: Entity, options: FindOneOptions<Entity, Hint, Fields> = {}): Promise<Loaded<Entity, Hint, Fields> | null> {
     const fork = this.fork();
     const entityName = entity.constructor.name;
@@ -509,7 +510,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
   async findOne<
     Entity extends object,
     Hint extends string = never,
-    Fields extends string = '*',
+    Fields extends string = never,
   >(entityName: EntityName<Entity>, where: FilterQuery<Entity>, options: FindOneOptions<Entity, Hint, Fields> = {}): Promise<Loaded<Entity, Hint, Fields> | null> {
     if (options.disableIdentityMap) {
       const em = this.getContext(false);
@@ -584,7 +585,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
   async findOneOrFail<
     Entity extends object,
     Hint extends string = never,
-    Fields extends string = '*',
+    Fields extends string = never,
   >(entityName: EntityName<Entity>, where: FilterQuery<Entity>, options: FindOneOrFailOptions<Entity, Hint, Fields> = {}): Promise<Loaded<Entity, Hint, Fields>> {
     let entity: Loaded<Entity, Hint, Fields> | null;
     let isStrictViolation = false;
@@ -871,7 +872,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
       }
 
       const pks = await this.driver.find(meta.className, { $or: [...loadPK.values()] as Dictionary[] }, {
-        fields: meta.primaryKeys.concat(...add),
+        fields: meta.primaryKeys.concat(...add) as any,
         ctx: em.transactionContext,
         convertCustomTypes: true,
       });
@@ -1347,7 +1348,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
   /**
    * @internal
    */
-  async tryFlush<Entity extends object>(entityName: EntityName<Entity>, options: { flushMode?: FlushMode }): Promise<void> {
+  async tryFlush<Entity extends object>(entityName: EntityName<Entity>, options: { flushMode?: FlushMode | AnyString }): Promise<void> {
     const em = this.getContext();
     const flushMode = options.flushMode ?? em.flushMode ?? em.config.get('flushMode');
     entityName = Utils.className(entityName);
@@ -1586,7 +1587,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
     return entity as Loaded<T, P, F>;
   }
 
-  private buildFields<T extends object, P extends string>(fields: readonly EntityField<T, P>[]): readonly AutoPath<T, P>[] {
+  private buildFields<T extends object, P extends string>(fields: readonly EntityField<T, P>[]): string[] {
     return fields.reduce((ret, f) => {
       if (Utils.isPlainObject(f)) {
         Utils.keys(f).forEach(ff => ret.push(...this.buildFields(f[ff]!).map(field => `${ff as string}.${field}` as never)));
@@ -1601,15 +1602,42 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
   private preparePopulate<
     Entity extends object,
     Hint extends string = never,
-    Fields extends string = '*',
+    Fields extends string = never,
   >(entityName: string, options: Pick<FindOptions<Entity, Hint, Fields>, 'populate' | 'strategy' | 'fields'>): PopulateOptions<Entity>[] {
     // infer populate hint if only `fields` are available
     if (!options.populate && options.fields) {
-      options.populate = this.buildFields(options.fields) as any;
+      const meta = this.metadata.find(entityName)!;
+      // we need to prune the `populate` hint from to-one relations, as partially loading them does not require their population, we want just the FK
+      const pruneToOneRelations = (meta: EntityMetadata, fields: string[]): string[] => {
+        return fields.filter(field => {
+          if (!field.includes('.')) {
+            if (field === '*') {
+              return true;
+            }
+
+            return ![ReferenceKind.MANY_TO_ONE, ReferenceKind.ONE_TO_ONE].includes(meta.properties[field].kind);
+          }
+
+          const parts = field.split('.');
+          const key = parts.shift()!;
+
+          /* istanbul ignore next */
+          if (key === '*') {
+            return true;
+          }
+
+          const prop = meta.properties[key];
+          const ret = pruneToOneRelations(prop.targetMeta!, [parts.join('.')]);
+
+          return ret.length > 0;
+        });
+      };
+
+      options.populate = pruneToOneRelations(meta, this.buildFields(options.fields)) as any;
     }
 
     if (!options.populate) {
-      return this.entityLoader.normalizePopulate<Entity>(entityName, [], options.strategy);
+      return this.entityLoader.normalizePopulate<Entity>(entityName, [], options.strategy as LoadStrategy);
     }
 
     if (Array.isArray(options.populate)) {
@@ -1622,7 +1650,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
       }) as unknown as Populate<Entity>;
     }
 
-    const ret: PopulateOptions<Entity>[] = this.entityLoader.normalizePopulate<Entity>(entityName, options.populate as true, options.strategy);
+    const ret: PopulateOptions<Entity>[] = this.entityLoader.normalizePopulate<Entity>(entityName, options.populate as true, options.strategy as LoadStrategy);
     const invalid = ret.find(({ field }) => !this.canPopulate(entityName, field));
 
     if (invalid) {
@@ -1631,7 +1659,7 @@ export class EntityManager<D extends IDatabaseDriver = IDatabaseDriver> {
 
     return ret.map(field => {
       // force select-in strategy when populating all relations as otherwise we could cause infinite loops when self-referencing
-      field.strategy = options.populate === true ? LoadStrategy.SELECT_IN : (options.strategy ?? field.strategy);
+      field.strategy = options.populate === true ? LoadStrategy.SELECT_IN : (options.strategy ?? field.strategy) as LoadStrategy;
       return field;
     });
   }

packages/core/src/decorators/Entity.ts

@@ -31,6 +31,6 @@ export type EntityOptions<T> = {
   virtual?: boolean;
   // we need to use `em: any` here otherwise an expression would not be assignable with more narrow type like `SqlEntityManager`
   // also return type is unknown as it can be either QB instance (which we cannot type here) or array of POJOs (e.g. for mongodb)
-  expression?: string | ((em: any, where: FilterQuery<T>, options: FindOptions<T, any>) => object);
+  expression?: string | ((em: any, where: FilterQuery<T>, options: FindOptions<T, any, any>) => object);
   repository?: () => Constructor;
 };

packages/core/src/drivers/DatabaseDriver.ts

@@ -78,7 +78,7 @@ export abstract class DatabaseDriver<C extends Connection> implements IDatabaseD
   }
 
   /* istanbul ignore next */
-  async findVirtual<T extends object>(entityName: string, where: FilterQuery<T>, options: FindOptions<T, any>): Promise<EntityData<T>[]> {
+  async findVirtual<T extends object>(entityName: string, where: FilterQuery<T>, options: FindOptions<T, any, any>): Promise<EntityData<T>[]> {
     throw new Error(`Virtual entities are not supported by ${this.constructor.name} driver.`);
   }
 
@@ -91,7 +91,7 @@ export abstract class DatabaseDriver<C extends Connection> implements IDatabaseD
     throw new Error(`Aggregations are not supported by ${this.constructor.name} driver`);
   }
 
-  async loadFromPivotTable<T extends object, O extends object>(prop: EntityProperty, owners: Primary<O>[][], where?: FilterQuery<any>, orderBy?: QueryOrderMap<T>[], ctx?: Transaction, options?: FindOptions<T, any>): Promise<Dictionary<T[]>> {
+  async loadFromPivotTable<T extends object, O extends object>(prop: EntityProperty, owners: Primary<O>[][], where?: FilterQuery<any>, orderBy?: OrderDefinition<T>, ctx?: Transaction, options?: FindOptions<T, any, any>): Promise<Dictionary<T[]>> {
     throw new Error(`${this.constructor.name} does not use pivot tables`);
   }
 
@@ -313,9 +313,9 @@ export abstract class DatabaseDriver<C extends Connection> implements IDatabaseD
     });
   }
 
-  protected getPivotOrderBy<T>(prop: EntityProperty<T>, orderBy?: QueryOrderMap<T>[]): QueryOrderMap<T>[] {
+  protected getPivotOrderBy<T>(prop: EntityProperty<T>, orderBy?: OrderDefinition<T>): QueryOrderMap<T>[] {
     if (!Utils.isEmpty(orderBy)) {
-      return orderBy!;
+      return orderBy as QueryOrderMap<T>[];
     }
 
     if (!Utils.isEmpty(prop.orderBy)) {