feat(core): type safe references

EM now returns `Loaded<T, P>` instead of the entity (`T`). This type automatically
adds synchronous method `get()` that returns the entity (for references) or array
of entities (for collections).

There is also `$` property that contains the same as return value of this `get()`
method.

```typescript
const book1 = await orm.em.findOneOrFail(Book, bible, { populate: ['publisher', 'tags'] });
expect(book1.publisher!.$.name).toBe('Publisher 123');
expect(book1.tags!.$[0].name).toBe('t1');
expect(book1.tags!.$[1].name).toBe('t2');
expect(book1.tags!.$[2].name).toBe('t3');
orm.em.clear();

const books = await orm.em.find(Book, { id: bible.id }, { populate:
  { publisher: { books: { publisher: true } } },
});
expect(books[0].publisher!.$.books.$[0].publisher!.$.name).toBe('Publisher 123');

const book5 = await orm.em.findOneOrFail(Book, bible, { populate:
  { publisher: true, tags: true, perex: true },
});
expect(book5.publisher!.$.name).toBe('Publisher 123');
expect(book5.tags.$[0].name).toBe('t1');
```

Closes #214

BREAKING CHANGE:
`Reference.get()` is now available only with correct `Loaded` type hint and is used
as a sync getter for the entity, just like `unwrap()`. You can use `Reference.load(prop)`
for the original `get()` method functionality.

`em.find()` and similar methods now have two type arguments, due to TypeScript not supporting
partial type inference, it is no longer possible to specify the `T` explicitly (without also
explicitly specifying the load hint).

ROADMAP.md

@@ -9,7 +9,6 @@ discuss specifics.
 
 ## Planned features
 
-- Schema sync (allow automatic synchronization during development)
 - Collection expressions - support querying parts of collection
 - Collection pagination
 - Map collections
@@ -18,43 +17,6 @@ discuss specifics.
 - Multi tenant support ([schema per tenant](https://dzone.com/articles/spring-boot-hibernate-multitenancy-implementation))
 - Use `bigint` type natively in `BigIntType` (node 12+)
 
-## Planned changes for v4
-
-- [x] Single table inheritance ([#33](https://github.com/mikro-orm/mikro-orm/issues/33))
-- [x] Allow adding items to not initialized collections
-- [x] Use absolute path to entity file in static MetadataStorage keys (fix for 'Multiple property decorators' validation issues)
-- [x] Embedded entities (allow in-lining child entity into parent one with prefixed keys)
-- [x] Use `jsonb` as default for object columns in postgres
-- [x] Support subqueries in QB
-- [x] Nested conditions in `qb.update()` queries via subqueries (#319)
-- [x] Nested conditions in `em.remove()` via subqueries (#492)
-- [x] Use custom errors for specific cases (unique constraint violation, db not accessible, ...)
-- [x] Paginator helper or something similar ([doctrine docs](https://www.doctrine-project.org/projects/doctrine-orm/en/latest/tutorials/pagination.html))
-- [x] Add `groupBy` and `distinct` to `FindOptions` and `FindOneOptions`
-- [x] Support computed properties via `@Formula()` decorator
-- [ ] Lazy scalar properties (allow having props that won't be loaded by default, but can be populated)
-- [ ] Association scopes/filters ([hibernate docs](https://docs.jboss.org/hibernate/orm/3.6/reference/en-US/html/filters.html))
-- [ ] Support external hooks when using EntitySchema (hooks outside of entity)
-- [ ] Cache metadata only with ts-morph provider
-- [ ] Diffing entity level indexes in schema generator
-- [ ] Add custom types for blob, array, json
-- [ ] Seeds (#251)
-- [ ] Static checking of population (#214)
-- [ ] Nested object validation (#466)
-- [ ] Migrations Support for MongoDB (#295)
-
-## Planned breaking changes for v4
-
-- [x] Require node 10+
-- [x] Require TS 3.7 or newer
-- [x] Split into multiple packages (core, driver packages, TS support, SQL support, CLI)
-- [x] Remove `autoFlush` option
-- [x] Drop default value for platform `type` (currently defaults to `mongodb`)
-- [x] Remove `IdEntity/UuidEntity/MongoEntity` interfaces
-- [x] Do not define internal methods like `init` directly on the entity prototype ([#506](https://github.com/mikro-orm/mikro-orm/issues/506))
-- [x] Support multiple M:N with same properties without manually specifying `pivotTable`
-- [x] Change behaviour of property `default`, add `defaultRaw`
-
 ## Docs
 
 - Use code tabs https://docusaurus.io/docs/en/doc-markdown#language-specific-code-tabs

docs/docs/upgrading-v3-to-v4.md

@@ -154,6 +154,20 @@ await em.nativeDelete(Author, 1);
 
 > `em.removeEntity()` has been removed in favour of `em.remove()` (that now has almost the same signature).
 
+## Type safe references
+
+EM now returns `Loaded<T, P>` instead of the entity (`T`). This type automatically
+adds synchronous method `get()` that returns the entity (for references) or array
+of entities (for collections).
+
+`Reference.get()` is now available only with correct `Loaded` type hint and is used 
+as a sync getter for the entity, just like `unwrap()`. You can use `Reference.load(prop)` 
+for the original `get()` method functionality. 
+
+`em.find()` and similar methods now have two type arguments, due to TypeScript not supporting partial
+type inference, it is no longer possible to specify the `T` explicitly (without also explicitly 
+specifying the load hint).
+
 ## Custom types are now type safe
 
 Generic `Type` class has now two type arguments - the input and output types. 

packages/core/src/drivers/DatabaseDriver.ts

@@ -1,5 +1,5 @@
-import { EntityManagerType, FindOneOptions, FindOptions, IDatabaseDriver, PopulateOptions } from './IDatabaseDriver';
-import { EntityData, EntityMetadata, EntityProperty, FilterQuery, AnyEntity, Dictionary, Primary } from '../typings';
+import { EntityManagerType, FindOneOptions, FindOptions, IDatabaseDriver } from './IDatabaseDriver';
+import { EntityData, EntityMetadata, EntityProperty, FilterQuery, AnyEntity, Dictionary, Primary, PopulateOptions } from '../typings';
 import { MetadataStorage } from '../metadata';
 import { Connection, QueryResult, Transaction } from '../connections';
 import { Configuration, ConnectionOptions, Utils } from '../utils';

packages/core/src/drivers/IDatabaseDriver.ts

@@ -1,4 +1,4 @@
-import { EntityData, EntityMetadata, EntityProperty, AnyEntity, FilterQuery, Primary, Dictionary, QBFilterQuery, CollectionItem, ReferencedEntity } from '../typings';
+import { EntityData, EntityMetadata, EntityProperty, AnyEntity, FilterQuery, Primary, Dictionary, QBFilterQuery, IPrimaryKey, Populate, PopulateOptions } from '../typings';
 import { Connection, QueryResult, Transaction } from '../connections';
 import { QueryOrderMap, QueryFlag } from '../enums';
 import { Platform } from '../platforms';
@@ -76,8 +76,8 @@ export interface IDatabaseDriver<C extends Connection = Connection> {
 
 }
 
-export interface FindOptions<T> {
-  populate?: Populate<T>;
+export interface FindOptions<T, P extends Populate<T> = Populate<T>> {
+  populate?: P;
   orderBy?: QueryOrderMap;
   limit?: number;
   offset?: number;
@@ -91,11 +91,15 @@ export interface FindOptions<T> {
   filters?: Dictionary<boolean | Dictionary> | string[] | boolean;
 }
 
-export interface FindOneOptions<T> extends Omit<FindOptions<T>, 'limit' | 'offset'> {
+export interface FindOneOptions<T, P extends Populate<T> = Populate<T>> extends Omit<FindOptions<T, P>, 'limit' | 'offset'> {
   lockMode?: LockMode;
   lockVersion?: number | Date;
 }
 
+export interface FindOneOrFailOptions<T, P extends Populate<T> = Populate<T>> extends FindOneOptions<T, P> {
+  failHandler?: (entityName: string, where: Dictionary | IPrimaryKey | any) => Error;
+}
+
 export interface CountOptions<T>  {
   filters?: Dictionary<boolean | Dictionary> | string[] | boolean;
 }
@@ -107,14 +111,3 @@ export interface UpdateOptions<T>  {
 export interface DeleteOptions<T>  {
   filters?: Dictionary<boolean | Dictionary> | string[] | boolean;
 }
-
-export type PopulateChildren<T> = { [K in keyof T]?: PopulateMap<ReferencedEntity<T[K]> | CollectionItem<T[K]>> };
-export type PopulateMap<T> = boolean | LoadStrategy | PopulateChildren<T> | [LoadStrategy, PopulateChildren<T>];
-export type Populate<T> = (string | PopulateOptions<T>)[] | boolean | PopulateMap<T>;
-
-export type PopulateOptions<T> = {
-  field: string;
-  strategy?: LoadStrategy;
-  all?: boolean;
-  children?: PopulateOptions<T[keyof T]>[];
-};

packages/core/src/entity/ArrayCollection.ts

@@ -120,7 +120,7 @@ export class ArrayCollection<T extends AnyEntity<T>, O extends AnyEntity<O>> {
   /**
    * @internal
    */
-  get property(): EntityProperty {
+  get property(): EntityProperty<T> {
     if (!this._property) {
       const meta = wrap(this.owner, true).__meta;
       const field = Object.keys(meta.properties).find(k => this.owner[k] === this);

packages/core/src/entity/Collection.ts

@@ -1,4 +1,4 @@
-import { AnyEntity, Dictionary, EntityData, FilterQuery, Primary } from '../typings';
+import { AnyEntity, Dictionary, EntityData, FilterQuery, Loaded, Populate, Primary } from '../typings';
 import { ArrayCollection } from './index';
 import { ReferenceType } from './enums';
 import { Utils, ValidationError } from '../utils';
@@ -8,7 +8,7 @@ import { wrap } from './wrap';
 
 export class Collection<T extends AnyEntity<T>, O extends AnyEntity<O> = AnyEntity> extends ArrayCollection<T, O> {
 
-  private snapshot: T[] | undefined = []; // used to create a diff of the collection at commit time
+  private snapshot: T[] | undefined = []; // used to create a diff of the collection at commit time, undefined marks overridden values so we need to wipe when flushing
   private initialized = false;
   private dirty = false;
   private _populated = false;
@@ -18,6 +18,8 @@ export class Collection<T extends AnyEntity<T>, O extends AnyEntity<O> = AnyEnti
     this.initialized = !!items || initialized;
     Object.defineProperty(this, 'snapshot', { enumerable: false });
     Object.defineProperty(this, '_populated', { enumerable: false });
+    Object.defineProperty(this, '$', { value: this.items });
+    Object.defineProperty(this, 'get', { value: () => this.items });
   }
 
   /**
@@ -300,7 +302,7 @@ export class Collection<T extends AnyEntity<T>, O extends AnyEntity<O> = AnyEnti
 }
 
 export interface InitOptions<T> {
-  populate?: string[];
+  populate?: Populate<T>;
   orderBy?: QueryOrderMap;
   where?: FilterQuery<T>;
 }

packages/core/src/entity/EntityLoader.ts

@@ -1,12 +1,11 @@
-import { AnyEntity, Dictionary, EntityProperty, FilterQuery } from '../typings';
+import { AnyEntity, Dictionary, EntityProperty, FilterQuery, PopulateOptions } from '../typings';
 import { EntityManager } from '../index';
 import { LoadStrategy, ReferenceType } from './enums';
 import { Utils, ValidationError } from '../utils';
 import { Collection } from './Collection';
 import { QueryOrder, QueryOrderMap } from '../enums';
 import { Reference } from './Reference';
 import { wrap } from './wrap';
-import { PopulateOptions } from '../drivers';
 
 type Options<T extends AnyEntity<T>> = {
   where?: FilterQuery<T>;
@@ -175,7 +174,7 @@ export class EntityLoader {
     const ids = Utils.unique(children.map(e => Utils.getPrimaryKeyValues(e, wrap(e, true).__meta.primaryKeys, true)));
     const where = { [fk]: { $in: ids }, ...(options.where as Dictionary) };
 
-    return this.em.find<T>(prop.type, where, {
+    return this.em.find<T, any>(prop.type, where, {
       orderBy: options.orderBy || prop.orderBy || { [fk]: QueryOrder.ASC },
       refresh: options.refresh,
       filters: options.filters,

packages/core/src/entity/EntityRepository.ts

@@ -1,9 +1,9 @@
-import { EntityManager, FindOneOrFailOptions } from '../EntityManager';
-import { EntityData, EntityName, AnyEntity, Primary } from '../typings';
+import { EntityManager } from '../EntityManager';
+import { EntityData, EntityName, AnyEntity, Primary, Populate, Loaded } from '../typings';
 import { QueryOrderMap } from '../enums';
-import { FilterQuery, FindOneOptions, FindOptions, IdentifiedReference, Reference } from '..';
+import { FilterQuery, FindOneOptions, FindOptions, FindOneOrFailOptions, IdentifiedReference, Reference } from '..';
 
-export class EntityRepository<T> {
+export class EntityRepository<T extends AnyEntity<T>> {
 
   constructor(protected readonly em: EntityManager,
               protected readonly entityName: EntityName<T>) { }
@@ -23,34 +23,34 @@ export class EntityRepository<T> {
     this.em.persistLater(entity);
   }
 
-  async findOne(where: FilterQuery<T>, populate?: string[] | boolean, orderBy?: QueryOrderMap): Promise<T | null>;
-  async findOne(where: FilterQuery<T>, populate?: FindOneOptions<T>, orderBy?: QueryOrderMap): Promise<T | null>;
-  async findOne(where: FilterQuery<T>, populate: string[] | boolean | FindOneOptions<T> = [], orderBy?: QueryOrderMap): Promise<T | null> {
-    return this.em.findOne<T>(this.entityName, where, populate as string[], orderBy);
+  async findOne<P extends Populate<T>>(where: FilterQuery<T>, populate?: P, orderBy?: QueryOrderMap): Promise<Loaded<T, P> | null>;
+  async findOne<P extends Populate<T>>(where: FilterQuery<T>, populate?: FindOneOptions<T, P>, orderBy?: QueryOrderMap): Promise<Loaded<T, P> | null>;
+  async findOne<P extends Populate<T>>(where: FilterQuery<T>, populate?: P | FindOneOptions<T, P>, orderBy?: QueryOrderMap): Promise<Loaded<T, P> | null> {
+    return this.em.findOne<T, P>(this.entityName, where, populate as P, orderBy);
   }
 
-  async findOneOrFail(where: FilterQuery<T>, populate?: string[] | boolean, orderBy?: QueryOrderMap): Promise<T>;
-  async findOneOrFail(where: FilterQuery<T>, populate?: FindOneOrFailOptions<T>, orderBy?: QueryOrderMap): Promise<T>;
-  async findOneOrFail(where: FilterQuery<T>, populate: string[] | boolean | FindOneOrFailOptions<T> = [], orderBy?: QueryOrderMap): Promise<T> {
-    return this.em.findOneOrFail<T>(this.entityName, where, populate as string[], orderBy);
+  async findOneOrFail<P extends Populate<T>>(where: FilterQuery<T>, populate?: P, orderBy?: QueryOrderMap): Promise<Loaded<T, P>>;
+  async findOneOrFail<P extends Populate<T>>(where: FilterQuery<T>, populate?: FindOneOrFailOptions<T, P>, orderBy?: QueryOrderMap): Promise<Loaded<T, P>>;
+  async findOneOrFail<P extends Populate<T>>(where: FilterQuery<T>, populate?: P | FindOneOrFailOptions<T, P>, orderBy?: QueryOrderMap): Promise<Loaded<T, P>> {
+    return this.em.findOneOrFail<T, P>(this.entityName, where, populate as P, orderBy);
   }
 
-  async find(where: FilterQuery<T>, options?: FindOptions<T>): Promise<T[]>;
-  async find(where: FilterQuery<T>, populate?: string[] | boolean, orderBy?: QueryOrderMap, limit?: number, offset?: number): Promise<T[]>;
-  async find(where: FilterQuery<T>, populate: string[] | boolean | FindOptions<T> = [], orderBy: QueryOrderMap = {}, limit?: number, offset?: number): Promise<T[]> {
-    return this.em.find<T>(this.entityName, where as FilterQuery<T>, populate as string[], orderBy, limit, offset);
+  async find<P extends Populate<T>>(where: FilterQuery<T>, options?: FindOptions<T, P>): Promise<Loaded<T, P>[]>;
+  async find<P extends Populate<T>>(where: FilterQuery<T>, populate?: string[] | boolean, orderBy?: QueryOrderMap, limit?: number, offset?: number): Promise<Loaded<T, P>[]>;
+  async find<P extends Populate<T>>(where: FilterQuery<T>, populate: string[] | boolean | FindOptions<T, P> = [], orderBy: QueryOrderMap = {}, limit?: number, offset?: number): Promise<Loaded<T, P>[]> {
+    return this.em.find<T, P>(this.entityName, where as FilterQuery<T>, populate as P, orderBy, limit, offset);
   }
 
-  async findAndCount(where: FilterQuery<T>, options?: FindOptions<T>): Promise<[T[], number]>;
-  async findAndCount(where: FilterQuery<T>, populate?: string[] | boolean, orderBy?: QueryOrderMap, limit?: number, offset?: number): Promise<[T[], number]>;
-  async findAndCount(where: FilterQuery<T>, populate: string[] | boolean | FindOptions<T> = [], orderBy: QueryOrderMap = {}, limit?: number, offset?: number): Promise<[T[], number]> {
-    return this.em.findAndCount<T>(this.entityName, where as FilterQuery<T>, populate as string[], orderBy, limit, offset);
+  async findAndCount<P extends Populate<T>>(where: FilterQuery<T>, options?: FindOptions<T>): Promise<[Loaded<T, P>[], number]>;
+  async findAndCount<P extends Populate<T>>(where: FilterQuery<T>, populate?: string[] | boolean, orderBy?: QueryOrderMap, limit?: number, offset?: number): Promise<[Loaded<T, P>[], number]>;
+  async findAndCount<P extends Populate<T>>(where: FilterQuery<T>, populate: string[] | boolean | FindOptions<T> = [], orderBy: QueryOrderMap = {}, limit?: number, offset?: number): Promise<[Loaded<T, P>[], number]> {
+    return this.em.findAndCount<T, P>(this.entityName, where as FilterQuery<T>, populate as P, orderBy, limit, offset);
   }
 
-  async findAll(options?: FindOptions<T>): Promise<T[]>;
-  async findAll(populate?: string[] | boolean | true, orderBy?: QueryOrderMap, limit?: number, offset?: number): Promise<T[]>;
-  async findAll(populate: string[] | boolean | true | FindOptions<T> = [], orderBy?: QueryOrderMap, limit?: number, offset?: number): Promise<T[]> {
-    return this.em.find<T>(this.entityName, {}, populate as string[], orderBy, limit, offset);
+  async findAll<P extends Populate<T>>(options?: FindOptions<T>): Promise<Loaded<T, P>[]>;
+  async findAll<P extends Populate<T>>(populate?: string[] | boolean | true, orderBy?: QueryOrderMap, limit?: number, offset?: number): Promise<Loaded<T, P>[]>;
+  async findAll<P extends Populate<T>>(populate: string[] | boolean | true | FindOptions<T> = [], orderBy?: QueryOrderMap, limit?: number, offset?: number): Promise<Loaded<T, P>[]> {
+    return this.em.find<T, P>(this.entityName, {}, populate as string[], orderBy, limit, offset);
   }
 
   remove(entity: AnyEntity): EntityManager {
@@ -102,10 +102,11 @@ export class EntityRepository<T> {
     return this.em.canPopulate(this.entityName, property);
   }
 
-  async populate(entities: T, populate: string | string[] | boolean, where?: FilterQuery<T>, orderBy?: QueryOrderMap, refresh?: boolean, validate?: boolean): Promise<T>;
-  async populate(entities: T[], populate: string | string[] | boolean, where?: FilterQuery<T>, orderBy?: QueryOrderMap, refresh?: boolean, validate?: boolean): Promise<T[]>;
-  async populate(entities: T | T[], populate: string | string[] | boolean, where: FilterQuery<T> = {}, orderBy: QueryOrderMap = {}, refresh = false, validate = true): Promise<T | T[]> {
-    return this.em.populate<T>(entities as T, populate, where, orderBy, refresh, validate);
+  async populate<P extends string | keyof T | Populate<T>>(entities: T, populate: P, where?: FilterQuery<T>, orderBy?: QueryOrderMap, refresh?: boolean, validate?: boolean): Promise<Loaded<T, P>>;
+  async populate<P extends string | keyof T | Populate<T>>(entities: T[], populate: P, where?: FilterQuery<T>, orderBy?: QueryOrderMap, refresh?: boolean, validate?: boolean): Promise<Loaded<T, P>[]>;
+  async populate<P extends string | keyof T | Populate<T>>(entities: T | T[], populate: P, where?: FilterQuery<T>, orderBy?: QueryOrderMap, refresh?: boolean, validate?: boolean): Promise<Loaded<T, P> | T[]>;
+  async populate<P extends string | keyof T | Populate<T>>(entities: T | T[], populate: P, where: FilterQuery<T> = {}, orderBy: QueryOrderMap = {}, refresh = false, validate = true): Promise<Loaded<T, P> | T[]> {
+    return this.em.populate(entities, populate, where, orderBy, refresh, validate);
   }
 
   /**

packages/core/src/entity/Reference.ts

@@ -60,17 +60,18 @@ export class Reference<T> {
     return Reference.isReference<T>(ref) ? (ref as Reference<T>).unwrap() : ref;
   }
 
-  async load(): Promise<T> {
-    if (this.isInitialized()) {
-      return this.entity;
+  async load(): Promise<T>;
+  async load<K extends keyof T>(prop: K): Promise<T[K]>;
+  async load<K extends keyof T = never>(prop?: K): Promise<T | T[K]> {
+    if (!this.isInitialized()) {
+      await wrap(this.entity).init();
     }
 
-    return wrap(this.entity, true).init();
-  }
+    if (prop) {
+      return this.entity[prop];
+    }
 
-  async get<K extends keyof T>(prop: K): Promise<T[K]> {
-    await this.load();
-    return this.entity[prop];
+    return this.entity;
   }
 
   set(entity: T | IdentifiedReference<T>): void {
@@ -80,6 +81,8 @@ export class Reference<T> {
 
     this.entity = entity;
     Object.defineProperty(this, '__helper', { value: wrap(this.entity, true), writable: true });
+    Object.defineProperty(this, '$', { value: this.entity, writable: true });
+    Object.defineProperty(this, 'get', { value: () => this.entity, writable: true });
   }
 
   unwrap(): T {

packages/core/src/events/EventSubscriber.ts

@@ -1,4 +1,4 @@
-import { AnyEntity, EntityName } from '../typings';
+import { EntityName } from '../typings';
 import { EntityManager } from '../EntityManager';
 import { ChangeSet, UnitOfWork } from '../unit-of-work';
 
@@ -12,7 +12,7 @@ export interface FlushEventArgs extends Omit<EventArgs<unknown>, 'entity'> {
   uow: UnitOfWork;
 }
 
-export interface EventSubscriber<T = AnyEntity> {
+export interface EventSubscriber<T = any> {
   getSubscribedEntities?(): EntityName<T>[];
   onInit?(args: EventArgs<T>): void;
   beforeCreate?(args: EventArgs<T>): Promise<void>;

packages/core/src/index.ts

@@ -1,7 +1,7 @@
 /* istanbul ignore file */
 export {
   Constructor, Dictionary, PrimaryKeyType, Primary, IPrimaryKey, FilterQuery, IWrappedEntity, EntityName, EntityData,
-  AnyEntity, EntityProperty, EntityMetadata, QBFilterQuery,
+  AnyEntity, EntityProperty, EntityMetadata, QBFilterQuery, PopulateOptions, Populate,
 } from './typings';
 export * from './enums';
 export * from './exceptions';

packages/core/src/metadata/EntitySchema.ts

@@ -200,7 +200,7 @@ export class EntitySchema<T extends AnyEntity<T> = AnyEntity, U extends AnyEntit
     return this._meta;
   }
 
-  get name() {
+  get name(): EntityName<T>  {
     return this._meta.name;
   }
 
@@ -213,7 +213,8 @@ export class EntitySchema<T extends AnyEntity<T> = AnyEntity, U extends AnyEntit
     }
 
     if (!this._meta.class) {
-      this._meta.class = ({ [this.name]: class {} })[this.name] as Constructor<T>;
+      const name = this.name as string;
+      this._meta.class = ({ [name]: class {} })[name] as Constructor<T>;
     }
 
     this.setClass(this._meta.class);