MikroORM v5

[B4nan]

Progress

• Seeder package (feat(seeder): add seeder package #929)
• Improved schema diffing (should address most of the currently known issues) (feat(schema): rework schema diffing #1641)
  • Better mapping - adds mapped types to abstract most of the column types (e.g. { type: t.decimal })
  • FK diffing
  • Proper index diffing (before we compared just names)
  • Custom index expressions
  • Comment diffing
  • Column length diffing (e.g. numeric(10,2) or varchar(100))
  • Changing PK types
  • Schena/namespace diffing (posgtres only)
• Support FKs in sqlite (including not null FKs)
• Use AsyncLocalStorage in the RequestContext helper
• ESM support via gen-esm-wrapper (ECMAScript Modules support #1010)
• Support index hints with the QueryBuilder (Support index hints with the QueryBuilder #1663)
• Strict typings for em.create() (Strict typing for em.create #1456)
• Strict toObject(), toJSON() and toPOJO() return types
• Add more methods to naming strategy (index/unique/pk names, property names for entity generator)
• Allow column names with spaces
• Support partial loading with joined strategy
• Add support for more lock modes (skip locked/nowait) (Postgres SKIP LOCKED #1786)
• Allow setting transaction isolation level
• Snapshotting of current db state in migrations (feat(migrations): use snapshots for generating diffs in new migrations #1815)
• Allow using short lived tokens in config (Support temporary password function #1818)
• Support entities inside embeddables (Allow using entities in embeddables #1932)
• Rework deep assigning and enable it by default (Rework deep assigning of entities and collection and enable it by default #1811)
• Strict typings for populate array (dot notation and string literal types)
• Strict typings for fields array (dot notation and string literal types)
• Automatic down migrations
• Strictly typed FindOptions.orderBy, allowing arrays of objects too (Order of multiple nested orderBy params #2010)
• Support for multiple schemas (including UoW) (PostgreSQL schemas for multitenancy #2074)
• Validate usage of forked EM (disallow using identity map on the global instance) (feat(core): validate usage of global context #2381)
• Improve aliasing of tables (e0/1/2/3...) and make it configurable (feat(core): allow configuring aliasing naming strategy #2419)
• Upgrade mongodb driver to v4 (feat(mongo): upgrade node-mongodb to v4 #2425)
• Polymorphic embeddables ([Feature] Polymorphic embedded entities #1165, feat(core): add support for polymorphic embeddables #2426)
• Allow awaiting the QueryBuilder instance (feat(query-builder): allow awaiting the QueryBuilder instance #2446)
• Allow providing custom Logger instance (feat(core): allow providing custom Logger instance #2443)
• Support populating lazy scalar properties via em.populate() (EntityManager.populate ignores lazy scalar properties. #1479)
• Support flushing via Promise.all() (Support running EntityManager inside Promise.all #2412)
• Support auto flush like in hibernate (Support "Auto Flush" #2359, feat(core): implement auto-flush mode #2491)

Breaking changes

• Require TS 4.1+ to allow using template literals (maybe even 4.2 it if makes sense)
• Require Node v14+
• Some methods are now strictly typed (em.create(), em.assign(), qb.insert(), qb.update(), e.toObject(), e.toJSON(), ...)
• FindOptions.orderBy/fields/ are now strictly typed
• Remove additional find/findOne/... parameters (populate/orderBy/limit/offset) in favour of FindOptions
• Use options parameter in em.fork()
• Remove @Repository decorator (use @Entity({ customRepository: () => CustomRepository }) instead)
• Enable populateAfterFlush by default
• migrations.pattern is removed in favour of migrations.glob (chore: upgrade umzug to 3.0 #2556)

https://mikro-orm.io/docs/next/upgrading-v4-to-v5/
mikro-orm/nestjs-realworld-example-app#36

[JanJakes]

@B4nan Awesome!

One point, maybe kind of related to schema diffing, could be improving naming strategy capabilities. If I'm not mistaken, there is currently no way of defining naming strategies for PKs, indexes, FKs, and others. (TypeORM for instance has a bit more powerful naming strategies.)

Another (maybe a little bit wild) idea is supporting Prisma schema – not sure if doable or reasonable in any way, but it is an interesting way of defining DB schema – it may be a feature that brings some traction, and it may allow using Prisma migrations for people who want it, etc.

One more thing on my mind – with the current CLI (and migrations) it is hard to use dynamic configuration provided by some kind of configuration services (for instance fetching a config service from DI and passing that as a config to migrations). You either end up duplicating the config data in mikro-orm.config.ts or using migrations programmatically in your own CLI commands. Maybe just simply if mikro-orm.config.ts could export a function returning a promise with the config, it would probably solve the issue.

And last idea (I hope, haha) is adding the possibility of locking migrations table exclusively when running migrations. It will prevent devs from ever starting multiple migration runs in parallel – this is a common mistake in Docker environments when devs use either k8s init containers or run the migration command on container start. I fix similar to the following one prevents the error (and forces all other containers to wait for their turn):

await this.orm.em.transactional(async (em) => {
  await em.execute('LOCK TABLE migrations IN EXCLUSIVE MODE');
  await this.orm
    .getMigrator<Migrator>()
    .up({ transaction: em.getTransactionContext() });
});

(This is for Postgres and of course this may not be doable for all drivers.)

Well, those are just a few ideas, let me know what you think :)
And thanks for your amazing work!

[B4nan]

One point, maybe kind of related to schema diffing, could be improving naming strategy capabilities. If I'm not mistaken, there is currently no way of defining naming strategies for PKs, indexes, FKs, and others. (TypeORM for instance has a bit more powerful naming strategies.)

Sure, good idea. Currently it lives in the schema helper, so it is possible to override with a custom driver only.

Another (maybe a little bit wild) idea is supporting Prisma schema – not sure if doable or reasonable in any way, but it is an interesting way of defining DB schema – it may be a feature that brings some traction, and it may allow using Prisma migrations for people who want it, etc.

Sounds a bit out of scope, on the other hand entity definition is already abstracted so it could be just about having another metadata provider for this.

One more thing on my mind – with the current CLI (and migrations) it is hard to use dynamic configuration provided by some kind of configuration services (for instance fetching a config service from DI and passing that as a config to migrations). You either end up duplicating the config data in mikro-orm.config.ts or using migrations programmatically in your own CLI commands. Maybe just simply if mikro-orm.config.ts could export a function returning a promise with the config, it would probably solve the issue.

It is already possible to export a promise from the ORM config.

And last idea (I hope, haha) is adding the possibility of locking migrations table exclusively when running migrations. It will prevent devs from ever starting multiple migration runs in parallel – this is a common mistake in Docker environments when devs use either k8s init containers or run the migration command on container start. I fix similar to the following one prevents the error (and forces or other containers to wait for their turn):

Agreed, if this single query is enough for postgres, we can definitely do it at least for postgres. Would be great if users of other drivers could come up with similar solutions for those, I'd expect at least mysql to be capable of this too.

[JanJakes]

It is already possible to export a promise from the ORM config.

Hm, I thought I tried that, maybe not. OK, will give it a shot!

Agreed, if this single query is enough for postgres, we can definitely do it at least for postgres. Would be great if users of other drivers could come up with similar solutions for those, I'd expect at least mysql to be capable of this too.

Yep, this works for me, exclusive lock indeed makes any other queries to the table wait. MySQL doesn't support transactional DDL but I guess LOCK TABLES migrations WRITE could probably do the same trick (I can try it out when I have some time).

[B4nan]

The config promise support was added in 4.5 so its very recent addition: #1495

Not sure if we need to support also returning async callback, as this just awaits the returned config...

[JanJakes]

The config promise support was added in 4.5 so its very recent addition: #1495

Not sure if we need to support also returning async callback, as this just awaits the returned config...

I see, perfect! No need for a callback, I think, this should cover pretty much any case and it's even simpler.

[TheNoim]

This is my wish list for micro orm v5:

• custom indexes (define the sql of an index). Use case: Improved fulltext search
• generated columns in postgres / real virtual columns in mysql. Use case: again improved fulltext search. I made a pr for typeorm and I wish micro orm would support it, too.
• typeorm -> micro orm migration support in some way

[rubiin]

One of the things that I would like to add is a named migration file similar to typeorm. Currently afaik , the file names are generated implicitly

[merceyz]

Since MikroORM now supports batched updates, inserts, and deletes it would be great if it would support batched selects (#266) as well, even if it was just for simple SELECT * FROM X WHERE PrimaryKey = Y queries. It probably wouldn't be that useful for REST APIs but would be really powerful for GraphQL APIs.

[yayvery]

Since MikroORM now supports batched updates, inserts, and deletes it would be great if it would support batched selects (#266) as well, even if it was just for simple SELECT * FROM X WHERE PrimaryKey = Y queries. It probably wouldn't be that useful for REST APIs but would be really powerful for GraphQL APIs.

This would be incredible. Right now I have 20 DataLoaders, being able to eliminate these would be awesome!

[ikdi]

As for me it would be great to have Oracle DB support.

[B4nan]

custom indexes (define the sql of an index). Use case: Improved fulltext search

Implemented as @Index({ expression: 'alter table book add index custom_index_expr(title)' }).

typeorm -> micro orm migration support in some way

I said this many times. I never really used typeorm, so can't provide much info about this. Better ask someone else who does. Also as the internals are very different (typeorm is kind of a QB when compared to MikroORM, as there is no UoW/in-memory state management), automatic migrations would never be really performant. Best what I can offer is this commit where I migrated the typeorm version to MikroORM, but it is a bit dated.

generated columns in postgres / real virtual columns in mysql

Blocked by knex not supporting it. Probably possible to get around, but I am tired of hacking around how knex works. Maybe I will give it a try and remove it as a whole from schema diffing, as the code gets bloated because of this a lot, and I already had to implement better part of things myself anyway. But I'd still like to leverage knex for the new drivers to come before I remove it.

You could probably use @Formula() to have similar experience if the issue is about selecting dynamic things, but it is somehow limited.

One of the things that I would like to add is a named migration file similar to typeorm. Currently afaik, the file names are generated implicitly

The filenames are generated based on configurable callback, and if you adjust the pattern option, you can have the timestamps on the beginning, and use pretty much any file name after that. I don't see a need to change anything about how that works. We might add a recipe for this.

Since MikroORM now supports batched updates, inserts, and deletes it would be great if it would support batched selects (#266) as well, even if it was just for simple SELECT * FROM X WHERE PrimaryKey = Y queries. It probably wouldn't be that useful for REST APIs but would be really powerful for GraphQL APIs.

I was looking into this finally, and after some time playing with this, I finally see how it works. I must say that it feels funny now, all those "N+1 safety" claims, as the dataloader approach is valid only in very specific constraints (e.g. calling Promise.all() on the top level for methods that use the data loader directly). I can see how this is valid for GQL, but even there it feels funny to claim something about N+1 safety when its enough to load something in a for loop while awaiting it (for me the most common way to introduce N+1), and boom, we have the N+1 problem, dataloader or not. I can't imagine how it could be usefull for other than GQL apps...

Anyway, as mentioned, I do understand how this would be helpful for GQL users, so will try to make it work at least for the reference/collection loading. But not really a priority for me.

[rubiin]

recipe for named migration file will be great

[merceyz]

I can't imagine how it could be usefull for other than GQL apps...

It's actually really useful for batch processing of entities as well, where you fetch a batch of entities then based on values in each entity fetch different kinds of entities

[B4nan]

Can you provide some code, or a test case ideally? Maybe I just wasn't able to make it work the way it should. All the examples I came up with felt much better written with populate hints so I haven't seen any real benefits when used manually (again, for GQL it's a different story). Only way I was able to make the dataloader to batch things was via Promise.all() for the first level, if you want to benefit on other levels, it would have to be again in a single tick, so it can't be from inside a list of promises, right?