Fixing performance issues of RelationId.
#7318
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When we join a lot of relations we can get 1k+ records from db even it is only 10 entities. Then when relationId are loaded the query contains too many duplciates for the same ids. The solution is to check that the relationId query fetches only unique values. Closes: #5691
|
Thank you for contribution. I'm not sure I should merge this PR because last time we wanted to deprecate this file in favor of another RelationIdLoader we already have. I believe it already have such optimization. We just need to replace old usages to new usages and make necessary modifications. |
|
Hi @pleerock, thanks, I'll take a look how it works and update the case. My idea was - if you plan to deprecate / remove |
Even if it gets deprecated, in its last version with this feature, it should be in it's best shape and best performance. Therefore, I think it should get merged. |
|
Thank you for contribution! 🎉 |
|
Thanks @pleerock, happy coding! |
gogotaro
added a commit
to flowaccount/typeorm
that referenced
this pull request
Mar 17, 2021
* docs: fix small typo on package.json script example (typeorm#7408) Add missing colon in JSON property at `package.json` `"script"` example * feat: output Javascript Migrations instead of TypeScript (typeorm#7294) * docs / test: Added tests and documentation for Feature 7253 - Migrations Javascript output * Change in the test * test: Re-arranged the tests to move them to the core tests directory * tests: Adjusted Tests a bit * tests - renamed tests to follow the other functional tests naming * tests - renamed tests to follow the other functional tests naming * tests - Fixed issues with the test connections setup * tests - Removed unnecesary restore * fix: improve EntityManager.save() return type (typeorm#7391) This brings it in line with the equivalent method in Repository. * fix: resolve issue building tree entities with embeded primary column (typeorm#7416) Closes: typeorm#7415 * Adjust mongodb driver options & connect driver to support replica set (typeorm#7402) - Dupplicate buildDriverOptions for mongodb especially - Add hostReplicaSet to MongoConnectionOptions properties for collect host replica list - Adjust buildConnectionUrl to build replica set connection url * fix: performance issues of `RelationId`. (typeorm#7318) * test: relationId is too slow * perf: RelationId is too slow When we join a lot of relations we can get 1k+ records from db even it is only 10 entities. Then when relationId are loaded the query contains too many duplciates for the same ids. The solution is to check that the relationId query fetches only unique values. Closes: typeorm#5691 * fix: Array type default value should not generate SQL commands without change (typeorm#7409) * fix(1532) Array type default value should not generate SQL commands without change * Update PostgresDriver.ts * removed `arrayCast` from `normalizeDefault` since casting for default value is already removed in `PostgresQueryRunner.loadTables()` method; * removed support for function definition in `default` because function syntax suppose to support raw sql, we don't have to confuse things by applying custom modifications. * Update User.ts removed incorrect `default` definition with functions Co-authored-by: AlexMesser <dmzt08@gmail.com> * feat: add check and dryrun to migration generate (typeorm#7275) Adds support for “check” and “drynrun” modes to the migration generate command. Fixes typeorm#3037 Refs typeorm#6978 * chore: typescript version upgrade (typeorm#7422) * chore: dependencies update (typeorm#7424) * typescript version upgrade * fixing linting * fixing mongo query runner issues * fixing linting * updated all dependencies * fixes typeorm#7418 * fixes typeorm#7418 * adding missing ILike operator docs (took from next branch) * fix: mongodb connectionurl parse options (#1) * fix: mongodb connectionurl parse options - Loop every options in mongodb connection url and turn it as object to merge with connection url object before return of method "parseMongoDBConnectionUrl" - unit test of mongodb replicaset parse connectionurl of typeorm#7401 - unit test of mongodb options parse connectionurl of typeorm#7437 * fix: add semicolon by lint suggestion /home/circleci/typeorm/src/driver/DriverUtils.ts 192:39 error Missing semicolon @typescript-eslint/semi * chore: @beamdev package scope (#2) * chore: update master (typeorm#3) * fix: fixed all known enum issues (typeorm#7419) * fix typeorm#5371 * fix typeorm#6471; fix: `enumName` changes not handled; fix: `enumName` does not handle table schema; * fixed falling test; * added test for typeorm#7217 * fix typeorm#6047, typeorm#7283; * fix typeorm#5871 * added support for `enumName` in `joinColumns` (typeorm#5729) * fix typeorm#5478 * fixed falling test; updated `postgres-enum` test; * added column `array` property change detection (typeorm#5882); updated `postgres-enum` test; * fix typeorm#5275 * added validation for `enum` property (typeorm#2233) * fix typeorm#5648 * improved missing "enum" or "enumName" properties validation; * fix typeorm#4897, typeorm#6376 * lint fix; * fixed falling tests; * fixed falling tests; * removed .only * fix typeorm#6115 * refactor: improve README.md and DEVLOPER.md code examples formatting (typeorm#7436) * fix: correctly get referenceColumn value in `getEntityValueMap` (typeorm#7005) * test: add test case (typeorm#7002) * fix: correctly get referenceColumn value in `getEntityValueMap` * test: reproduction for issue typeorm#3246 (typeorm#3247) * Add reproduction for issue 3246 * Update test/github-issues/3246/issue-3246.ts Co-authored-by: Json Choi <1890mah@gmail.com> Co-authored-by: Dan Imbrogno <dan.imbrogno@gmail.com> Co-authored-by: AlexMesser <dmzt08@gmail.com> Co-authored-by: Json Choi <1890mah@gmail.com> * code refactoring in test; * added test for typeorm#2758 * feat: allow to pass the given table name as string in RelationDecorators (typeorm#7448) * feat(RelationDecorators): allow to pass the given table name as string * Update EntityMetadataBuilder.ts added parentheses; Co-authored-by: Emily Marigold Klassen <forivall@users.noreply.github.com> * feat: add option for installing package using CLI (typeorm#6889) * init cli: add options for installing package * yarg choice, add await, revert formatter changes * init flag - set default to npm Co-authored-by: AlexMesser <dmzt08@gmail.com> Co-authored-by: Henry Boisdequin <boisdequinhenry19@gmail.com> Co-authored-by: Json Choi <1890mah@gmail.com> Co-authored-by: Dan Imbrogno <41128441+danimbrogno-pml@users.noreply.github.com> Co-authored-by: Dan Imbrogno <dan.imbrogno@gmail.com> Co-authored-by: Emily Marigold Klassen <forivall@gmail.com> Co-authored-by: Emily Marigold Klassen <forivall@users.noreply.github.com> Co-authored-by: Gaurav Sharma <gtpan77@gmail.com> * chore: update master typeorm/typeorm (typeorm#5) * fix: fixed all known enum issues (typeorm#7419) * fix typeorm#5371 * fix typeorm#6471; fix: `enumName` changes not handled; fix: `enumName` does not handle table schema; * fixed falling test; * added test for typeorm#7217 * fix typeorm#6047, typeorm#7283; * fix typeorm#5871 * added support for `enumName` in `joinColumns` (typeorm#5729) * fix typeorm#5478 * fixed falling test; updated `postgres-enum` test; * added column `array` property change detection (typeorm#5882); updated `postgres-enum` test; * fix typeorm#5275 * added validation for `enum` property (typeorm#2233) * fix typeorm#5648 * improved missing "enum" or "enumName" properties validation; * fix typeorm#4897, typeorm#6376 * lint fix; * fixed falling tests; * fixed falling tests; * removed .only * fix typeorm#6115 * refactor: improve README.md and DEVLOPER.md code examples formatting (typeorm#7436) * fix: correctly get referenceColumn value in `getEntityValueMap` (typeorm#7005) * test: add test case (typeorm#7002) * fix: correctly get referenceColumn value in `getEntityValueMap` * test: reproduction for issue typeorm#3246 (typeorm#3247) * Add reproduction for issue 3246 * Update test/github-issues/3246/issue-3246.ts Co-authored-by: Json Choi <1890mah@gmail.com> Co-authored-by: Dan Imbrogno <dan.imbrogno@gmail.com> Co-authored-by: AlexMesser <dmzt08@gmail.com> Co-authored-by: Json Choi <1890mah@gmail.com> * code refactoring in test; * added test for typeorm#2758 * feat: allow to pass the given table name as string in RelationDecorators (typeorm#7448) * feat(RelationDecorators): allow to pass the given table name as string * Update EntityMetadataBuilder.ts added parentheses; Co-authored-by: Emily Marigold Klassen <forivall@users.noreply.github.com> * feat: add option for installing package using CLI (typeorm#6889) * init cli: add options for installing package * yarg choice, add await, revert formatter changes * init flag - set default to npm Co-authored-by: AlexMesser <dmzt08@gmail.com> Co-authored-by: Henry Boisdequin <boisdequinhenry19@gmail.com> Co-authored-by: Json Choi <1890mah@gmail.com> Co-authored-by: Dan Imbrogno <41128441+danimbrogno-pml@users.noreply.github.com> Co-authored-by: Dan Imbrogno <dan.imbrogno@gmail.com> Co-authored-by: Emily Marigold Klassen <forivall@gmail.com> Co-authored-by: Emily Marigold Klassen <forivall@users.noreply.github.com> Co-authored-by: Gaurav Sharma <gtpan77@gmail.com> Co-authored-by: rccursach <rccursach@gmail.com> Co-authored-by: Jorge Luis Vargas <Jorge.Vargas@albelli.com> Co-authored-by: Anthony Rosequist <arosequist@users.noreply.github.com> Co-authored-by: Tomas Zaluckij <mrtomaszal@gmail.com> Co-authored-by: MG <m@sudo.eu> Co-authored-by: Ed Mitchell <edeesis@gmail.com> Co-authored-by: AlexMesser <dmzt08@gmail.com> Co-authored-by: Christian Holm <cho@cubitech.dk> Co-authored-by: Umed Khudoiberdiev <pleerock.me@gmail.com> Co-authored-by: Henry Boisdequin <boisdequinhenry19@gmail.com> Co-authored-by: Json Choi <1890mah@gmail.com> Co-authored-by: Dan Imbrogno <41128441+danimbrogno-pml@users.noreply.github.com> Co-authored-by: Dan Imbrogno <dan.imbrogno@gmail.com> Co-authored-by: Emily Marigold Klassen <forivall@gmail.com> Co-authored-by: Emily Marigold Klassen <forivall@users.noreply.github.com> Co-authored-by: Gaurav Sharma <gtpan77@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
Fixes #5691
Fixing performance issues of
RelationId.For example, we have users and a company.
We want to select the company with relation ids.
There are 100k users, and all of them belong to the same company.
So we select
users.company,it selects 100k users and 100k companies, but these 100k companies are the same entity,
because all users belong to the same company.
Let's imagine, that the company has adminId, and we want it to be selected via
RelationId.The issue is that currently typeorm simply joins ids of entities,
so the query has 100k times
company.id = NNN or company.id = NNN or ..., whereas NNN is the same id,and because the query is very long, MySQL dies when parses it.
The changes reduce ids to be an unique array, so 100k ids will become a single id.
Pull-Request Checklist
masterbranchnpm run lintpasses with this changenpm run testpasses with this changeFixes #0000