Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RelationId is too slow #5691

Closed
satanTime opened this issue Mar 15, 2020 · 16 comments · Fixed by #7318
Closed

RelationId is too slow #5691

satanTime opened this issue Mar 15, 2020 · 16 comments · Fixed by #7318

Comments

@satanTime
Copy link
Contributor

Issue type:

[ ] question
[x] bug report
[ ] feature request
[ ] documentation issue

Database system/driver:

[ ] cordova
[ ] mongodb
[ ] mssql
[x] mysql / mariadb
[ ] oracle
[ ] postgres
[ ] cockroachdb
[ ] sqlite
[ ] sqljs
[ ] react-native
[ ] expo

TypeORM version:

[x] latest
[ ] @next
[ ] 0.x.x (or put your version here)

Steps to reproduce or a small repository showing the problem:
@RelationId() is too slow on big amount of related records (1k+ rows), see test in the PR.
@satanTime satanTime mentioned this issue Mar 15, 2020
Closed
@satanTime
Copy link
Contributor Author

Hi @pleerock, @Kononnable, could you review and provide your feedback?

@satanTime
Copy link
Contributor Author

Hi, any update on this?

@satanTime
Copy link
Contributor Author

Hi there, any update?

@satanTime
Copy link
Contributor Author

Hi @dhritzkiv, could you review?

@satanTime
Copy link
Contributor Author

Hi @pleerock, any news? I need this fix for out prod.

@dhritzkiv
Copy link
Contributor

dhritzkiv commented Apr 2, 2020 via email

@satanTime
Copy link
Contributor Author

thanks, I'll try, not sure if it helps with packages where typeorm is a dependency.

@satanTime
Copy link
Contributor Author

satanTime commented Apr 2, 2020 •
edited
Loading

Unfortunately it doesn't work because it requires a build:

ENOENT: no such file or directory, chmod '.../node_modules/typeorm/cli.js'

I could do a fork of the repo and so on, but I think the better way would be to add the solution to the main package.

@satanTime
Copy link
Contributor Author

closing based on the comment #5692

@satanTime
Copy link
Contributor Author

I would like to bring this topic back, because there is no interface for 3rd-party plugins in typeorm.

@satanTime satanTime reopened this Jan 25, 2021
@satanTime
Copy link
Contributor Author

satanTime commented Jan 25, 2021 •
edited
Loading

ping @pleerock, might you check this comment: #5692 (comment).

thank you in advance!

@satanTime satanTime mentioned this issue Jan 26, 2021
Merged
7 tasks
@satanTime
Copy link
Contributor Author

Hi @imnotjames and @pleerock, just a friendly reminder, please take a look when you have time.

Thanks.

@nebkat
Copy link
Contributor

nebkat commented Jan 30, 2021

@satanTime Unfortunately they have not been very active lately but spamming them will not help. If your issue is important enough people should upvote it and eventually that will get it attention.

@satanTime
Copy link
Contributor Author

@nebkat, if it only was like that, for example, there is another issue with enough upvotes, but the same effect. Therefore, I simply post friendly reminders time to time.

@nebkat
Copy link
Contributor

nebkat commented Jan 30, 2021

By your logic I might ask if even the issues with a large amount of upvotes are not getting attention what makes you think your one will? Pleerock is currently busy with other commitments. I say this in your own interest - if his inbox is full of hundreds of mentions he may as well have no mentions because he will just ignore everything. If you need something critical you should use your own fork for now.

@satanTime
Copy link
Contributor Author

That is exactly why I remind them time to time about the issues.

pleerock pushed a commit that referenced this issue Feb 24, 2021
@satanTime
fix: performance issues of RelationId. (#7318)
01a215a 
* test: relationId is too slow

* perf: RelationId is too slow

When we join a lot of relations we can get 1k+ records from db even it is only 10 entities. Then when relationId are loaded the query contains too many duplciates for the same ids. The solution is to check that the relationId query fetches only unique values.

Closes: #5691
gogotaro added a commit to flowaccount/typeorm that referenced this issue Mar 17, 2021
@gogotaro @rccursach
@foxxor @arosequist @Tomaszal @satanTime @edeesis @AlexMesser @holm @pleerock @henrytheswimmer @dolsup @danimbrogno-pml @danimbrogno @forivall @gtpan77
Chore/flowaccount update (#2)
5f1ebbf 
* docs: fix small typo on package.json script example (typeorm#7408)

Add missing colon in JSON property at `package.json` `"script"` example

* feat: output Javascript Migrations instead of TypeScript (typeorm#7294)

* docs / test: Added tests and documentation for Feature 7253 - Migrations Javascript output

* Change in the test

* test: Re-arranged the tests to move them to the core tests directory

* tests: Adjusted Tests a bit

* tests - renamed tests to follow the other functional tests naming

* tests - renamed tests to follow the other functional tests naming

* tests - Fixed issues with the test connections setup

* tests - Removed unnecesary restore

* fix: improve EntityManager.save() return type (typeorm#7391)

This brings it in line with the equivalent method in Repository.

* fix: resolve issue building tree entities with embeded primary column (typeorm#7416)

Closes: typeorm#7415

* Adjust mongodb driver options & connect driver to support replica set (typeorm#7402)

- Dupplicate buildDriverOptions for mongodb especially
- Add hostReplicaSet to MongoConnectionOptions properties for collect host replica list
- Adjust buildConnectionUrl to build replica set connection url

* fix: performance issues of `RelationId`. (typeorm#7318)

* test: relationId is too slow

* perf: RelationId is too slow

When we join a lot of relations we can get 1k+ records from db even it is only 10 entities. Then when relationId are loaded the query contains too many duplciates for the same ids. The solution is to check that the relationId query fetches only unique values.

Closes: typeorm#5691

* fix: Array type default value should not generate SQL commands without change (typeorm#7409)

* fix(1532) Array type default value should not generate SQL commands without change

* Update PostgresDriver.ts

* removed `arrayCast` from `normalizeDefault` since casting for default value is already removed in `PostgresQueryRunner.loadTables()` method;
* removed support for function definition in `default` because function syntax suppose to support raw sql, we don't have to confuse things by applying custom modifications.

* Update User.ts

removed incorrect `default` definition with functions

Co-authored-by: AlexMesser <dmzt08@gmail.com>

* feat: add check and dryrun to migration generate (typeorm#7275)

Adds support for “check” and “drynrun” modes to the migration generate command.

Fixes typeorm#3037
Refs typeorm#6978

* chore: typescript version upgrade (typeorm#7422)

* chore: dependencies update (typeorm#7424)

* typescript version upgrade

* fixing linting

* fixing mongo query runner issues

* fixing linting

* updated all dependencies

* fixes typeorm#7418

* fixes typeorm#7418

* adding missing ILike operator docs (took from next branch)

* fix: mongodb connectionurl parse options (#1)

* fix: mongodb connectionurl parse options

- Loop every options in mongodb connection url and turn it as object to merge with connection url object before return of method "parseMongoDBConnectionUrl"
- unit test of mongodb replicaset parse connectionurl of typeorm#7401
- unit test of mongodb options parse connectionurl of typeorm#7437

* fix: add semicolon by lint suggestion

/home/circleci/typeorm/src/driver/DriverUtils.ts
  192:39  error  Missing semicolon  @typescript-eslint/semi

* chore: @beamdev package scope (#2)

* chore: update master (typeorm#3)

* fix: fixed all known enum issues (typeorm#7419)

* fix typeorm#5371

* fix typeorm#6471;
fix: `enumName` changes not handled;
fix: `enumName` does not handle table schema;

* fixed falling test;

* added test for typeorm#7217

* fix typeorm#6047, typeorm#7283;

* fix typeorm#5871

* added support for `enumName` in `joinColumns` (typeorm#5729)

* fix typeorm#5478

* fixed falling test;
updated `postgres-enum` test;

* added column `array` property change detection (typeorm#5882);
updated `postgres-enum` test;

* fix typeorm#5275

* added validation for `enum` property (typeorm#2233)

* fix typeorm#5648

* improved missing "enum" or "enumName" properties validation;

* fix typeorm#4897, typeorm#6376

* lint fix;

* fixed falling tests;

* fixed falling tests;

* removed .only

* fix typeorm#6115

* refactor: improve README.md and DEVLOPER.md code examples formatting (typeorm#7436)

* fix: correctly get referenceColumn value in `getEntityValueMap` (typeorm#7005)

* test: add test case (typeorm#7002)

* fix: correctly get referenceColumn value in `getEntityValueMap`

* test: reproduction for issue typeorm#3246 (typeorm#3247)

* Add reproduction for issue 3246

* Update test/github-issues/3246/issue-3246.ts

Co-authored-by: Json Choi <1890mah@gmail.com>

Co-authored-by: Dan Imbrogno <dan.imbrogno@gmail.com>
Co-authored-by: AlexMesser <dmzt08@gmail.com>
Co-authored-by: Json Choi <1890mah@gmail.com>

* code refactoring in test;

* added test for typeorm#2758

* feat: allow to pass the given table name as string in RelationDecorators (typeorm#7448)

* feat(RelationDecorators): allow to pass the given table name as string

* Update EntityMetadataBuilder.ts

added parentheses;

Co-authored-by: Emily Marigold Klassen <forivall@users.noreply.github.com>

* feat: add option for installing package using CLI (typeorm#6889)

* init cli: add options for installing package

* yarg choice, add await, revert formatter changes

* init flag - set default to npm

Co-authored-by: AlexMesser <dmzt08@gmail.com>
Co-authored-by: Henry Boisdequin <boisdequinhenry19@gmail.com>
Co-authored-by: Json Choi <1890mah@gmail.com>
Co-authored-by: Dan Imbrogno <41128441+danimbrogno-pml@users.noreply.github.com>
Co-authored-by: Dan Imbrogno <dan.imbrogno@gmail.com>
Co-authored-by: Emily Marigold Klassen <forivall@gmail.com>
Co-authored-by: Emily Marigold Klassen <forivall@users.noreply.github.com>
Co-authored-by: Gaurav Sharma <gtpan77@gmail.com>

* chore: update master typeorm/typeorm (typeorm#5)

* fix: fixed all known enum issues (typeorm#7419)

* fix typeorm#5371

* fix typeorm#6471;
fix: `enumName` changes not handled;
fix: `enumName` does not handle table schema;

* fixed falling test;

* added test for typeorm#7217

* fix typeorm#6047, typeorm#7283;

* fix typeorm#5871

* added support for `enumName` in `joinColumns` (typeorm#5729)

* fix typeorm#5478

* fixed falling test;
updated `postgres-enum` test;

* added column `array` property change detection (typeorm#5882);
updated `postgres-enum` test;

* fix typeorm#5275

* added validation for `enum` property (typeorm#2233)

* fix typeorm#5648

* improved missing "enum" or "enumName" properties validation;

* fix typeorm#4897, typeorm#6376

* lint fix;

* fixed falling tests;

* fixed falling tests;

* removed .only

* fix typeorm#6115

* refactor: improve README.md and DEVLOPER.md code examples formatting (typeorm#7436)

* fix: correctly get referenceColumn value in `getEntityValueMap` (typeorm#7005)

* test: add test case (typeorm#7002)

* fix: correctly get referenceColumn value in `getEntityValueMap`

* test: reproduction for issue typeorm#3246 (typeorm#3247)

* Add reproduction for issue 3246

* Update test/github-issues/3246/issue-3246.ts

Co-authored-by: Json Choi <1890mah@gmail.com>

Co-authored-by: Dan Imbrogno <dan.imbrogno@gmail.com>
Co-authored-by: AlexMesser <dmzt08@gmail.com>
Co-authored-by: Json Choi <1890mah@gmail.com>

* code refactoring in test;

* added test for typeorm#2758

* feat: allow to pass the given table name as string in RelationDecorators (typeorm#7448)

* feat(RelationDecorators): allow to pass the given table name as string

* Update EntityMetadataBuilder.ts

added parentheses;

Co-authored-by: Emily Marigold Klassen <forivall@users.noreply.github.com>

* feat: add option for installing package using CLI (typeorm#6889)

* init cli: add options for installing package

* yarg choice, add await, revert formatter changes

* init flag - set default to npm

Co-authored-by: AlexMesser <dmzt08@gmail.com>
Co-authored-by: Henry Boisdequin <boisdequinhenry19@gmail.com>
Co-authored-by: Json Choi <1890mah@gmail.com>
Co-authored-by: Dan Imbrogno <41128441+danimbrogno-pml@users.noreply.github.com>
Co-authored-by: Dan Imbrogno <dan.imbrogno@gmail.com>
Co-authored-by: Emily Marigold Klassen <forivall@gmail.com>
Co-authored-by: Emily Marigold Klassen <forivall@users.noreply.github.com>
Co-authored-by: Gaurav Sharma <gtpan77@gmail.com>

Co-authored-by: rccursach <rccursach@gmail.com>
Co-authored-by: Jorge Luis Vargas <Jorge.Vargas@albelli.com>
Co-authored-by: Anthony Rosequist <arosequist@users.noreply.github.com>
Co-authored-by: Tomas Zaluckij <mrtomaszal@gmail.com>
Co-authored-by: MG <m@sudo.eu>
Co-authored-by: Ed Mitchell <edeesis@gmail.com>
Co-authored-by: AlexMesser <dmzt08@gmail.com>
Co-authored-by: Christian Holm <cho@cubitech.dk>
Co-authored-by: Umed Khudoiberdiev <pleerock.me@gmail.com>
Co-authored-by: Henry Boisdequin <boisdequinhenry19@gmail.com>
Co-authored-by: Json Choi <1890mah@gmail.com>
Co-authored-by: Dan Imbrogno <41128441+danimbrogno-pml@users.noreply.github.com>
Co-authored-by: Dan Imbrogno <dan.imbrogno@gmail.com>
Co-authored-by: Emily Marigold Klassen <forivall@gmail.com>
Co-authored-by: Emily Marigold Klassen <forivall@users.noreply.github.com>
Co-authored-by: Gaurav Sharma <gtpan77@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixing performance issues of RelationId.
perf: RelationId is too slow
3 participants
@nebkat @satanTime @dhritzkiv