Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance issue, result set processing #1672

Closed
agreatfool opened this issue Feb 28, 2018 · 5 comments
Closed

Performance issue, result set processing #1672

agreatfool opened this issue Feb 28, 2018 · 5 comments

Comments

@agreatfool
Copy link

agreatfool commented Feb 28, 2018 •
edited
Loading

Issue type:

[x] question
[ ] bug report
[ ] feature request
[ ] documentation issue

Database system/driver:

[ ] cordova
[ ] mongodb
[ ] mssql
[x] mysql / mariadb
[ ] oracle
[ ] postgres
[ ] sqlite
[ ] sqljs
[ ] websql

TypeORM version:

[ ] latest
[ ] @next
[x] 0.1.13 (or put your version here)

Complete codes can be found here: gist

Simple mysql search:

$ tsc src/app.js
$ node --prof build/app.js
DB Connected, isConnected: true
executing query:  SELECT `TestEntity`.`id` AS `TestEntity_id`, `TestEntity`.`column_b` AS `TestEntity_column_b`, `TestEntity`.`column_c` AS `TestEntity_column_c`, `TestEntity`.`column_d` AS `TestEntity_column_d`, `TestEntity`.`column_e` AS `TestEntity_column_e`, `TestEntity`.`column_f` AS `TestEntity_column_f`, `TestEntity`.`column_g` AS `TestEntity_column_g`, `TestEntity`.`column_h` AS `TestEntity_column_h` FROM `test_entity` `TestEntity` WHERE column_b = 1 AND column_d = 1
Ms consumed: 25137
# 25 seconds consumed

Run the same sql in mysql directly:

$ mysql -h x.x.x.x -u xxx -p 
mysql> use test_entity;
Database changed
mysql> select count(*) from test_entity;
+----------+
| count(*) |
+----------+
|    47474 |
+----------+
1 row in set (0.02 sec)
mysql > SELECT `TestEntity`.`id` AS `TestEntity_id`, `TestEntity`.`column_b` AS `TestEntity_column_b`, `TestEntity`.`column_c` AS `TestEntity_column_c`, `TestEntity`.`column_d` AS `TestEntity_column_d`, `TestEntity`.`column_e` AS `TestEntity_column_e`, `TestEntity`.`column_f` AS `TestEntity_column_f`, `TestEntity`.`column_g` AS `TestEntity_column_g`, `TestEntity`.`column_h` AS `TestEntity_column_h` FROM `test_entity` `TestEntity` WHERE column_b = 1 AND column_d = 1;
...
47474 rows in set (0.20 sec)
# 0.2 seconds consumed

Time consumption is 25 : 0.2. Seems not good.

Result of node --prof build/app.js & node --prof-process isolate-0x102802400-v8.log:

Statistical profiling result from isolate-0x102802400-v8.log, (21100 ticks, 421 unaccounted, 0 excluded).

 [Shared libraries]:
   ticks  total  nonlib   name
     52    0.2%          /usr/lib/system/libsystem_kernel.dylib
     ...

 [JavaScript]:
   ticks  total  nonlib   name
   6250   29.6%   29.8%  LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:43:54
   4286   20.3%   20.4%  Builtin: ArgumentsAdaptorTrampoline
   3567   16.9%   17.0%  Builtin: StringEqual
   3371   16.0%   16.1%  LazyCompile: *InnerArrayFind native array.js:843:24
    897    4.3%    4.3%  Builtin: CallFunction_ReceiverIsAny
    470    2.2%    2.2%  Builtin: Call_ReceiverIsAny
     25    0.1%    0.1%  LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
     ...

 [C++]:
   ticks  total  nonlib   name
    138    0.7%    0.7%  t node::(anonymous namespace)::ContextifyScript::New(v8::FunctionCallbackInfo<v8::Value> const&)
    137    0.6%    0.7%  t v8::internal::Object* v8::internal::VisitWeakList<v8::internal::JSFunction>(v8::internal::Heap*, v8::internal::Object*, v8::internal::WeakObjectRetainer*)
    ...

 [Summary]:
   ticks  total  nonlib   name
  19213   91.1%   91.7%  JavaScript
   1326    6.3%    6.3%  C++
    552    2.6%    2.6%  GC
    140    0.7%          Shared libraries
    421    2.0%          Unaccounted

 [C++ entry points]:
   ticks    cpp   total   name
    344   50.2%    1.6%  T v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*)
    103   15.0%    0.5%  T v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*)
     ...

 [Bottom up (heavy) profile]:
  Note: percentage shows a share of a particular caller in the total
  amount of its parent calls.
  Callers occupying less than 1.0% are not shown.

   ticks parent  name
   6250   29.6%  LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:43:54
   6250  100.0%    LazyCompile: *InnerArrayFind native array.js:843:24
   5951   95.2%      LazyCompile: *find native array.js:855:19
   5685   95.5%        LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
   5685  100.0%          Builtin: ArrayForEach
   5685  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
    266    4.5%        Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    266  100.0%          Builtin: ArrayForEach
    266  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
    299    4.8%      Function: ~find native array.js:855:19
    261   87.3%        LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    261  100.0%          Builtin: ArrayForEach
    261  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
     38   12.7%        Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
     38  100.0%          Builtin: ArrayForEach
     38  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65

   4286   20.3%  Builtin: ArgumentsAdaptorTrampoline
   3736   87.2%    LazyCompile: *InnerArrayFind native array.js:843:24
   3600   96.4%      LazyCompile: *find native array.js:855:19
   3445   95.7%        LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
   3445  100.0%          Builtin: ArrayForEach
   3445  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
    153    4.3%        Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    153  100.0%          Builtin: ArrayForEach
    153  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
    136    3.6%      Function: ~find native array.js:855:19
    117   86.0%        LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    117  100.0%          Builtin: ArrayForEach
    117  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
     19   14.0%        Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
     19  100.0%          Builtin: ArrayForEach
     19  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
    524   12.2%    LazyCompile: *find native array.js:855:19
    498   95.0%      LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    498  100.0%        Builtin: ArrayForEach
    498  100.0%          Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
    498  100.0%            Function: ~RawSqlResultsToEntityTransformer.transform /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:25:69
     26    5.0%      Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
     26  100.0%        Builtin: ArrayForEach
     26  100.0%          Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
     26  100.0%            Function: ~RawSqlResultsToEntityTransformer.transform /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:25:69

   3567   16.9%  Builtin: StringEqual
   3567  100.0%    LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:43:54
   3567  100.0%      LazyCompile: *InnerArrayFind native array.js:843:24
   3441   96.5%        LazyCompile: *find native array.js:855:19
   3278   95.3%          LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
   3278  100.0%            Builtin: ArrayForEach
    163    4.7%          Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    163  100.0%            Builtin: ArrayForEach
    126    3.5%        Function: ~find native array.js:855:19
    114   90.5%          LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    114  100.0%            Builtin: ArrayForEach
     12    9.5%          Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
     12  100.0%            Builtin: ArrayForEach

   3371   16.0%  LazyCompile: *InnerArrayFind native array.js:843:24
   3250   96.4%    LazyCompile: *find native array.js:855:19
   3095   95.2%      LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
   3095  100.0%        Builtin: ArrayForEach
   3095  100.0%          Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
   3095  100.0%            Function: ~RawSqlResultsToEntityTransformer.transform /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:25:69
    149    4.6%      Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    149  100.0%        Builtin: ArrayForEach
    149  100.0%          Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
    149  100.0%            Function: ~RawSqlResultsToEntityTransformer.transform /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:25:69
    121    3.6%    Function: ~find native array.js:855:19
    111   91.7%      LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    111  100.0%        Builtin: ArrayForEach
    111  100.0%          Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
    111  100.0%            Function: ~RawSqlResultsToEntityTransformer.transform /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:25:69
     10    8.3%      Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
     10  100.0%        Builtin: ArrayForEach
     10  100.0%          Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
     10  100.0%            Function: ~RawSqlResultsToEntityTransformer.transform /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:25:69

    897    4.3%  Builtin: CallFunction_ReceiverIsAny
    887   98.9%    LazyCompile: *InnerArrayFind native array.js:843:24
    846   95.4%      LazyCompile: *find native array.js:855:19
    811   95.9%        LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    811  100.0%          Builtin: ArrayForEach
    811  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
     35    4.1%        Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
     35  100.0%          Builtin: ArrayForEach
     35  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
     41    4.6%      Function: ~find native array.js:855:19
     35   85.4%        LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
     35  100.0%          Builtin: ArrayForEach
     35  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
      6   14.6%        Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
      6  100.0%          Builtin: ArrayForEach
      6  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65

    470    2.2%  Builtin: Call_ReceiverIsAny
    469   99.8%    LazyCompile: *InnerArrayFind native array.js:843:24
    452   96.4%      LazyCompile: *find native array.js:855:19
    441   97.6%        LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
    441  100.0%          Builtin: ArrayForEach
    441  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
     11    2.4%        Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
     11  100.0%          Builtin: ArrayForEach
     11  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
     17    3.6%      Function: ~find native array.js:855:19
     15   88.2%        LazyCompile: *<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
     15  100.0%          Builtin: ArrayForEach
     15  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65
      2   11.8%        Function: ~<anonymous> /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:39:37
      2  100.0%          Builtin: ArrayForEach
      2  100.0%            Function: ~RawSqlResultsToEntityTransformer.group /Users/Jonathan/Downloads/sasdnDatabaseTest/node_modules/typeorm/query-builder/transformer/RawSqlResultsToEntityTransformer.js:37:65

    421    2.0%  UNKNOWN

Is there anything I made wrong? Please help.
@agreatfool agreatfool changed the title Performance issue with simple mysql find Performance issue, simple mysql search Feb 28, 2018
@agreatfool agreatfool changed the title Performance issue, simple mysql search Performance issue, result set processing Feb 28, 2018
@agreatfool
Copy link
Author

agreatfool commented Feb 28, 2018 •
edited
Loading

I just thought a bit more, this issue comes from the size of the result set from mysql. As the sample provided above, the result set is 47474 . Time consumption is 25 seconds.

And I tried several different size of result set:

rows typeorm/sec mysql/sec
1 0.031 0.00
1000 0.085 0.00
2000 0.134 0.01
5000 0.399 0.02
10000 1.233 0.06
20000 5.062 0.14
30000 11.651 0.14
47474 25.214 0.24

Seems it's not good to handle queries with big size of result using typeorm. What do you guys suggest to do with this use case? Is it possible to enhance the performance of this case?

@pleerock
Copy link
Member

thanks for the report. I'll try to fix this issue quickly and will return back to you with results.

pleerock pushed a commit that referenced this issue Feb 28, 2018
issue #1672 - hydration performance issue
e118a8c
pleerock pushed a commit that referenced this issue Feb 28, 2018
fixing #1672
1ddf9f8
@pleerock
Copy link
Member

pleerock commented Feb 28, 2018 •
edited
Loading

Optimized and released in 0.1.14

Results for 100000 rows hydration on my local machine:

select using raw sql query (node-mysql): 197.424ms // nothing can be faster then this
select using orm hydrator: 579.445ms // 2, ~2.5 times slower, acceptable

There is still room for optimization but all other places aren't as serious as this one fixed.

@agreatfool
Copy link
Author

That's awesome. Thank you for your work!

@pleerock
Copy link
Member

thanks for the report!

pleerock pushed a commit that referenced this issue Mar 9, 2018
Merge branch 'master' into next
3e4098c 
* master:
  skipping test for now to make CI working
  version bump
  fixed bug with partial selections via find options
  version bump
  fixed issue with find options accepting keys as expressions
  Update parameter name to stay consistent
  Update mongodb.md
  add unit test for issue 1685
  Feature to provide custom migration table name
  add tests for issue
  fix connection url parsing
  fixing #1672
  Fix #1656
  check if entity file already exists

# Conflicts:
#	CHANGELOG.md
#	package.json
#	src/decorator/transaction/Transaction.ts
#	src/migration/MigrationExecutor.ts
#	src/query-builder/QueryBuilder.ts
#	src/query-builder/transformer/RawSqlResultsToEntityTransformer.ts
@hielkehoeve hielkehoeve mentioned this issue Jun 28, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@agreatfool @pleerock