Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of mysql generator #309

Merged

Conversation

jupp0r
Copy link
Contributor

@jupp0r jupp0r commented Feb 3, 2024

This change improves performance for generating mysql models for databases with large number of tables by eliminating an N+1 query. In my local testing for a database with about 1000 tables and 140k columns, generation time was reduced from about 1h to less than one second.

This is on MySQL 5.7 (I know....), I think those original queries were faster on MySQL 8. The change is sensible in any case.

@jupp0r jupp0r force-pushed the feature/optimize-mysql-generator-performance branch from 95f3c43 to a3ce5eb Compare February 4, 2024 00:55
@codecov-commenter
Copy link

codecov-commenter commented Feb 4, 2024

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (61841dc) 90.54% compared to head (bffec36) 90.55%.

Files Patch % Lines
generator/mysql/query_set.go 92.85% 1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #309      +/-   ##
==========================================
+ Coverage   90.54%   90.55%   +0.01%     
==========================================
  Files         124      124              
  Lines        7307     7288      -19     
==========================================
- Hits         6616     6600      -16     
+ Misses        530      528       -2     
+ Partials      161      160       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@go-jet
Copy link
Owner

go-jet commented Feb 5, 2024

Hi @jupp0r,

Whats your setup? Are you running MySQL 5.7 on a docker? Mac silicon?

generation time was reduced from about 1h to less than one second.

One minut or one second?

generator/mysql/query_set.go Outdated Show resolved Hide resolved
@jupp0r
Copy link
Contributor Author

jupp0r commented Feb 6, 2024

Hi @jupp0r,

Whats your setup? Are you running MySQL 5.7 on a docker? Mac silicon?

generation time was reduced from about 1h to less than one second.

One minut or one second?

One second. This is on an Apple M1 Max Macbook Pro running Percona 5.7.35-38 natively (the x64 version though, ie emulated using Rosetta. It's generally pretty performant compared to production on dev machines though). Each of the per-table queries was taking more than 30s for some reason. The single compound query is much faster. Might be an old mysql bug (missing index on the meta schema? lock contention? no idea why it was so slow). MySQL was saturating one core while doing those queries.

@jupp0r jupp0r force-pushed the feature/optimize-mysql-generator-performance branch from a3ce5eb to 11a4ffe Compare February 16, 2024 05:34
@jupp0r jupp0r requested a review from go-jet February 16, 2024 05:37
This change improves performance for generating mysql models
for databases with large number of tables. In my local testing
for a database with about 1000 tables and 140k columns, generation
time was reduced from about 1h to less than one second.
@jupp0r jupp0r force-pushed the feature/optimize-mysql-generator-performance branch from 11a4ffe to bffec36 Compare February 17, 2024 13:36
@jupp0r jupp0r requested a review from go-jet February 17, 2024 13:36
@go-jet
Copy link
Owner

go-jet commented Feb 19, 2024

Looks good. 👍

@go-jet go-jet merged commit 493d3a4 into go-jet:master Feb 19, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants