-
Notifications
You must be signed in to change notification settings - Fork 307
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Schema loading performance regression after update from rails 5.1 to rails 5.2 #1720
Comments
I've now been able to isolate this issue with a script. It shows even more drastic numbers than the example above. To use this script
Results: Rails 5.1 / Adapter 1.8.2
Total execution time: 42.108s Log Results: Rails 5.2 / Adapter 5.2.2
Total execution time: 201.29s Log ConclusionNot only loading the model classes (that leads to structural queries) but also the migration takes way longer with Rails 5.2 than with Rails 5.1. This leads to extremely poor performance in our Rails 5.2 applications. The total execution time is 4.7 times slower in 5.2 than in 5.1. Thank you a lot for having a look at this 👍 |
To rule out an issue with Rails that is independent of the oracle enhanced adapter, I've also created a script for MySQL. Results: Rails 5.1
Total execution time: 14.170s Results: Rails 5.2
Total execution time: 17.014s Conclusion
|
Thanks for providing test cases. Would you upload the entire log files of I'd like to know which statements taking time. |
Thanks for your answer @yahonda. Sure, I've added links to the respective log files above, right after the total execution time. The queries look very similar to me, i.e.:
But for some reason, 5.2 is 650x slower for the same query. Is there maybe a general problem that affects performance in 5.2? |
It is likely due to execution plan change for this query oracle-enhanced/lib/active_record/connection_adapters/oracle_enhanced_adapter.rb Lines 557 to 565 in fb1cbbf
Would you change each change to see if you see performance improvement.
SQL> exec dbms_stats.gather_dictionary_stats()
Background: Oracle enhanced adapter 5.2.0 #1498 introduces bind values However, to address #1678 Oracle enhanced adapter 5.2.1 reverted some of changes made #1498 to use sql literals. The query for primary key is one of them. Thank you. |
Thanks for your answer. Here the results:
Edit: Now I've run it again and 5.2 with both "settings" now takes 85s again. |
Thanks for testing. I'd like to request you to upload the log files for these tests.
Thank you. |
Sure, I'll upload it as soon as possible. Wouldn't it be easier for you running the script yourself to avoid constant back and forth communication? I'm more than happy to run it for you though. |
Thanks. Actually, I did execute your script file before but unable to reproduce the slowness. |
Ah ok. Sure, I'll be getting back to you asap. |
Sorry for the delay. I've now performed the following sequence of actions:
So the peculiar thing is that the run after turning off cursor sharing, but with cursor sharing, is very slow again. Does this help? Thanks for looking into it. |
We are also experiencing performance problems now when upgrading to 5.2. We found that queries to Oracle data dictionary took at least 10 times more when using default optimizer mode compared to RULE based optimizer. Maybe you should find out if those dictionary queries are the problem here also. I changed all select queries accessing data dictionary with Before doing this I just added I do not know if this approach is good or not. More information about optimizer hints https://docs.oracle.com/cd/B10500_01/server.920/a96533/hintsref.htm#4991 At least that deprecation warning worries me. |
Thanks for the update and useful information. Let me create a branch to add rule hints for some of the queries for Oracle database dictionary to see if this issue resolves. As far as I remember rule hint has not been "supported" since 10g but there is a similar issue reported dbeaver/dbeaver#360 then I hope it would help ours. |
I am testing this with Oracle Database 12c Standard Edition Release 12.1.0.2.0 - 64bit Production |
I was able to increase db:migration time a lot with https://github.com/jlahtinen/oracle-enhanced/tree/optimize_schema_statement I used rule optimizer also, but left outer join increased performance from 25s to 1.5s in that modified query. It also seems like this query is used multiple times in 5.2 branch and not in 5.1 branch. @yahonda maybe you are able to check if this is done correctly and merge it to your branches |
Are there any news on this subject? We'd like to upgrade to Rails 5.2 but this issue is standing in our way. Thank you for your work! |
right now using @jlahtinen's fork is the only way for us to even be able to use |
We are experiencing this performance issue as well. Any news? Anything you'd like us to test/try? |
This causing us headaches too, so another volunteer here if there is anything that can be helped with. |
@joshpencheon you could verify if 51bc17b is a working fix |
@jlahtinen I've been testing that, thank you - it does appear to help significantly in our case - a I have also experimented with including an additional |
@jlahtinen My colleague who is working on this is out this morning. I'll get with him this afternoon and we'll see if we can do our testing on |
@jlahtinen, I work with @scrozier. As I read through this issue, it appears to focus on Rails 5.2 with the 5.2.2 adapter, right? We're seeing performance issues with Rails 5.1.5 and the 1.8.2 adapter. Our problem appears to involve CLOB columns in Oracle (text attributes in Rails). Across several applications, we're seeing query times of 400ms for queries that should take 20ms, but only if a CLOB attribute is part of the query. If we remove the CLOB from the query, the queries are back to 20ms. Any suggestions? Is this a known issue? Thanks for your help! |
@gyut maybe report that as a separate issue. This issue might be more about |
Would someone update your Gemfile as follows and let us know if your performance problem solved. gem "activerecord-oracle_enhanced-adapter", github: 'yahonda/oracle-enhanced', branch: "diag1720_release52" This branch has some changes:
Thank you. |
@yahonda running the main issue is retrieving indexes with the query as defined below.
edit:
SELECT LOWER(i.table_name) AS table_name,
LOWER(i.index_name) AS index_name,
i.uniqueness,
i.index_type,
i.ityp_owner,
i.ityp_name,
i.parameters,
LOWER(i.tablespace_name) AS tablespace_name,
LOWER(c.column_name) AS column_name,
e.column_expression,
atc.virtual_column
FROM all_indexes i
JOIN all_ind_columns c ON c.index_name = i.index_name AND c.index_owner = i.owner
LEFT OUTER JOIN all_ind_expressions e
ON e.index_name = i.index_name AND e.index_owner = i.owner AND e.column_position = c.column_position
LEFT OUTER JOIN all_tab_cols atc
ON i.table_name = atc.table_name AND c.column_name = atc.column_name AND i.owner = atc.owner AND
atc.hidden_column = 'NO'
WHERE i.owner = :owner
AND i.table_owner = :owner
AND NOT EXISTS(SELECT uc.index_name
FROM all_constraints uc
WHERE uc.index_name = i.index_name
AND uc.owner = i.owner
AND uc.constraint_type = 'P')
ORDER BY i.index_name, c.column_position |
Thanks a lot for this update @yahonda. The schema dumping performance in our application is way better with this branch, yet still about 30% slower than on Rails 5.1. On the other hand, the application is still quite unusably slow because the schema loading statements take way longer in the 5.2 adapter compared to 5.1. To illustrate this, we've performed one action in our application on both 5.1 and 5.2: In these logs you can see that certain statements take about 35x longer than compared to 5.1 (i.e. column definition load takes ~580ms instead of ~15ms). Note that these logs have been stripped to only show structural queries. Is there a particular reason these changes must be there for the 5.2 adapter, or is there a way we could continue using the 5.1 adapter for Rails 5.2? Unfortunately, the performance regressions currently prohibit us from upgrading to 5.2 (which we desperately need because of concurrency issues). Thanks a lot & let me know if I can be of any more help. |
@yahonda thank you for taking a look at this. Using your branch, my findings were sadly similar to those of @InteNs - the However, I found the further addition of: - result = select_all(<<-SQL.strip.gsub(/\s+/, " "), "indexes", [bind_string("owner", owner), bind_string("owner", owner)])
+ result = select_all(<<-SQL.strip.gsub(/\s+/, " "), "indexes", [bind_string("owner", owner), bind_string("owner", owner), bind_string("table_name", table_name)])
SELECT LOWER(i.table_name) AS table_name, LOWER(i.index_name) AS index_name, i.uniqueness,
i.index_type, i.ityp_owner, i.ityp_name, i.parameters,
LOWER(i.tablespace_name) AS tablespace_name,
LOWER(c.column_name) AS column_name, e.column_expression,
atc.virtual_column
FROM all_indexes#{db_link} i
JOIN all_ind_columns#{db_link} c ON c.index_name = i.index_name AND c.index_owner = i.owner
LEFT OUTER JOIN all_ind_expressions#{db_link} e ON e.index_name = i.index_name AND
e.index_owner = i.owner AND e.column_position = c.column_position
LEFT OUTER JOIN all_tab_cols#{db_link} atc ON i.table_name = atc.table_name AND
c.column_name = atc.column_name AND i.owner = atc.owner AND atc.hidden_column = 'NO'
WHERE i.owner = :owner
AND i.table_owner = :owner
+ AND i.table_name = :table_name
AND NOT EXISTS (SELECT uc.index_name FROM all_constraints uc
WHERE uc.index_name = i.index_name AND uc.owner = i.owner AND uc.constraint_type = 'P')
ORDER BY i.index_name, c.column_position
SQL to the query reduced the execution time from ~6000ms to ~15ms - perhaps worth rolling in too? As far as I can see, because this query is no longer cached it needn't get results for all indexes every time. |
Thanks for the testing everyone. I have made several changes: Actually my environment does not get performance improvement by adding a search condition. gem "activerecord-oracle_enhanced-adapter", github: 'yahonda/oracle-enhanced', branch: "diag1720_release52"
Please give it a try and let me know the result. Thank you. |
Thanks for the update. The performance is now similar to 5.1 in our tests. What is very weird though: I've prepared a script that uses multiple models and therefore causes the adapter to load the database structure of many tables (the same one that generated the log outputs of my previous post). Then I've tried the following:
So far so good.
So running the old version again seems to slow down subsequent queries by the new version significantly. Do you have an idea what could cause this effect? Does it have something to do with dictionary statistics? |
@yahonda Thank you for turning. I'm using multiple activerecord versions at my gem CI
Benchmark
Thank you. |
One possible reason is changing the execution plan may cause a longer elapsed time. If you are using Oracle Database 12c Release 1, how about disabling
Since these parameters are changing in every database versions, I am not thinking about setting these parameters in Oracle enhanced adapter itself. |
rsim#1490 removed Oracle enhanced adapter own column cache feature to use Rails `db:schema:cache:dump`. However, it caused slower performance about metadata search, such as column information. This commit restores Oracle enhanced adapter own column cache feature. It used to be class level caching, it is connection level caching now since it is much easier and cleaner than class level ones. Related to rsim#1720
yahonda/oracle-enhanced@2dc2ef9 seems to fix most performance problems we had. thanks for looking into this everyone! |
it actually reverts rsim#1503 rsim#1498 implemented prepared statements for dictionary queries. However rsim#1713 restored non-prepared statements for `column_definitions` and `pk_and_sequence_for`, which likely have caused longer elapsed time for these queries reported at rsim#1720 Those who needs 'cursor_sharing = exact' can specify its own value to .database.yml
it actually reverts rsim#1503 rsim#1498 implemented prepared statements for dictionary queries. However rsim#1713 restored non-prepared statements for `column_definitions` and `pk_and_sequence_for`, which likely have caused longer elapsed time for these queries reported at rsim#1720 Those who needs 'cursor_sharing = exact' can specify its own value to .database.yml
rsim#1490 removed Oracle enhanced adapter own column cache feature to use Rails `db:schema:cache:dump`. However, it caused slower performance about metadata search, such as column information. This commit restores Oracle enhanced adapter own column cache feature. It used to be class level caching, it is connection level caching now since it is much easier and cleaner than class level ones. Related to rsim#1720
rsim#1490 removed Oracle enhanced adapter own column cache feature to use Rails `db:schema:cache:dump`. However, it caused slower performance about metadata search, such as column information. This commit restores Oracle enhanced adapter own column cache feature. It used to be class level caching, it is connection level caching now since it is much easier and cleaner than class level ones. Related to rsim#1720
rsim#1490 removed Oracle enhanced adapter own column cache feature to use Rails `db:schema:cache:dump`. However, it caused slower performance about metadata search, such as column information. This commit restores Oracle enhanced adapter own column cache feature. It used to be class level caching, it is connection level caching now since it is much easier and cleaner than class level ones. Related to rsim#1720
Oracle enhanced adapter 5.2.3 has been released which should address this issue. |
Huge thanks, it works like a charm 👍 |
The problem persist when I use db:schema:dump ruby '2.6.3' indexes (9006.2ms) SELECT LOWER(i.table_name) AS table_name, LOWER(i.index_name) AS index_name, i.uniqueness, i.index_type, i.ityp_owner, i.ityp_name, i.parameters, LOWER(i.tablespace_name) AS tablespace_name, LOWER(c.column_name) AS column_name, e.column_expression, atc.virtual_column FROM all_indexes i JOIN all_ind_columns c ON c.index_name = i.index_name AND c.index_owner = i.owner LEFT OUTER JOIN all_ind_expressions e ON e.index_name = i.index_name AND e.index_owner = i.owner AND e.column_position = c.column_position LEFT OUTER JOIN all_tab_cols atc ON i.table_name = atc.table_name AND c.column_name = atc.column_name AND i.owner = atc.owner AND atc.hidden_column = 'NO' WHERE i.owner = :owner AND i.table_owner = :owner AND i.table_name = :table_name AND NOT EXISTS (SELECT uc.index_name FROM all_constraints uc WHERE uc.index_name = i.index_name AND uc.owner = i.owner AND uc.constraint_type = 'P') ORDER BY i.index_name, c.column_position |
Hi @remofritzsche, ruby '2.6.3'
We have a large (40+ tables) application in which this is a significant issue, taking some minutes. I'm including debugging from a smaller project that's also slow to perform schema operations. In the example below there are only 12 tables, none of which has more than 10 columns a piece, and none of which have more than 4 indexes. The few tables that do have that many are 'X-to-many' tables, one index for primary key, one index per foreign key (x2), and one unique index for the pair. It's these tables that take the longest. Total run time of the below log was just shy of 45 seconds. The (specific schema/table names stripped for privacy.)
|
Hopefully this could help you to narrow reason for slowness:
Our 94 table dump takes 45s. All |
Steps to reproduce
Background is that we're experiencing massive performance issues since updating to rails 5.2. Looking at the logs, it seems that structural database statements make the difference. To isolate this, we performed
db:cache:dump
multiple times, both with rails 5.1 and rails 5.2.The difference is startling (see next 2 sections).
This may be related to #1673.
Expected behavior
Executing
db:cache:dump
of a large application under Rails 5.1:Corresponding statement log
The results are grouped by query type (the string before the parenthesized time in the rails log) and are the average over multiple runs.
Actual behavior
Executing
db:cache:dump
of the same application under Rails 5.2:Corresponding statement log
This is over 12x slower than with Rails 5.1.
System configuration
Rails version: 5.1 / 5.2
Oracle enhanced adapter version: 1.8.2 / 5.2.2
Ruby version: jruby 9.1.14.0 (2.3.3) 2017-11-08 2176f24 Java HotSpot(TM) 64-Bit Server VM 25.66-b17 on 1.8.0_66-b17 +jit [darwin-x86_64]
Oracle Database version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
Final thoughts
As always a ton of thanks to you for looking at this and I'm sorry I couln't provide an isolated performance test – I hope that comparing the database statements of the two log files will do the trick!
The text was updated successfully, but these errors were encountered: