RuntimeError: can't add a new key into hash during iteration on lib/active_record/connection_adapters/abstract/query_cache.rb #45287
Comments
|
Hi @sobrinho , thanks for opening this issue! |
|
Checking the source code there is almost no changes between 6.1.3.2 and master in that regard. Upgrading from 6.1 to 7.0 is also not easy due to the size/nature of the application, it will take a while. |
|
Fair enough. I'm planning to take a look at this in the next few weeks and will report back if I find anything interesting. That being said, the Rails maintenance policy is to only apply bug fixes to the latest release series https://guides.rubyonrails.org/maintenance_policy.html#bug-fixes, so if anyone reading this bug is able to reproduce this in Rails 7, that would be helpful! |
|
The reason that the other issue didn't have a solution is because we're pretty sure that this isn't a bug in Rails and no one has provided a way to reproduce or prove that it is Rails. The code already has lock synchronization so it is likely an external gem that is incorrectly iterating over the query cache. Since we can't see your application, we can't guess what that gem might be. I'd start with scout-apm as it's in your stack trace and is on a version that is 3 years old. |
|
@eileencodes I can provide further details in the case of need. So far I can tell that there is no references to That includes scout_apm, of course, no mentions to query_cache at all. |
|
It might not be a direct reference to the query cache, but a lack of a lock synch in code overriding Active Record. |
|
@eileencodes alright, let me try to update the gem and see what happens |
|
This just started happening to us also, and I get the same search result as @sobrinho: $ ag '\bquery_cache\b' $(bundle list --paths)Click to show results
If the only manipulation of / access of @query_cache happens in @query_cache = Hash.new { |h, sql| h[sql] = {} }
@query_cache.clear
@query_cache[sql]Right? Does anyone know how to trigger a |
|
@jordan-brough please keep me updated if you find anything new, we are going to release the gem upgrade as suggested by Eileen but since there is no manipulation in the query cache at all, I'm guessing it won't take any effect. I will keep here updated as well. |
|
This issue has been automatically marked as stale because it has not been commented on for at least three months. |
|
I'm still seeing this in a few different places, even during HTTP requests. |
|
This issue has been automatically marked as stale because it has not been commented on for at least three months. |
Reproduced almost the same issue at 7.0.8.1 - also mostly in Sidekiq workers. |
|
Not stale then. |
|
All I found looks related to concurrent-ruby and working on shared hash between threads. Could it be related? concurrent-ruby: 1.2.3 |
Steps to reproduce
No idea how to reproduce.
Expected behavior
No concurrency errors in query cache.
Actual behavior
A concurrency error in query cache.
It is happening inside a Sidekiq worker, FWIW.
And it has been happening for at least a year now:
Here's the backtrace:
Pseudo code from the original source code:
This has been reported before here but without a specific solution.
sentry-raven,sentry-rubyandactiverecord-importwere mentioned but we aren't using it but we do haveprometheus_exporter.What I don't get is how this error could happen considering there is a monitor around the query cache.
System configuration
Rails version: 6.1.3.2
Ruby version: 2.7.2
The text was updated successfully, but these errors were encountered: