Dalli / memcached slowness in Rails 3.2 project after upgrade #236
Comments
|
|
|
I will profile around the write blocks and the read blocks and post the graphs here. |
|
Here's the profile, single cache write. |
|
You'll need to profile a lot more calls to get an statistically accurate profile with timings that help me: profile do
10_000.times do
dalli.write('xyz', 123)
end
end |
|
NP Will have that profile A.S.A.P |
|
There you go |
|
First thing I noticed: marshalling of the value is taking >50% of the total time. |
|
Everything else looks about normal. Can you get the same profile with Dalli 1.1 so we can compare total time and profiles? |
|
Sure. Sincerely, Tel: 972-54-5960553 English blog: http://www.kensodev.com On Monday, July 9, 2012 at 4:36 PM, Mike Perham wrote:
|
|
Here's the profile from the Rails 3.0.9 version |
|
Marshalling takes 57% of the time in 3.0.9, even worse than 3.2. Both the 3.2 and 3.0.9 versions take 42 seconds. Can you reproduce the slowdown? Your NR graphs show a 4x slowdown but the profile doesn't reflect that. |
|
I'll try to reproduce using the Profiling, you can see the slowness in the benchmark as well. |
|
Any news? |
|
No, not yet unfortunately. |
|
Have you tried something like this? # development.rb
store = ActiveSupport::Cache.lookup_store(:dalli_store, :expires_in => 1.minute)
store.send(:extend, ActiveSupport::Cache::Strategy::LocalCache)
config.cache_store = store
config.middleware.use(store.middleware)It only doesn't cache nil-results, so those would still hit the memcached on repeated negative lookups, but it should still provide some improvement assuming you have repeated lookups within the same request. Edit: The |
|
KensoDev, Rtger |
|
As I said above, the dalli's active support store isn't implemented "properly" (as in inheriting from the With a thread-local storage the actual objects are cached in memory for the duration of a request, meaning repeated lookups are more akin to hitting a hashmap rather than network request + unmarshaling. This can have a huge performance impact compared to the other activesupport stores, depending on your usage patterns. |
|
Send me a PR to pull in LocalCache. |
|
|
Another performance issue: cache = ActiveSupport::Cache.lookup_store(:dalli_store)
Benchmark.ms{10000.times{cache.fetch("b1"){"bar"}}} # 3587.053924
Benchmark.ms{10000.times{cache.fetch("b2"){"baz"}}} # 3715.264835
Benchmark.ms{10000.times{cache.fetch("b3"){nil}}} # 8313.579682
cache = ActiveSupport::Cache.lookup_store(:memory_store)
Benchmark.ms{10000.times{cache.fetch("b4"){nil}}} # 321.031518
Benchmark.ms{10000.times{cache.fetch("b5"){"baz"}}} # 369.558517 Something in the code must be treating nil as a cache miss instead of checking the presence of (any) data in the cache. |
|
An issue with |
It could be wrapped with a placeholder/unwrapped on fetch. Caching nils is important for i18n where negative hits need to be stored too as they might trigger fallbacks/full evaluation of the whole i18n lookup chain. Basically: Rails.cache.fetch("some.key") do
do_some_really_expensive_lookup_that_might_be_nil()
endIf nil is seen as a cache miss then this lookup will happen every time and therefore degrade performance The other cache stores don't have that issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
upfront warning, I am not 100% sure this is coming from Dalli but I wanted to post an issue because there's a good chance it is and I wanted to discuss possible solutions (if any)
We upgraded the Gogobot app 2 days ago to run on Rails 3.2.
The code base is exactly the same, other then the upgrade nothing obviously changed from the current version of the app.
The current version of the app is Rails 3.0.9 and we have been using Dalli for a long while now, on the current app and it's running super smooth with no problems at all.
The minute we released the new version, the app started to be very slow and the memcached section in NewRelic showed significant slowness from the current app version.
Here's a screenshot from the 3.2 version from NewRelic.
As you can see, memcached is taking 80ms per request.
Now, here's a screenshot from the current version of the app (rolled back to Rails 3.0.9)
memcached time is 22.1ms.
From looking into memcached stats, the number of gets and sets is similar, it's not like cache is working harder for any apparent reason.
Some Benchmarks (From Rails 3.2)
Benchmarks from rails 3.0.9
Dalli version used on the 3.2 is the latest available, the 3.0 version uses 1.1.2.
During the upgrade, we obviously upgraded other gems to the newest version, let me know if the Gemfile.lock from the 2 versions would be helpful, I will post it here.
These were done on staging ENV, local memcached server, no requests but the benchmark one, so memcached was not working on anything other then the benchmark.
Thoughts?
The text was updated successfully, but these errors were encountered: