Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Main thread stuck when loading very large keys #562

Closed
yoav-steinberg opened this Issue · 7 comments

2 participants

@yoav-steinberg

Similar to #539 the main thread is stuck and redis becomes unresponsive during replication. In this case this happens because rdbLoad processes client events once every 1000 loaded keys but for rdb's with a small number of very large keys this doesn't work well. For example an rdb file with a single list that contains 10 million 1k strings will block for a very long time while reading this list.

I suggest changing the way we periodically process client events by processing them every X bytes loaded from the rdb and not every N (1000) keys.

Here's a patch that seemed to fix the issue for me:
yoav-steinberg/redis_garantia@4d7f4ef

@antirez
Owner

2.6 -> 2.8

@antirez
Owner

Hello, I understand you use the current patch in product, is it right? Everything stable so far? Thank you.

@yoav-steinberg

This patch was modified to accommodate the "rio" layer added in 2.6 (I think it was originally written for 2.4).
The latest version is used successfully in production. You can find it here: RedisLabs@3aab06b

@antirez
Owner

Hello @yoav-steinberg, the patch seems good to me, however it must be noted that it introduces an important difference compared to the past, that is, when re-enter the event loop in the current implementation (before applying your patch) we may have a non completely loaded dataset, that is, just a percentage of keys could be present, not all, but keys are well-formed and accessible.

With the new implementation we may have half-loaded keys, and maybe the representation of keys may not even be ok to access without triggering a crash.

Now given that currently the commands allowed to run in loading state should never touch the dataset, this is fine, but care must be used in the future with the loading flag of the command table.

I think I'll merge ASAP, just doing the last code review right now.

@yoav-steinberg

Isn't the key/value added "atomically" in the addKey(db,key,val) call? Assuming that's the case I don't see how a partial key can exist because addKey doesn't perform any disk access and therefore won't trigger the process events..
What am I missing?

@antirez
Owner

That's a good point indeed, we create the whole value, and then add it to the key space, so actually the same property should hold true after we apply your patch. Thanks!

@antirez
Owner

Merged, thanks! Closing.

@antirez antirez closed this
@JackieXie168 JackieXie168 referenced this issue from a commit
Slava Akhmechet Adding support for declaring machines dead in the UI. It works and it…
… is lovely (reference #562)
861536d
@JackieXie168 JackieXie168 referenced this issue from a commit
Slava Akhmechet Name conflict issue now supports resolution via renaming (see #562). …
…Also, #616 should use rename dialog code written here, as it is designed to be generic and is not specific to name conflict resolution.
62af87a
@JackieXie168 JackieXie168 referenced this issue from a commit
Slava Akhmechet Persistence issue support in the UI. See #562. d28c476
@JackieXie168 JackieXie168 referenced this issue from a commit
@srh srh Revert "addresses local issue reporting in #562"
This reverts commit 1ebb6f0.

An commit that depends on the merge that was just reverted.
eaf7381
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.