Added chunking logic to the SUNION call to avoid lengthy blocking.

[davidalger]

Added support to getIdsMatchingAnyTags for chunking the SUNION calls to mitigate long-running SUNION calls from blocking redis connection requests. The chunk size defaults to 500 but is configurable via the sunion_chunk_size setting in the backend_options of the redis cache configuration.

[davidalger]

Note: Being unfamiliar with using the Lua scripting, I did not attempt to alter that to use the new configuration option for chunking when in Lua mode.

[colinmollenhour]

Thanks, David!

[sksonic]

@colinmollenhour We have experienced the same issue with the SUNION call blocking redis. The chunking option above has helped us but we had to disable lua scripting. Can a similar chunking logic be applied when lua scripting is enabled?

[colinmollenhour]

It probably could be, but unfortunately this is not something I have the time to tackle. However, I would suggest investigating to see why you have so many tags; to me having to query for ids matching over 500 tags indicates an issue with improper tagging strategy.