How to specify a different Redis instance/replica for specific operations?

[axsuul]

Hello. I have a separate Redis read replica that I'd like to direct O(n) Sidekiq client queries to (e.g. SCAN) so that it won't negatively impact performance of the master instance. However, I'm not seeing a thread-safe way of specifying which Redis instance to use.

I saw some documentation on being able to provide a block for the connection but I don't think it helps me since there doesn't seem to be any context passed into the block. In other words, I'd like to be able to achieve

retries = Sidekiq::RetrySet.new

# Query replica/slave
retries.scan(..)

# Query master
retries.size

Just to clarify, the replica would only have read-only queries directed towards it — any enqueuing of jobs would still be directed to master. Any suggestions on how I could achieve this? Thanks

[mperham]

You didn’t note your versions in use. The API is locked to Sidekiq.redis. You can configure each process to use a given Redis via REDIS_URL.

…

On Tue, Feb 1, 2022 at 17:58 James Hu ***@***.***> wrote: Hello. I have a separate Redis read replica that I'd like to direct O(n) Sidekiq client queries to (e.g. SCAN). However, I'm not seeing a thread-safe way of specifying which Redis instance to use. I saw some documentation on being able to provide a block for the connection but I don't think it helps me since there doesn't seem to be any context passed into the block. In other words, I'd like to be able to achieve retries = Sidekiq::RetrySet.new # Query replica/slaveretries.scan(..) # Query masterretries.size Any suggestions on how I could achieve this? Thanks — Reply to this email directly, view it on GitHub <#5168>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAWXZFMZ2MKK2FIHR3IYDUZCFVLANCNFSM5NKZVT3Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[axsuul]

You didn’t note your versions in use. The API is locked to Sidekiq.redis. You can configure each process to use a given Redis via REDIS_URL.
…
On Tue, Feb 1, 2022 at 17:58 James Hu @.> wrote: Hello. I have a separate Redis read replica that I'd like to direct O(n) Sidekiq client queries to (e.g. SCAN). However, I'm not seeing a thread-safe way of specifying which Redis instance to use. I saw some documentation on being able to provide a block for the connection but I don't think it helps me since there doesn't seem to be any context passed into the block. In other words, I'd like to be able to achieve retries = Sidekiq::RetrySet.new # Query replica/slaveretries.scan(..) # Query masterretries.size Any suggestions on how I could achieve this? Thanks — Reply to this email directly, view it on GitHub <#5168>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAWXZFMZ2MKK2FIHR3IYDUZCFVLANCNFSM5NKZVT3Q . You are receiving this because you are subscribed to this thread.Message ID: @.>

Thanks for the quick response! I'm on latest 6.4.0.

Don't think REDIS_URL will work for me since I need to switch it up based on context, within the same process. Does that mean there's no way of doing what I need to do right now?

[mperham]

Well, API sharding is built into Sidekiq Pro. You can get something working with Sidekiq Pro, but not Sidekiq.

[axsuul]

Ah ok gotcha, open to getting Sidekiq Pro then. Could you point me to the feature in Sidekiq Pro that I would need to make this work?

[mperham]

It’s the Web UI sharding support. Since the web UI uses the API to fetch all of its data, the API needs to be shard aware.

…

On Tue, Feb 1, 2022 at 23:57 James Hu ***@***.***> wrote: Ah ok gotcha, open to getting Sidekiq Pro then. Could you point me to the feature in Sidekiq Pro that I would need to make this work? — Reply to this email directly, view it on GitHub <#5168 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAWX7QN4UJVCDX2NZM55TUZDPUZANCNFSM5NKZVT3Q> . You are receiving this because you commented.Message ID: ***@***.***>

[axsuul]

Thanks! Do you know if is there a performance hit if I need to do to lots of ConnectionPool.new { Redis.new(...) } within a single thread? Hope that makes sense.

[mperham]

That’s not ideal at all. Pools are designed to be created once and provide a cache of established network connections. You’d suffer a lot of TCP setup latency.

…

On Wed, Feb 2, 2022 at 15:36 James Hu ***@***.***> wrote: Thanks! Do you know if is there a performance hit if I need to do to lots of ConnectionPool.new { Redis.new(...) } within a single thread? Hope that makes sense. — Reply to this email directly, view it on GitHub <#5168 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAWX63AZAAVPGY56ES2O3UZG5WZANCNFSM5NKZVT3Q> . You are receiving this because you commented.Message ID: ***@***.***>