Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?) #39

Closed
x1wins opened this issue Nov 14, 2020 · 0 comments · Fixed by #41
Closed

Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?) #39

x1wins opened this issue Nov 14, 2020 · 0 comments · Fixed by #41
Assignees
@x1wins
Labels
bug Something isn't working invalid This doesn't seem right wontfix This will not be worked on

Comments

@x1wins
Copy link
Owner

x1wins commented Nov 14, 2020 

CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:43:08.763Z pid=1 tid=gmo29afdp class=ThumbnailWorker jid=477de07d4ec15035dd8781fc INFO:  cp_thumbnail_to_cdn_cmd vod-hls 2020/11/12/22/thumbnail/10_00_00_38_000.png /storage/22_thumbnail/10_00_00_38_000.png
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:43:08.776Z pid=1 tid=gmo29afdp class=ThumbnailWorker jid=477de07d4ec15035dd8781fc elapsed=23.667 INFO: done
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.754Z pid=1 tid=gmo29afdp ERROR: Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.755Z pid=1 tid=gmo29afdp WARN: Redis::CommandError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.755Z pid=1 tid=gmo29afdp WARN: /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:127:in `call'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:221:in `block in call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:295:in `with_socket_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:220:in `call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1224:in `block in _bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `block in synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1221:in `_bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1266:in `brpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `block in retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:98:in `block in redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:95:in `redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:83:in `get_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:95:in `fetch'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:77:in `process_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:68:in `run'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:15:in `watchdog'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.755Z pid=1 tid=gmo29afjt ERROR: Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.755Z pid=1 tid=gmo29afjt WARN: Redis::CommandError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.755Z pid=1 tid=gmo29afjt WARN: /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:127:in `call'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:221:in `block in call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:295:in `with_socket_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:220:in `call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1224:in `block in _bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `block in synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1221:in `_bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1266:in `brpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `block in retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:98:in `block in redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:95:in `redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:83:in `get_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:95:in `fetch'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:77:in `process_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:68:in `run'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:15:in `watchdog'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.754Z pid=1 tid=gmo280z4p ERROR: Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.755Z pid=1 tid=gmo280z4p WARN: Redis::CommandError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.756Z pid=1 tid=gmo280z4p WARN: /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:127:in `call'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:221:in `block in call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:295:in `with_socket_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:220:in `call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1224:in `block in _bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `block in synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1221:in `_bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1266:in `brpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `block in retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:98:in `block in redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:95:in `redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:83:in `get_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:95:in `fetch'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:77:in `process_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:68:in `run'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:15:in `watchdog'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.755Z pid=1 tid=gmo280ysh ERROR: Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.756Z pid=1 tid=gmo280ysh WARN: Redis::CommandError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.756Z pid=1 tid=gmo280ysh WARN: /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:127:in `call'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:221:in `block in call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:295:in `with_socket_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:220:in `call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1224:in `block in _bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `block in synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1221:in `_bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1266:in `brpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `block in retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:98:in `block in redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:95:in `redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:83:in `get_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:95:in `fetch'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:77:in `process_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:68:in `run'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:15:in `watchdog'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.755Z pid=1 tid=gmo29af1h ERROR: Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.756Z pid=1 tid=gmo29af1h WARN: Redis::CommandError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.757Z pid=1 tid=gmo29af1h WARN: /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:127:in `call'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:221:in `block in call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:295:in `with_socket_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:220:in `call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1224:in `block in _bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `block in synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1221:in `_bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1266:in `brpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `block in retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:98:in `block in redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:95:in `redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:83:in `get_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:95:in `fetch'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:77:in `process_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:68:in `run'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:15:in `watchdog'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.755Z pid=1 tid=gmo29af7l ERROR: Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.757Z pid=1 tid=gmo29af7l WARN: Redis::CommandError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.757Z pid=1 tid=gmo29af7l WARN: /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:127:in `call'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:221:in `block in call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:295:in `with_socket_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:220:in `call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1224:in `block in _bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `block in synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1221:in `_bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1266:in `brpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `block in retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:98:in `block in redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:95:in `redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:83:in `get_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:95:in `fetch'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:77:in `process_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:68:in `run'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:15:in `watchdog'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.757Z pid=1 tid=gmo280yyl ERROR: Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.757Z pid=1 tid=gmo280yyl WARN: Redis::CommandError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.757Z pid=1 tid=gmo280yyl WARN: /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:127:in `call'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:221:in `block in call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:295:in `with_socket_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:220:in `call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1224:in `block in _bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `block in synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1221:in `_bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1266:in `brpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `block in retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:98:in `block in redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:95:in `redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:83:in `get_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:95:in `fetch'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:77:in `process_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:68:in `run'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:15:in `watchdog'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.758Z pid=1 tid=gmo29afpx ERROR: Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.758Z pid=1 tid=gmo29afpx WARN: Redis::CommandError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.758Z pid=1 tid=gmo29afpx WARN: /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:127:in `call'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:221:in `block in call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:295:in `with_socket_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:220:in `call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1224:in `block in _bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `block in synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1221:in `_bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1266:in `brpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `block in retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:98:in `block in redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:95:in `redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:83:in `get_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:95:in `fetch'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:77:in `process_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:68:in `run'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:15:in `watchdog'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.766Z pid=1 tid=gmo29aep9 ERROR: Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.766Z pid=1 tid=gmo29aep9 WARN: Redis::CommandError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.766Z pid=1 tid=gmo29aep9 WARN: /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:127:in `call'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:221:in `block in call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:295:in `with_socket_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:220:in `call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1224:in `block in _bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `block in synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1221:in `_bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1266:in `brpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `block in retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:98:in `block in redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:95:in `redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:83:in `get_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:95:in `fetch'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:77:in `process_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:68:in `run'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:15:in `watchdog'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.766Z pid=1 tid=gmo29aevd ERROR: Error fetching job: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.766Z pid=1 tid=gmo29aevd WARN: Redis::CommandError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:26.766Z pid=1 tid=gmo29aevd WARN: /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:127:in `call'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:221:in `block in call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:295:in `with_socket_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis/client.rb:220:in `call_with_timeout'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1224:in `block in _bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `block in synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:69:in `synchronize'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1221:in `_bpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/redis-4.2.1/lib/redis.rb:1266:in `brpop'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `block in retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:98:in `block in redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq.rb:95:in `redis'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/fetch.rb:39:in `retrieve_work'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:83:in `get_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:95:in `fetch'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:77:in `process_one'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/processor.rb:68:in `run'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:15:in `watchdog'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | /usr/local/bundle/ruby/2.6.0/gems/sidekiq-6.1.0/lib/sidekiq/util.rb:24:in `block in safe_thread'
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:31.366Z pid=1 tid=gmo280zi1 ERROR: heartbeat: READONLY You can't write against a read only replica.
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:36.856Z pid=1 tid=gmo29afjt INFO: Redis is online, 10.101205447106622 sec downtime
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:36.856Z pid=1 tid=gmo280yyl INFO: Redis is online, 10.09892669110559 sec downtime
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:36.858Z pid=1 tid=gmo29afdp INFO: Redis is online, 10.104536039987579 sec downtime
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:36.859Z pid=1 tid=gmo29aevd INFO: Redis is online, 10.092739504994825 sec downtime
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:36.856Z pid=1 tid=gmo29afpx INFO: Redis is online, 10.098559829988517 sec downtime
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:36.856Z pid=1 tid=gmo29af7l INFO: Redis is online, 10.101458562072366 sec downtime
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:36.857Z pid=1 tid=gmo280ysh INFO: Redis is online, 10.101731691975147 sec downtime
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:36.859Z pid=1 tid=gmo29aep9 INFO: Redis is online, 10.092762653017417 sec downtime
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:36.856Z pid=1 tid=gmo280z4p INFO: Redis is online, 10.102081213030033 sec downtime
CW-OVP_sidekiq.2.7bo7qn41qt07@aws-worker1    | 2020-11-12T05:48:36.856Z pid=1 tid=gmo29af1h INFO: Redis is online, 10.101761467987671 sec downtime
error from daemon in stream: Error grabbing logs: rpc error: code = Unknown desc = warning: incomplete log stream. some logs could not be retrieved for the following reasons: node ns3j1r7qgumbneet7i54ffu5n disconnected unexpectedly
@x1wins x1wins self-assigned this Nov 14, 2020
@x1wins x1wins added bug Something isn't working invalid This doesn't seem right wontfix This will not be worked on labels Nov 14, 2020
@x1wins x1wins linked a pull request Nov 14, 2020 that will close this issue
Feature/redis config #41
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@x1wins x1wins

Labels
bug Something isn't working invalid This doesn't seem right wontfix This will not be worked on
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feature/redis config x1wins/CW-OVP
1 participant
@x1wins