Skip to content

Commit

Permalink
Allow apps to coexist in a single Redis database
Browse files Browse the repository at this point in the history
Adds the ability to set a `key_prefix` which will be prepended to all Redis keys. This allows multiple applications to keep their schedules in the same Redis database without clobbering each other.
  • Loading branch information
TALlama authored and marcelolx committed Sep 8, 2022
1 parent 88db34a commit 8ad1f74
Show file tree
Hide file tree
Showing 3 changed files with 79 additions and 18 deletions.
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,20 @@ Redis.

See https://github.com/sidekiq-scheduler/sidekiq-scheduler/issues/361 for a more details.

## Notes when running multiple applications in the same Redis database

If you need to run multiple applications with differing schedules, the easiest way is to use a different Redis database per application. Doing that will ensure that each application will have its own schedule, web interface and statistics.

However, you may want to have a set of related applications share the same Redis database in order to aggregate statistics and manage them all in a single web interface. To do this while maintaining a different schedule for each application, you can configure each application to use a different `key_prefix` in Redis. This prevents the applications overwriting each others' schedules and schedule data.

```ruby
Rails.application.reloader.to_prepare do
SidekiqScheduler::RedisManager.key_prefix = "my-app"
end
```

Note that this must be set before the schedule is loaded (or it will go into the wrong key). If you are using the web integration, make sure that the prefix is set in the web process so that you see the correct schedule.

## Sidekiq Web Integration

sidekiq-scheduler provides an extension to the Sidekiq web interface that adds a `Recurring Jobs` page.
Expand Down
27 changes: 21 additions & 6 deletions lib/sidekiq-scheduler/redis_manager.rb
Original file line number Diff line number Diff line change
Expand Up @@ -156,42 +156,57 @@ def self.remove_elder_job_instances(job_name)
#
# @return [String] the pushed job key
def self.pushed_job_key(job_name)
"sidekiq-scheduler:pushed:#{job_name}"
"#{key_prefix}sidekiq-scheduler:pushed:#{job_name}"
end

# Returns the key of the Redis hash for job's execution times hash
#
# @return [String] with the key
def self.next_times_key
'sidekiq-scheduler:next_times'
"#{key_prefix}sidekiq-scheduler:next_times"
end

# Returns the key of the Redis hash for job's last execution times hash
#
# @return [String] with the key
def self.last_times_key
'sidekiq-scheduler:last_times'
"#{key_prefix}sidekiq-scheduler:last_times"
end

# Returns the Redis's key for saving schedule states.
#
# @return [String] with the key
def self.schedules_state_key
'sidekiq-scheduler:states'
"#{key_prefix}sidekiq-scheduler:states"
end

# Returns the Redis's key for saving schedules.
#
# @return [String] with the key
def self.schedules_key
'schedules'
"#{key_prefix}schedules"
end

# Returns the Redis's key for saving schedule changes.
#
# @return [String] with the key
def self.schedules_changed_key
'schedules_changed'
"#{key_prefix}schedules_changed"
end

# Returns the key prefix used to generate all scheduler keys
#
# @return [String] with the key prefix
def self.key_prefix
@key_prefix
end

# Sets the key prefix used to scope all scheduler keys
#
# @param [String] value The string to use as the prefix. A ":" will be appended as a delimiter if needed.
def self.key_prefix=(value)
value = "#{value}:" if value && !%w[. :].include?(value[-1])
@key_prefix = value
end

private
Expand Down
56 changes: 44 additions & 12 deletions spec/sidekiq-scheduler/redis_manager_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

describe SidekiqScheduler::RedisManager do

before { SidekiqScheduler::RedisManager.key_prefix = nil }
before { SidekiqScheduler::Store.clean }

describe '.get_job_schedule' do
Expand All @@ -21,7 +22,7 @@
let(:job_name) { 'some_job' }
let(:state) { JSON.generate('enabled' => true) }

before { SidekiqScheduler::Store.hset('sidekiq-scheduler:states', job_name, state) }
before { SidekiqScheduler::Store.hset(SidekiqScheduler::RedisManager.schedules_state_key, job_name, state) }

it { is_expected.to eq(state) }
end
Expand All @@ -32,7 +33,7 @@
let(:job_name) { 'some_job' }
let(:next_time) { 'some_time' }

before { SidekiqScheduler::Store.hset('sidekiq-scheduler:next_times', job_name, next_time) }
before { SidekiqScheduler::Store.hset(SidekiqScheduler::RedisManager.next_times_key, job_name, next_time) }

it { is_expected.to eq(next_time) }
end
Expand All @@ -43,7 +44,7 @@
let(:job_name) { 'some_job' }
let(:last_time) { 'some_time' }

before { SidekiqScheduler::Store.hset('sidekiq-scheduler:last_times', job_name, last_time) }
before { SidekiqScheduler::Store.hset(SidekiqScheduler::RedisManager.last_times_key, job_name, last_time) }

it { is_expected.to eq(last_time) }
end
Expand Down Expand Up @@ -71,7 +72,7 @@
it 'should store the job state' do
subject

stored_state = SidekiqScheduler::Store.hget('sidekiq-scheduler:states', job_name)
stored_state = SidekiqScheduler::Store.hget(SidekiqScheduler::RedisManager.schedules_state_key, job_name)
expect(JSON.parse(stored_state)).to eq(state)
end
end
Expand All @@ -85,7 +86,7 @@
it 'should store the job next_time' do
subject

stored_next_time = SidekiqScheduler::Store.hget('sidekiq-scheduler:next_times', job_name)
stored_next_time = SidekiqScheduler::Store.hget(SidekiqScheduler::RedisManager.next_times_key, job_name)
expect(stored_next_time).to eq(next_time)
end
end
Expand All @@ -99,7 +100,7 @@
it 'should store the job last_time' do
subject

stored_last_time = SidekiqScheduler::Store.hget('sidekiq-scheduler:last_times', job_name)
stored_last_time = SidekiqScheduler::Store.hget(SidekiqScheduler::RedisManager.last_times_key, job_name)
expect(stored_last_time).to eq(last_time)
end
end
Expand Down Expand Up @@ -137,12 +138,12 @@
let(:job_name) { 'some_job' }
let(:next_time) { 'some_time' }

before { SidekiqScheduler::Store.hset('sidekiq-scheduler:next_times', job_name, next_time) }
before { SidekiqScheduler::Store.hset(SidekiqScheduler::RedisManager.next_times_key, job_name, next_time) }

it 'should remove the job next_time' do
subject

stored_next_time = SidekiqScheduler::Store.hget('sidekiq-scheduler:next_times', job_name)
stored_next_time = SidekiqScheduler::Store.hget(SidekiqScheduler::RedisManager.next_times_key, job_name)
expect(stored_next_time).to be_nil
end

Expand All @@ -152,7 +153,7 @@
it 'should maintain inexisting' do
subject

stored_next_time = SidekiqScheduler::Store.hget('sidekiq-scheduler:next_times', job_name)
stored_next_time = SidekiqScheduler::Store.hget(SidekiqScheduler::RedisManager.next_times_key, job_name)
expect(stored_next_time).to be_nil
end
end
Expand Down Expand Up @@ -284,20 +285,21 @@

let(:current_time) { Time.now }
let(:job_name) { 'some_job' }
let(:job_key) { SidekiqScheduler::RedisManager.pushed_job_key('some_job') }
let(:time) { current_time }

it { expect(subject).to be_truthy }

it 'should store the job instance' do
subject

expect(SidekiqScheduler::Store.zrange('sidekiq-scheduler:pushed:some_job', 0, -1)).to eql([time.to_i.to_s])
expect(SidekiqScheduler::Store.zrange(job_key, 0, -1)).to eql([time.to_i.to_s])
end

it 'should add an expiration key' do
subject

ttl = Sidekiq.redis { |r| r.ttl('sidekiq-scheduler:pushed:some_job') }
ttl = Sidekiq.redis { |r| r.ttl(job_key) }
expect(ttl).to eql(SidekiqScheduler::RedisManager::REGISTERED_JOBS_THRESHOLD_IN_SECONDS)
end

Expand All @@ -319,7 +321,7 @@

let(:current_time) { Time.now }
let(:job_name) { 'some_job' }
let(:job_key) { 'sidekiq-scheduler:pushed:some_job' }
let(:job_key) { SidekiqScheduler::RedisManager.pushed_job_key('some_job') }
let(:job_instance) { current_time.to_i }
let(:other_job_instance) { (current_time - (20 * 60)).to_i }
let(:old_job_instance) { (current_time - SidekiqScheduler::RedisManager::REGISTERED_JOBS_THRESHOLD_IN_SECONDS).to_i }
Expand All @@ -343,35 +345,65 @@
let(:job_name) { 'some_job' }

it { is_expected.to eq('sidekiq-scheduler:pushed:some_job') }

context 'when a key prefix is set' do
before { described_class.key_prefix = 'some-prefix' }
it { is_expected.to eq('some-prefix:sidekiq-scheduler:pushed:some_job') }
end
end

describe '.next_times_key' do
subject { described_class.next_times_key }

it { is_expected.to eq('sidekiq-scheduler:next_times') }

context 'when a key prefix is set' do
before { described_class.key_prefix = 'some-prefix' }
it { is_expected.to eq('some-prefix:sidekiq-scheduler:next_times') }
end
end

describe '.last_times_key' do
subject { described_class.last_times_key }

it { is_expected.to eq('sidekiq-scheduler:last_times') }

context 'when a key prefix is set' do
before { described_class.key_prefix = 'some-prefix' }
it { is_expected.to eq('some-prefix:sidekiq-scheduler:last_times') }
end
end

describe '.schedules_state_key' do
subject { described_class.schedules_state_key }

it { is_expected.to eq('sidekiq-scheduler:states') }

context 'when a key prefix is set' do
before { described_class.key_prefix = 'some-prefix' }
it { is_expected.to eq('some-prefix:sidekiq-scheduler:states') }
end
end

describe ".schedules_key" do
subject { described_class.schedules_key }

it { is_expected.to eq('schedules') }

context 'when a key prefix is set' do
before { described_class.key_prefix = 'some-prefix' }
it { is_expected.to eq('some-prefix:schedules') }
end
end

describe ".schedules_changed_key" do
subject { described_class.schedules_changed_key }

it { is_expected.to eq('schedules_changed') }

context 'when a key prefix is set' do
before { described_class.key_prefix = 'some-prefix' }
it { is_expected.to eq('some-prefix:schedules_changed') }
end
end
end

0 comments on commit 8ad1f74

Please sign in to comment.