Skip to content
Browse files

Base `JobBatch#complete?` on the presence of `completed_at`.

The old logic was designed to figure out when to mark the
job batch as complete by setting the timestamp -- but
that has been moved into the lua script. Basing `complete?`
on the job jid sets is potentially risky since we have
discussed making some of the job batch redis keys expire
sooner than others. If the job jid set keys vanished from
redis the old logic would wrongly report the job batch
was not complete even though it was. The new logic
relies only on the `meta` key which is the primary key
used by a job batch to track its state.
  • Loading branch information...
1 parent f032f2e commit 17ea826dd6cd2546fcab979d328842dab13372b3 @myronmarston myronmarston committed
Showing with 6 additions and 17 deletions.
  1. +1 −5 lib/plines/job_batch.rb
  2. +5 −12 spec/unit/plines/job_batch_spec.rb
6 lib/plines/job_batch.rb
@@ -132,7 +132,7 @@ def complete_job(qless_job)
def complete?
- _complete?(pending_job_jids.length, completed_job_jids.length)
+ !!completed_at
def resolve_external_dependency(dep_name)
@@ -319,10 +319,6 @@ def update_external_dependency(dep_name, meth, jids)
- def _complete?(pending_size, complete_size)
- pending_size == 0 && complete_size > 0
- end
def time_from(meta_entry)
date_string = meta[meta_entry]
Time.iso8601(date_string) if date_string
17 spec/unit/plines/job_batch_spec.rb
@@ -501,23 +501,16 @@ def create_batch_with_job
describe "#complete?" do
- it 'returns false when there are no pending or completed jobs' do
+ it 'returns true if the job batch has a completed_at timestamp' do
batch = JobBatch.create(qless, pipeline_module, "foo", {})
- expect(batch).not_to be_complete
+ batch.meta['completed_at'] =
+ expect(batch).to be_complete
- it 'returns false when there are pending jobs and completed jobs' do
+ it 'returns false if it lacks a completed_at timestamp' do
batch = JobBatch.create(qless, pipeline_module, "foo", {})
- batch.pending_job_jids << "a"
- batch.completed_job_jids << "b"
expect(batch).not_to be_complete
- it 'returns true when there are only completed jobs' do
- batch = JobBatch.create(qless, pipeline_module, "foo", {})
- batch.completed_job_jids << "b"
- expect(batch).to be_complete
- end
describe "#pending_qless_jobs" do
@@ -742,7 +735,7 @@ def update_dependency(batch, name)
it 'returns true for a complete job batch' do
- batch.completed_job_jids << "a"
+ batch.meta['completed_at'] =
expect(batch.in_terminal_state?).to be_true

0 comments on commit 17ea826

Please sign in to comment.
Something went wrong with that request. Please try again.