SessionContext::UploadLoop becomes a CPU spinloop

[mcb30]

The existing code in SessionContext::UploadLoop():

cvmfs/cvmfs/session_context.cc

Lines 414 to 429 in 8fc9e82

	while (!ctx->ShouldTerminate()) {
	while (jobs_processed < ctx->NumJobsSubmitted()) {
	UploadJob* job = ctx->upload_jobs_->Dequeue();
	if (!ctx->DoUpload(job)) {
	PANIC(kLogStderr,
	"SessionContext: could not submit payload. Aborting.");
	}
	job->result->Set(true);
	delete job->pack;
	delete job;
	jobs_processed++;
	}
	if (ctx->queue_was_flushed_.IsEmpty()) {
	ctx->queue_was_flushed_.Enqueue(true);
	}
	}

uses a simple (atomic) integer flag to indicate that the loop should terminate:

cvmfs/cvmfs/session_context.cc

Lines 434 to 436 in 8fc9e82

	bool SessionContext::ShouldTerminate() {
	return atomic_read32(&worker_terminate_);
	}

At the point that all jobs have been submitted (i.e. jobs_processed is equal to ctx->NumJobsSubmitted()) but the jobs have not yet completed (because uploading is still in progress), this loop becomes equivalent to:

while (1) {
  // do nothing
}

i.e. a CPU spinloop. This behaviour can be easily observed by publishing anything via a gateway: the publisher will hit 100% CPU usage within cvmfs_swissknife sync, and attaching gdb shows that one thread is just spinning inside SessionContext::UploadLoop().

[mcb30]

A similar problem exists in S3Uploader::MainCollectResults(), which uses the same overall approach to loop termination.