[batch] make create, delete, close not linear in jobs #6569
Conversation
batch/batch/batch.py
Outdated
| batch = await Batch.from_db(batch_id, user) | ||
| if not batch: | ||
| abort(404) | ||
| await batch.mark_deleted() | ||
| asyncio.ensure_future(batch.mark_deleted()) |
There was a problem hiding this comment.
Is the caller not getting feedback on whether mark_deleted succeeded the design we want? Similar for the others. Are we going to give the caller a way to track whether the operations they requested succeeded or not (say subscribing to an event stream)
There was a problem hiding this comment.
I changed it so we've done all of the work that is not O(n) (updating the jobs), but still return success only when we've updated the batch state in the database.
There was a problem hiding this comment.
Yep, that sounds about right.
batch/batch/batch.py
Outdated
| await j.cancel() | ||
| log.info(f'batch {self.id} cancelled') | ||
|
|
||
| async def close(self): | ||
| await db.batch.update_record(self.id, closed=True) | ||
| self.closed = True | ||
|
|
||
| for job in await self.get_jobs(): | ||
| if job._state == 'Ready': |
There was a problem hiding this comment.
Since start_job already checks for 'Ready', do we need the check here?
As an aside, it looks like we spawn 16 consumers, but in the same thread. Could we use a thread-safe Queue implementation, and spawn N threads? In particular, collections.dequeue appears to be thread safe for push/pop operations, and has performance very similar to lists (should be significantly better than Queue). This doesn't need to addressed for this PR, just something I thought could help future work.
I found this exploration by David Beazley (creator of Curio) helpful: https://www.youtube.com/watch?v=MCs5OvhV9S4
There was a problem hiding this comment.
Also, since start_job awaits on the get, do we need to await on the put? Could use put_nowait, provided the queue size is infinite.
Still, to put less pressure on the queue, I would prefer pushing (eventually, not in this PR), a single collection of jobs.
There was a problem hiding this comment.
To recap: only question regarding the "Ready" check is for this PR, queue comment is related to broader batch
- related to "Ready" question: do we potentially wait longer than necessary to start jobs that aren't ready but will be ready soon (Pending), or even never start these jobs (if
closeis never called again)? If so, it would be potentially slightly better to not check _state here at all, push the entire job list into the queue (easier to do as list in the future), and when the queue is ready to process that job in start_job, then the job is inspected for Ready state. This could protect against transitions happening in betweenclosebeing called andstart_jobgetting the item.
Also only start jobs once a batch is closed.