DATAUP-679: Update job to queued directly after submit #430
DATAUP-679: Update job to queued directly after submit #430MrCreosote merged 3 commits intodevelopfrom
Conversation
If many jobs are submitted in a batch, the jobs can start running before being updated to queued, which means the jobs never get a queued timestamp.
|
I realize from our discussion on the prior PR that there is an improvement that could be made re Kafka queuing - all the kafka messages get sent after all the jobs are queued (which is no different than before). This could be improved by sending the message right after each job, which reduces the chance the job will switch to running prior to sending the message. If that happens the queued message won't get sent based on the current code. Again this is no worse than before. Edit: alternate approach is to record whether there was an update or not for each job (looking at |
|
Another thing we could do is switch https://github.com/kbase/execution_engine2/blob/develop/lib/execution_engine2/sdk/EE2Runjob.py#L989 to use the new mongo method. If we also do the fix above we could probably use the same method for the mongo update & kafka message for both batch and normal jobs. |
|
Another another thing I could do is delete the batch queued method in MongoUtil if you want, since it's now unused. |
Description of PR purpose/changes
If many jobs are submitted in a batch, the jobs can start running before
being updated to queued, which means the jobs never get a queued timestamp.
Jira Ticket / Github Issue
Testing Instructions
Dev Checklist:
Updating Version and Release Notes (if applicable)