Create batch job object

[Kaixhin]

A batch job should be its own object stored in the database, with a reference to its parent project. This should allow querying of the status of jobs in the batch.

[Hiyorimi]

Can I suggest using some queue mechanism, like bull?

I would like to implement it, because I need to dispatch experiments in a batch and check the results couple of hours later, so what do you think about adding some queue-manager as a dependance ?

[Kaixhin]

I like the idea of a proper/fault-tolerant queue system, though if you know of a decent maintained solution which can be backed by MongoDB rather than having to introduce Redis that would help keep dependencies down. As for Redis-backed, any preferences between Kue and bull? I'll look at them both myself later.

Ideally a batch would be an object, with a project ID key. Probably experiment keys as well, because it would be useful to see which experiments failed in a batch, and then be able to retry them via a GUI. Don't see a need to refer to a batch from experiments or the project itself though.

As for a PR, I suggest working on the new queue branch. If you could have a quick look at these notes that would help as well 👍

[Hiyorimi]

Well, I used bull and found it really nice, so I have a preference towards it. As a matter of fact, I don't think that a separate object would be a nice idea, since having queue will allow user to plan a set of experiments, which will be executed when capacity for them is available.

So, some intermediate experiment state, rather than 'succeeded' or 'failed' would be used to mark queued experiments, waiting to be run.

Speaking of a backed, there is a nice article about using MongoDB as a queue backend, so it might be a worthy idea.

[Kaixhin]

OK have a look at using MongoDB as the backend, and fall back to bull if that doesn't work out.

Open up a PR for the queue branch, and then I'll be able to keep up to date with your commits.