Add configurable options to scheduler

[jpsamaroo]

This PR allows the scheduler to take options as keyword args. To start with, I'm adding an option called "single" which forces all work onto a single worker (master or a specific worker) by worker id.

[jpsamaroo]

I removed the other planned options for now to make this PR easier to review.

[aviks]

Are any of these functions performance critical? Should we worry about the overhead of kwargs?

[jpsamaroo]

Should not have any noticeable impact given that they stop propagating once they hit compute_dag (the entrypoint to the scheduler). At that point, scheduler and data transfer overhead will vastly outweigh the potentially minor cost of some dynamic dispatch or kwarg sorting that could occur in the upper levels. Also, constant propagation should remove any cost if the options are set explicitly by the user (not automatically computed by some algorithm, which would probably foil constant prop).

[andreasnoack]

From my own experience, sending arbitrary kwargs around like that can be quite tricky when things go wrong because you'll only see deep down the call chain. So are you sure this is the best option here?

[jpsamaroo]

If there wasn't the current scheduler plugin architecture built into Dagger (which I will be making good use of), then I would completely agree with you and only thread the kwargs that're needed. But with the plugin architecture being a thing, and different schedulers having the potential for different relevant options (multithreading, GPU support, etc.), we need some way to be generic about scheduler configuration.

As an alternative, we could extend the plugin API with a way to create a scheduler-specific struct that can be passed as a single kwarg which will contain the appropriate options. If you'd prefer that sort of API, I will change this PR to implement that approach instead.

[andreasnoack]

I suspect the latter solution would be simpler to manage down the road while for former is faster to get going right away. I'll let you decide since you'll be the one to actually do the work so if you prefer to stick with the current approach let us know and I'll merge.

[jpsamaroo]

I'd rather do this right the first time, so I'll go with the latter (struct) approach. I'll let you know once I've made the appropriate changes. Thanks for the helpful review :)

[jpsamaroo]

@andreasnoack let me know if the latest changes are to your liking when you get the chance to review.

[jpsamaroo]

Thanks for the review @andreasnoack ❤️