Add dynamic scheduler control

[jpsamaroo]

Allow thunks to dynamically control the scheduler

Todo:

• exec!
• add_thunk!
• get_dag_ids
• Change API to use task_local_storage for scheduler handle
• Export dynamic API from Sch
• Test add_thunk! and wait/fetch! better
• Update docs for TLS API
• Expose scheduler control through Context somehow For a future PR
• Simplify RPC interface to just use exec!
• Change set_return! to wait/fetch, to allow one thunk to wait on another
• Use special ThunkID instead of special Arg

[jpsamaroo]

@shashi this is the foundation of what I was discussing with you in Slack today.

Basically, we can expand this code to support modifying the DAG at execution time, with the scheduler support, of course.

[coveralls]

Coverage decreased (-0.4%) to 65.976% when pulling 7658756 on jps/dynamic-thunks into 61ff302 on master.

[jpsamaroo]

Bump. Any objections to the idea or implementation?

[jpsamaroo]

Thanks for the review! I'll address your recommendations once I come back to this PR, and I also think I'll hold off on merging this until we expose a few more controls through this API, so that we can see what it will ultimately look like. I'd like to at least support adding new thunks to the DAG, and querying the scheduler state and the full DAG structure from this interface.

[jpsamaroo]

Note to onlookers: locking was added to allow users to add their own process_dynamic! methods which interact with the scheduler state without stepping on the scheduler's proverbial toes. Finer-grained locking will probably be necessary later on to enable more efficient scheduler modifications. Providing a more fine-grained semi-internal DAG modification API should make concurrent modification much easier and more efficient.

[jpsamaroo]

I don't think I'm going to keep the dynamic kwarg; it's ugly and makes the thunk API ugly. We should be able to instead pass the scheduler handle through task_local_storage, which will let us keep thunks working as normal. The only price we pay is getting the scheduler handle each time, which should only be the cost of a dynamic dispatch; this should be insignificant given that most operations on the handle involve network traffic.