Support CUDA execution graphs

[neoblizz]

Added with CUDA 10.0, cuda graphs (though really terribly named as there are 7 "graph" things that cuda provides), is a take on task graphs within CUDA's programming model.

What's really cool about cuda graphs is the performance benefits you get when relying on task graphs to set things up for repeated use. Examples that benefit from this are machine learning (surprise, surprise), or any iterative-converging algorithm.

I will add a plan-of-attack soon to this issue for review.

Related issue #77.

[eyalroz]

Note that we can't (or at least, should not) offer merely imperative "add node" and "add edge" etc. It should be possible to pass graphs. or iterators over the structure of the graph, from common graph representation libraries (not specific ones, something generic and template based), and have that graph be fed in.

And even if we don't make it that far already in the beginning, we must at least lay the foundation for that to be possible later on.

[neoblizz]

An update, I still have this in my todo list. I plan to work in near future after some important deliverables are taken care of.

[eyalroz]

@neoblizz : Sure, whenever you have the time.

I should perhaps qualify what I said earlier: We can start with the simple imperative wrappers, then move up into the more abstract iterating-over-graphs version later.

[eyalroz]

I'm thinking of starting work on this myself.

[eyalroz]

So, I now have an untested, but compiling, initial version of wrappers for the graph functionality, on a new branch - see the link below.

It's not quite complete, especially w.r.t. "executable" graphs' methods for getting and setting node parameters; but some graph template methods are missing too.

I would appreciate feedback. @codecircuit - maybe you might also be interested in checking this out?

[eyalroz]

@neoblizz : So, I'm almost done. I mean, the graph support is working, but I have a couple of data structures which currently have ugly names; and node creation API could stand some beautification. But - feedback would be very much appreciated.

[neoblizz]

But - feedback would be very much appreciated.

This is awesome! Thanks for sharing, I'll take a look.