[WIP] Added Graph.get_subnetwork for constructing subnetworks

[mreppen]

This is work on #487

This PR contains code for extracting a subnetwork from an existing network.

Some examples:
Original network:

let nn = Graph.(
  let inp = input [| 2 |] in
  let x1 = lambda ~name:"x1" ~out_shape:[|1|] (fun x -> Algodiff.Maths.get_slice [[]; [0]] x) inp in
  let x2 = lambda ~name:"x2" ~out_shape:[|1|] (fun x -> Algodiff.Maths.get_slice [[]; [1]] x) inp in
  let f1 = fully_connected ~name:"f1" 1 x1 in
  let f2 = fully_connected ~name:"f2" 1 x2 in
  let sum = add ~name:"sum" [| f1; f2 |] in
  sum |> get_network)

let f1nn = Graph.get_subnetwork ~in_names:[|"x1"|] (Graph.get_node nn "f1")

f1nn can then be used in Graph.model f1nn with input of shape [|1|]. Without ~in_names, it uses the inputs of nn, so in this case Graph.get_subnetwork (Graph.get_node nn "f1") has input shape [|2|].

In this example, Graph.get_subnetwork (Graph.get_node nn "sum") has the same structure as nn. An exception is raised if ~in_names does not contain all necessary inputs:

# Graph.get_subnetwork ~in_names:[|"x1"|] (Graph.get_node nn "sum");;
Exception:
Failure "Owl_neural_graph:get_subnetwork Subnetwork depends on input input_0".

Comments:

• The code has 3 steps: 1. Recursively traverse the subgraph and create new nodes. 2. Sort the topology using the original network. 3. Connect the new nodes.
• Currently does not copy the neurons. Which are the use cases for copying/not copying?
• Currently only creates a subnetwork with one output. Extending to multi-outputs should be trivial.
• Fails if ~in_names contains nodes that the output does not depend on, e.g. Graph.get_subnetwork ~in_names:[|"x1"; "x2"|] (Graph.get_node nn "f1"). Should this be adjusted to include x2 as an unconnected input?
• Other approaches mentioned in Chaining networks and extracting intermediate calculations in Neural.Graph #487 might still be neat to have.
• I have coded naïvely, so likely one could add asserts and otherwise improve the code.

[jzstark]

Sorry for the delay. This new feature looks really good and helpful.

The suggraph function is similar to the graph cutting in TensorFlow for a given node, and perhaps it would be great if the inputs of a given output can be automatically found so that users don't have to keep tracking which inputs are required in a large graph.

One case I can think of for copying is that the network weights can be further changed, such as fine tuning the parameters in training. In that case create the subgraph as a seperate network maybe necessary.

Perhaps we could leave some notes as comment of the function about things to be done such as multi-output, assertions in code, and other possible future improvements.

One last thing is that currently Owl is using ocaml-format for a unified coding style. So running make format would be helpful.

[mreppen]

Thanks for the feedback. I have updated the formatting.

About the inputs: If you do not supply in_names, it takes the same inputs as the original network (trivial "autodetect"). Making it find the smallest set of inputs needed should not be a problem, but would require a little rewriting.

Nevertheless, I find in_names useful, as it allows me to specify where I want the input nodes of the subnetwork. In the above example I can then input data after the slice instead of before.

One case against copying is that it leaves no way of obtaining one with references, whereas not copying still leaves the door open to afterwards copy. I don't know if there is that much of a use case for references in the end, but if it makes sense, one could expose two different functions in the interface or offer another optional argument.

I will have a look at autodetecting the minimal set of inputs and doing multiple outputs. When there is a final set of features, I can think about where assertions would be useful.

Life is a bit busy the next two or so weeks, but I'll let you know when I have something.

[ryanrhymes]

@mreppen Looks good to me, great work! Shall we wait for autodetection thing or that can be submitted as another PR? I am fine with merging this one. @jzstark what's your opinion?

[mreppen]

@ryanrhymes I think I have a working version for autodetection, so I think it is worth waiting. I just need to have another look at it. However, the code (also the current PR) relies on names being unique, which is violated by Graph.inputs (see #493 )

[ryanrhymes]

@mreppen ok, ping me when you think it is ready, thank you!

[mreppen]

@ryanrhymes, this should be a correct version with autodetection of inputs. I also renamed in_names to make_inputs, which hopefully better explains the function.

I am fine with merging at any point. I could make further changes here or in separate PRs.

Some things left on my list:

• As the function returns a structure with references, would it make sense to append an underscore? Or have I misunderstood the Owl notation?
• I have a version with multiple outputs too. Would that be a separate _array function like lamda_array or should the current signature be rewritten?

Sidenote: As mentioned, this relies on the #493. @ryanrhymes, you added a verified commit. What happens to that after a rebase on the master branch? In this case it does not matter much, but I'm curious to know.