Create a "Graph-Building" Model

[Micky774]

Create an algorithm/agent which initializes a starting graph, then modifies the graph via discrete actions (e.g. split/combine nodes, change node/edge features). Initially I suggest an "energy" or "cost" based algorithm which optimizes (MLL-energy). The history of the agent over the course of a single video can also serve as a representation feature.

[Micky774]

This seems like it would lend itself well to a reinforcement-learning framework