Concept code for dependency management and custom pipelines

[lasalvavida]

Establishes a way for specifying dependency hierarchies and shows how the configuration would look for quantizeAttributes().

@pjcozzi, @lilleyse, could you take a look at this when you get a chance and tell me what you think?

Pipeline stages put their dependencies in a JSON file in lib/dependencies that matches their name.
You can then call runStages() with an Array of stage names and it will run them in order accounting for pre and post dependencies. The default.json configuration in bin is an example of a pipeline configuration that specifies a list of stages.

@JoshuaStorm, your input here will also be valuable in terms of finishing the implementation.

[coveralls]

Coverage decreased (-0.3%) to 93.862% when pulling f3679d3 on json-dependencies into 223b707 on master.

[lasalvavida]

This is going to require some thought for tests. Do we explicitly call necessary dependency functions in the specs, like I did here, or do we split out runStage from runStages and call that in the specs?

edit: something like

var quantizeAttributes = function(gltf, options) {
    runStage(gltf, 'quantizeAttributes', {}, {}, [], options);
};

[pjcozzi]

To make development easier and improve cohesion, it would be better to group the before/after dependencies in the same file as each stage.

For example, instead of a stage being just a function, it can be a class that has a function to execute the stage and array properties for the before/after dependencies.

I can provide a more specific example if you need.

[pjcozzi]

I briefly looked at the implementation and it looks OK.

The two main things to carefully account for and test are:

• Stage dependencies with their own dependencies...with their own dependencies...and so on.
• Not executing the same stage more than once.
  • However, we may learn of cases where we actually want to do this, e.g., "compact buffers now even though it happens later because doing now makes the rest of the pipeline faster." Let's add support for this when we need it.

[lasalvavida]

The two main things to carefully account for and test are:

• Stage dependencies with their own dependencies...with their own dependencies...and so on.
• Not executing the same stage more than once.
  • However, we may learn of cases where we actually want to do this, e.g., "compact buffers now even though it happens later because doing now makes the rest of the pipeline faster." Let's add support for this when we need it.

I think I actually did a pretty good job accounting for those cases already. Take a look at the included spec.

[pjcozzi]

history is only used for testing, right?

You should be able to remove it and use spies instead. @lilleyse can advise if you need help.

See http://jasmine.github.io/2.4/introduction.html#section-Spies

[lasalvavida]

history is only used for testing, right?

It is, but it might be useful to rework this a bit so that it resolves dependencies, outputs the list of necessary stages in order (history currently), and then executes that.

[pjcozzi]

Ah, yes, that could be a nice implementation.

[JoshuaStorm]

What exactly is the expectation for producing the custom pipeline JSONs on the client side? Are we expecting users to just hand modify a JSON to set up a pipeline? Just want to make sure I am on the right page here.

Also, do we plan on setting up the pipeline so full configurations can be made via command line (without JSON configuration)? I imagine that would become quite cumbersome on the user

[pjcozzi]

Are we expecting users to just hand modify a JSON to set up a pipeline?

Yes. End users of the command-line tool would do this.

Node.js developers would also call a top-level pipeline function and pass in the same JSON object.

Also, do we plan on setting up the pipeline so full configurations can be made via command line (without JSON configuration)? I imagine that would become quite cumbersome on the user

To start, I would say no. We could include a handful of typical pipelines in .json files.

Longer-term, if this is needed, PDAL has a pretty elegant approach. Did you review it?

[JoshuaStorm]

I did have a look at PDAL's approach, but wasn't really sure if that would be the best approach while the pipleline only has a handful of stages.

[pjcozzi]

Is it reasonable to come back to this and finish it off once the other outstanding PRs are merged?

[pjcozzi]

This is superseded by #233. Thanks for looking into this, @lasalvavida.

@shehzan10 can you review this to see if anything is useful for #233?