Partial reruns

[pattonw]

Modifying parts of your pipeline is very common, and rerunning a full grid search can be very impractical. To support partial reruns we should think of the following:

• Identifying (automatically based on config changes) which parts of the dacapo run need to be recomputed (validation results only or the full network training loop)
• Provide ergonomic options for users to define which jobs should be rerun (i.e. a bug was fixed in a specific model, no need to retrain all models, just the one that was fixed)
• Throw warnings when code has changed that potentially invalidated specific results ( code hashing )