need mechanism to register new validator and feasibility checks with job-ingest

[grondo]

We need to define a method for a loaded scheduler or other instance component to register a new validator (or validators) and/or feasibility checks with the job-ingest module, so that jobspec not supported by the current instance can be rejected with a meaningful message at ingest.

One current proposal is to have a "configuration object" in the kvs, on which the job-ingest modules could place a watch and get updates to configuration. One issue with this is how to support the possibility of multiple updates to configuration causing a race. Perhaps the ingest module (or some other agent) could support a service (on rank 0 only?) to ensure a single update at a time, with a simple add and remove interface. (In fact, if done via a separate module, maybe this could be the start of a more generic dynamic configuration api for instances?)

Beyond the configuration, we have to determine what sorts of validators are supported by the ingest module. For the near term, it may be enough to simply swap out the json schema validator for a different schema, longer term, the following features would be nice

• support addition of scripts/executables that can perform validation in parallel (all checks must pass) using the current workcrew
• registration of a messaging endpoint that is interested in validating jobs (e.g. scheduler or an agent of the scheduler could perform a feasibility check on the jobspec)