job-ingest: add ingest module, flux job command, libjob library #1626

Problem: DOTHEX-encoded FLUIDs can be used to map FLUIDs onto the KVS namespace, but without zero padding, flux-kvs ls doesn't display or sort them nicely. Pad each dotted hex number out to four digits.

Problem: fluid_decode (type=MNEMONIC) does not fail if presented with words not in its dictionary or a phrase too short to represent a uint64_t. mn_decode()'s inline documentation is incorrect: /* Return value: * This function may return all the value returned by mn_decode_word_index * plus the following result code: * * MN_EWORD - Unrecognized word. */ It actually returns the number of bytes successfully written to the output. Require mn_decode() to return 8 in order for fluid_decode() to be successful.

Problem: jobspec sharness test was not run. Add jobspec inputs to EXTRA_DIST and run jobspec test if ENABLE_JOBSPEC.

Problem: while there is an ENABLE_JOBSPEC Makefile conditional, there is nothing that can be tested in config.h to determine if jobspec is being built. Add HAVE_JOBSPEC to config.h

If user requests it by specifying --with-flux-security, have configure locate flux-security using pkg-config. Defines HAVE_FLUX_SECURITY in config.h Defines HAVE_FLUX_SECURITY for Makefile.am's

Add a library will provides an API for job creation, monitoring, and control. For now it contains only an interface for job submission.

Add a module that handles job-ingest.add RPCs to add new jobs to the KVS using a rudimentary form of the RFC 16 job schema foramt. The user is returned a jobid based on the FLUID proposal, which allows 64-bit id generation to occur in parallel across ranks, while retaining a loose ordering of ids based in the time submitted. KVS per-job directories are generated using the DOTHEX FLUID encoding, e.g. job.active.0000.011c.ae00.0002 The instance owner and any user with ROLE_USER may submit jobs. Jobs must be signed, and the user authenticated as the submitter (by the connector) must match the signature, but the job signature is not authenticated at ingest time. The connector-authenticated userid is recored in the KVS under the "userid" key. The signed blob is recorded under the "J-signed" key. The job submission consists of signed RFC 14 jobspec, which is validated by temporarily instantiating a C++ Jobspec object and recording any parse errors. The parsed Jobspec may be further validated in the future, for example to find resource requests that can not be fulfilled, but for now we ingest all valid jobspec. (Validation is performed in a standalone .cpp file linked against libjobspec.la. The standalone C++, which exports a validate function callable from C, is compiled with the C++ compiler; automake then knows to link job-ingest against libstdc++). The unwrapped jobspec is written to the KVS under the "jobspec" key. The module is completely event driven, and KVS overhead is reduced and ingest rate increased by batching job-ingest.submit requests that arrive toether within 10ms. A "job-ingest.submit" event is generated after the KVS commit which contains an array of new jobids. This can be consumed by the job-manager module in the future, which will handle listing jobs for users, and informing the scheduler when new active jobs have been submitted.

Add a front end command that will eventually contain the primary user interfaces for submitting and managing jobs such as list, run, or cancel. For now, it contains two subcommands for testing the job-ingest module: submitbench - test ingest throughput, maintaining a minimum number of outstanding RPCs id - convert jobid's between representations.

Ad a test_under_flux "profile" for testing the new execution system, starting with job-ingest module.

Pull in flux-security-0.2.0 via the travis-dep-builder script, then add --with-flux-security to some builders in the travis build matrix.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

job-ingest: add ingest module, flux job command, libjob library #1626

job-ingest: add ingest module, flux job command, libjob library #1626

Commits on Aug 28, 2018