Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flux job submit plumbing command, load new job modules in rc1 #1918

Merged
merged 3 commits into from Jan 15, 2019

Conversation

@garlick
Copy link
Member

commented Jan 11, 2019

This PR moves flux job submitbench to a standalone, noinst test program, and replaces it with flux job submit a "plumbing" command that submits JSON jobspec.

Sharness tests are updated to use one of the two job submission interfaces.

In addition, sharness tests that formerly forced the "none" security mechanism when configured --with-flux-security no longer do that, now that munge is being started in our travis docker images.

To make it easier to work with the new job-ingest and job-manager modules, add them to rc1/rc3 so that they are automatically loaded/unloaded.

Finally, fix a bug introduced in #1913, which moved the C++ jobspec validator without updating its path in code that checks for its existence used as evidence that job-ingest should be expected to reject invalid jobspec.

garlick added 2 commits Jan 10, 2019
Problem: previous "cleanup" of libjobspec renamed
the validator executable, but the ingest sharness test
was not updated, so conditional test of invalid jobspec
ingest was not running.

Change the sharness test to look for the validator
executable by its new name.
Load the functional portions of the new exec system
by default, as we begin to deprecate the old one.
@@ -18,6 +18,9 @@ flux module load -r 0 userdb ${FLUX_USERDB_OPTIONS}
# make flux wreckrun/submit -onokz the default (override w/ -o kz):
flux wreck setopt nokz

flux module load -r all job-ingest
flux module load -r 0 job-manager

This comment has been minimized.

Copy link
@grondo

grondo Jan 15, 2019

Contributor

Just making sure that adding the job-ingest and job-manager modules to the default rc1 script was meant to be part of this PR? (If so that is something to note for release notes...)

This comment has been minimized.

Copy link
@garlick

garlick Jan 15, 2019

Author Member

Yes it was intended. I'll amend the description.

@garlick garlick changed the title Add flux job submit plumbing command Add flux job submit plumbing command, load new job modules in rc1 Jan 15, 2019
@garlick

This comment has been minimized.

Copy link
Member Author

commented Jan 15, 2019

I think I'll tweak flux job submit so it reads from stdin if no jobspec is provided on the command line.

@codecov-io

This comment has been minimized.

Copy link

commented Jan 15, 2019

Codecov Report

Merging #1918 into master will increase coverage by <.01%.
The diff coverage is 93.93%.

@@            Coverage Diff            @@
##           master   #1918      +/-   ##
=========================================
+ Coverage    80.1%   80.1%   +<.01%     
=========================================
  Files         195     195              
  Lines       35109   35080      -29     
=========================================
- Hits        28123   28100      -23     
+ Misses       6986    6980       -6
Impacted Files Coverage Δ
src/cmd/flux-job.c 89.57% <93.93%> (-0.85%) ⬇️
src/modules/kvs-watch/kvs-watch.c 78.88% <0%> (-0.89%) ⬇️
src/common/libflux/message.c 81.39% <0%> (-0.25%) ⬇️
src/cmd/flux-module.c 83.96% <0%> (+0.23%) ⬆️
src/modules/connector-local/local.c 74.81% <0%> (+1.03%) ⬆️
src/modules/job-ingest/job-ingest.c 71.25% <0%> (+1.18%) ⬆️
@grondo

This comment has been minimized.

Copy link
Contributor

commented Jan 15, 2019

This seems great!

Is it ready to go in once incremental development is squashed?

@garlick

This comment has been minimized.

Copy link
Member Author

commented Jan 15, 2019

Sure, let me do that.

Problem: the "submitbench" subcommand of flux job
does not really belong in a user-facing program.

Create a standalone, noinst program that contains
the ingest throughput test, then rename flux job
submitbench to flux job submit and simplify it so
that it only handles one job submission at a time.

In flux job submit, if no jobspec argument is present,
presume that it should be read from stdin, so
"flux job submit" is equivalent to  "flux job submit -".

Convert sharness tests to call either submitbench
or flux job submit as appropriate.

Cleanup:  now that travis-ci is starting the munge
daemon to support the default signing mech in
flux-security, there is no need to force all jobs
submitted in test to use --sign-type=none when
configured with --flux-security.
@garlick garlick force-pushed the garlick:job_plumbing branch from 0cdb909 to d60f31e Jan 15, 2019
@grondo grondo merged commit 07fe98d into flux-framework:master Jan 15, 2019
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@garlick garlick deleted the garlick:job_plumbing branch Jan 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.