tighten up PMI implementation for OpenMPI

[garlick]

This PR fixes a couple more issues that came up while building openmpi support for flux.
Some general cleanup, environment cleanup as discussed in #923, implementing the clique functions, and suppressing a log for failed PMI_KVS_Get() since that's not necessarily a flux error.

All pretty minor.

[coveralls]

Coverage increased (+0.01%) to 76.323% when pulling d4430bd on garlick:pmi-pkg into e84a5b5 on flux-framework:master.

[codecov-io]

Current coverage is 76.01% (diff: 50.00%)

Merging #926 into master will increase coverage by 0.01%

@@             master       #926   diff @@
==========================================
  Files           149        149          
  Lines         26006      26010     +4   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          19764      19772     +8   
+ Misses         6242       6238     -4   
  Partials          0          0          

Diff Coverage	File Path
0%	src/modules/wreck/wrexecd.c
•••••••••• 100%	src/common/libpmi/simple_client.c
•••••••••• 100%	src/common/libpmi/pmi.c
•••••••••• 100%	src/broker/broker.c

Powered by Codecov. Last update 82e5275...9d3b730

[coveralls]

Coverage increased (+0.02%) to 76.328% when pulling 03dbebe on garlick:pmi-pkg into e84a5b5 on flux-framework:master.

[grondo]

@garlick, this all seems pretty straightforward. I think you just need to rebase on current master if everything else is good.

[garlick]

Just rebased on current master.

The only thing I'm uncertain about is setting TMPDIR to scratch-directory-rank (to avoid shared memory name collisions when running multiple flux ranks per node). Think that has any negatives?

[coveralls]

Coverage increased (+0.007%) to 76.309% when pulling 29b7c58 on garlick:pmi-pkg into 82e5275 on flux-framework:master.

[grondo]

The only thing I'm uncertain about is setting TMPDIR to scratch-directory-rank (to avoid shared memory name collisions when running multiple flux ranks per node). Think that has any negatives?

It does kind of seem strange to do this to all user's jobs by default, but I can't think of an actual problem with it. It would be nice if we could suggest to openmpi that they add node-rank to the shmem filename so that it works without outside intervention.

[garlick]

Let's go with this for now. I've probably given Ralph enough headaches for one week.

[grondo]

Does scratch-directory-rank get removed after session exit? If so, it might be surprising to have TMPDIR disappear after exit?

I do notice from the faq:

8. Where is the file that sm will mmap in?
   The file will be in the OMPI session directory, which is typically something like /tmp/openmpi-sessions-myusername@mynodename* . The file itself will have the name shared_mem_pool.mynodename. For example, the full path could be /tmp/openmpi-sessions-myusername@node0_0/1543/1/shared_mem_pool.node0.

To place the session directory in a non-default location, use the MCA parameter orte_tmpdir_base.

Can we try something like an env variable OMPI_mca_orte_tmpdir_base (I forget exactly how those MCA environment variables are named)?

[garlick]

Excellent point! Yes it does get removed.

If something like the above suggestion works that is much better! Let me poke at that.

[garlick]

OK just forced a push with that change. Seems to work!

[coveralls]

Coverage increased (+0.01%) to 76.316% when pulling 9d3b730 on garlick:pmi-pkg into 82e5275 on flux-framework:master.

[grondo]

Cool! Merging