APT whitelist request for mpich-bin

[BanzaiMan]

This replaces travis-ci/travis-ci#4339.

The original text by @QuLogic follows

---

#4164 conveniently added not only libhdf5-openmpi-dev, but also libhdf5-mpich-dev and libhdf5-lam-dev. The trouble is that mpich-bin and lam-runtime are not whitelisted, so you can't run any MPI-compiled binaries.

[BanzaiMan]

Ran tests and found setuid bits by purely textual search. Further analysis is required.

If these are found to be benign, add:

libmpich-dev libmpich12 libmpich2-dev mpich mpich-doc mpich2 mpich2-doc libmpl-dev libmpl1 libopa-dev libopa1

See https://travis-ci.org/travis-ci/apt-whitelist-checker/builds/72545568.

[jhamman]

@BanzaiMan - I'm wondering if you could provide update on the progress of this issue. I'm working on a project that needs mpich2.

[jhamman]

bump?

[jmerkow]

Any update on this whitelist?

[garlick]

I'd also really like to get mpich2. libmpich2-3, and libmpich2-dev whitelisted.

[gnzlbg]

I submited a PR for mpich2 and libmpich2-dev (that should pull libmpich2-3 also)

[gnzlbg]

@BanzaiMan those occurences of setuid are bening, two are just comments, and the other is in mpd which should not really be used (one should use hydra instead).

[zbeekman]

I would really like to be able to install MPICH, MVAPICH and OpenMPI from container based builds. Any status update, @BanzaiMan ?

[miho]

+1 We need this too. Are there still unresolved issues with these packages?

[justusc]

@BanzaiMan I saw that you have looked at this several times in other issues. Any progress?
travis-ci/travis-ci#3955

[jeffhammond]

Ran tests and found setuid bits by purely textual search. Further analysis is required.

There are three hits for setuid/setgid. Below is my analysis of them. Line numbers reference https://travis-ci.org/travis-ci/apt-whitelist-checker/builds/72545568.

Line 442

Offending text

./mpich-3.1/src/pm/mpd/mpd.py: os.setuid(uid)

Analysis

This code is called in the MPD process manager, which is the old processor manager for MPICH. It is not used by default and has been deprecated for a long time (https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_I_don.27t_like_.3CWHATEVER.3E_about_mpd.2C_or_I.27m_having_a_problem_with_mpdboot.2C_can_you_fix_it.3F).

I don't know if the MPICH binary includes all the process managers or not, but at least the default mpirun and mpiexec will use Hydra.

Line 456

Offending text

./mpich-3.1/src/pm/hydra/tools/topo/hwloc/hwloc/src/topology-solaris.c: * TODO: get cache info from prtdiag? (it is setgid sys to be

Analysis

First, this is in a comment in the source code, hence is inconsequently to the binary installation.

Second, the offending source code is part of hwloc, which is already whitelisted (https://github.com/travis-ci/apt-package-whitelist/blob/master/ubuntu-precise#L1551). Furthermore, hwloc is integrated into Open-MPI, which has already been whitelisted (https://github.com/travis-ci/apt-package-whitelist/blob/master/ubuntu-precise#L6480).

Third, this code is for Solaris. Unless Travis CI supports the Solaris operating system, it is not relevant.

Line 468

Offending text

./mpich-3.1/debian/changelog: * Make /usr/bin/mpdroot setuid. This is the default after the installation

Analysis

This text is in the changelog, which is not executable code.

If these are found to be benign, add:

I contend that all of these are benign. The only one that is even remotely relevant is in mpd.py, which is either not used or not used by default. I will investigate whether mpd.py even appears in a binary package install now.

libmpich-dev libmpich12 libmpich2-dev mpich mpich-doc mpich2 mpich2-doc libmpl-dev libmpl1 libopa-dev libopa1

See https://travis-ci.org/travis-ci/apt-whitelist-checker/builds/72545568.

[jeffhammond]

@jhamman

For what it's worth, MPICH is trivial to build from source as part of Travis, and the time to build is not onerous. You can even add --disable-fortran and/or --disable-romio to speed up your builds if you do not need Fortran bindings or MPI I/O, respectively.

# Install MPICH
if [ ! -d "${HOME}/mpich" ]; then
    wget --no-check-certificate -q http://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz
    tar -xzf mpich-3.2.tar.gz
    cd mpich-3.2
    mkdir build && cd build
    ../configure CC=$CC CXX=$CXX --prefix=${HOME}/mpich
    make -j2
    make install
else
    echo "MPICH installed..."
    find ${HOME}/mpich -name mpiexec
    find ${HOME}/mpich -name mpicc
fi

See MADNESS, or PRK for examples.

[jeffhammond]

Before we think about this any harder, let us first get the right package, meaning the binary one rather than the source one:

Fetching source package for mpich-bin
mpich-bin:
  Installed: (none)
  Candidate: 1.2.7-10ubuntu1
  Version table:
     1.2.7-10ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages
...
W: You may want to run apt-get update to correct these problems
Reading package lists...
Building dependency tree...
Reading state information...
Picking 'mpich' as source package instead of 'mpich-bin'

(that's from https://travis-ci.org/travis-ci/apt-whitelist-checker/builds/72545568).

[zbeekman]

@jeffhammond @BanzaiMan
Please don't --disable-fortran to speed up builds... Fortran is already the red-headed stepchild of modern software engineering services, and Travis-CI has great (although poorly documented) support, at least relative to other services.

[justusc]

@zbeekman The point of disabling Fortran, in the context of @jeffhammond's comment, was to speed up builds where project maintainers are forced to build MPICH themselves, as opposed to installing a package. It is certainly valid to disable Fortran wrappers when building MPICH for your project, if it does not require Fortran. If we are talking about installing packages, then it should certainly include Fortran wrappers so it is useful for everyone.

[zbeekman]

@justusc ah fair point. I read it too quickly, trying to catch up after a week of vacation.

[jeffhammond]

Just remember that Fortran modules may not be compatible across versions of
GCC (they certainly are not across different compilers), so building MPI
with Fortran is not a singular effort. There will need to be a package for
every unique gfortran module ABI. That no package manager supports this
today may be some indication of the lack of adoption of the MPI Fortran
modules...

Note also that I've written tens of thousands of lines of Fortran in my
life, so I'm by no means a bigot against Fortran - I just never call MPI
directly from Fortran, but rather through higher level libraries like
Global Arrays.

[zbeekman]

@jeffhammond yup, I know you're a Fortran guru. As I said, I read it too quickly and just wanted to make sure that any whitelisted apt package would include fortran bindings. Sorry again for miss-reading what you were saying.

[zbeekman]

@jeffhammond also, 👍 for using higher level libraries rather than interfacing directly. 😄 https://GitHub.com/sourceryinstitute/opencoarrays ++

[mjw99]

@BanzaiMan Is there any progress on this? This has been open for around half a year now and from what ascertain in the thread, the "Further analysis is required" has been answered via Jeff. Is there anything else blocking this?

[jhamman]

@BanzaiMan - bump! If we could get some feedback here, it would be much appreciated.

[thomasWeise]

@jeffhammond
Many thanks for your recommendation on how to build and install mpich. I am a rather novice Linux and Travis user, so I ran into the problem that after pasting your snippet directly into my .travis.yml file, I still could not build my mpich-based C project as it could find neither mpicc nor the headers. After some fiddelling around, I finally came with a solution.

I add the code below to my .travis.yml file in the before_install: section:

    # Install MPICH based on https://github.com/travis-ci/apt-package-whitelist/issues/406
        - currentDir=`pwd`
        - mpichVersion=3.2
        - wget --no-check-certificate -q http://www.mpich.org/static/downloads/$mpichVersion/mpich-$mpichVersion.tar.gz
        - tar -xzf mpich-$mpichVersion.tar.gz
        - cd mpich-$mpichVersion
        - mkdir build && cd build
        - sudo ../configure CC=$CC CXX=$CXX --disable-fortran --disable-romio
        - sudo make -j2
        - sudo make install 
        - cd "$currentDir"

The changes compared to your code are:

1. removed the if-then-else
2. removed "--prefix=${HOME}/mpich" (this was the reason the files were not found during my compiles)
3. added --disable-fortran --disable-romio: without --disable-fortran, I get errors during the compile/install of mpich, I then also added the --disable-romio for good measures
4. made all of this sudo, didn't test without sudo, seemed to be a good idea
5. make the mpich version a variable, just in case
6. make sure to be in the right directory afterwards (in a very primitive way, was too lazy to figure out how many .. to use for cd otherwise ^_^)

With these, it seems that I can compile with mpicc under travis.

[certik]

As @jeffhammond said, you can install mpich from source, and you can also cache the build directory, so that it doesn't have to be build every single time, as described in this blog post: https://d-meiser.github.io/2016/01/10/mpi-travis.html

For my projects, I just install openmpi, which is whitelisted already, and that works to test my codes in parallel on Travis. But it would be nice to white list mpich as well.

[gnzlbg]

The main reason I prefer to run the automatic tests with MPICH instead of with OpenMPI is its strongly typed validation layer that automatically works with clang. That is, if you try to pass a double to a function taking an MPI_INT through a void*, among other things, clang + MPICH will reject the code while with OpenMPI it will compile just fine.

There is a clang-tidy check in LLVM 3.9 that should do this independently of the MPI implementation used, but it is new, and it doesn't work through typedefs and such (there are actually two checks already, and more might come). This might turn out to be a solution in the future for those using OpenMPI, but integrating clang-tidy in an already existing project can take a lot of effort (if one wants to do it right).

[jeffhammond]

We do not need to debate Open-MPI vs MPICH here. What we need is for Travis to whitelist MPICH, not in the least because they have whitelisted packages that depend on it.

It is absurd that nothing is happening after 15 months and intense interest from users. I performed the detailed analysis requested 9 months ago (#406 (comment)), but it appears that no one will act on it.

[zbeekman]

Ping @BanzaiMan

[gnzlbg]

but it appears that no one will act on it.

FWIW I've contacted travis support multiple times saying that I would love to upgrade to paid/pro travis version if that would mean that they would provide "basic" C/C++ support, where for basic I meant "no months long test breakage due to non-white-listed new clang/gcc versions, and quickly whitelisting new releases of the fundamental C and C++ libraries that I needed (MPICH, parallel-netcdf, hdf5, ...)" (I think these are reasonable requirements).

Their answer was that they were not interested. I would hope they would change their mind. Maybe they'll do so if they see enough of a market in C and C++ testing.

[alehaa]

Is there any progress in this issue, or will it be closed as wontfix?

At the moment I'm using the OS X infrastructure for testing with MPICH, but I think this is overkill, as there has to be an entire VM booted for every run, especially because Travis OS X builds are heavily overcommitted these days.

[zbeekman]

Yes it's absurd that this hasn't been accepted yet, especially since Jeff Hammond did all of the required set uid set gid analysis.

…

On Wed, Jan 25, 2017 at 3:11 PM Alexander Haase ***@***.***> wrote: Is there any progress in this issue, or will it be closed as wontfix? At the moment I'm using the OS X infrastructure for testing with MPICH, but I think this is overkill, as there has to be an entire VM booted for every run, especially because Travis OS X builds are heavily overcommitted these days. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#406 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAREPH4k4IGk7ai-AgTEmm_gkI5GQTKLks5rV6yOgaJpZM4FffJp> .

[zbeekman]

@BanzaiMan any updates here?

[jeffhammond]

38 months is enough time to create four human beings (sequentially). Honestly, how is this ticket still unresolved?

[travisbot]

This is an automated comment.

Ran tests and found setuid bits by purely textual search. Further analysis is required.

If these are found to be benign, examine http://github.com/travis-ci/apt-package-whitelist/compare/test-apt-package-whitelist-406 and its PR.

Packages found: mpich-bin mpich-mpd-bin mpich-shmem-bin libmpich1.0-dev libmpich-mpd1.0-dev libmpich-shmem1.0-dev libmpich1.0gf libmpich-mpd1.0gf libmpich-shmem1.0gf mpi-doc mpe-source

See https://travis-ci.org/travis-ci/apt-whitelist-checker/builds/440490148 for details.