Add Memory Selection to SST, and prototype a mechanism for running st…

[eisenhauer]

…andard BP file tests in staging-common with SST

[scottwittenburg]

@eisenhauer Here is what I use to reproduce github actions builds locally. Please feel free to reach out if you run into trouble.

Steps

Make a root directory where you'll put the source trees and do the builds, this will be mounted as /builds when you run docker.

mkdir <path-to-some-working-dir>

Check out the code you want to test

Source is checked out twice in ci, once from your PR changes, once from master. Most of the time, I just check out the same branch twice.

cd <path-to-some-working-dir>
git clone <path-to-your-adios2-source> gha
git clone <path-to-your-adios2-source> source

Run the docker container

Run the container where the name matches the compiler you want to test. Note that images for the different gcc/clang compilers are based on the same underlying image where all three spack environments are available. So using the gcc10 image, you could run serial, ompi, or mpich tests.

docker run --rm -v <path-to-some-working-dir>:/builds -ti adios2:ci-spack-ubuntu20.04-gcc10

Once you're inside the container with the compiler you want to test, you have to set a bunch of variables as github actions would do based on the yaml.

Set the parallel value

Pick one of these based on what you want to test:

export GH_YML_MATRIX_PARALLEL=serial
export GH_YML_MATRIX_PARALLEL=mpi
export GH_YML_MATRIX_PARALLEL=mpich

Set the compiler

Here, you should choose from these, and make sure to match the container you're running:

export GH_YML_MATRIX_COMPILER=gcc8
export GH_YML_MATRIX_COMPILER=gcc9
export GH_YML_MATRIX_COMPILER=gcc10
export GH_YML_MATRIX_COMPILER=gcc11
export GH_YML_MATRIX_COMPILER=clang6
export GH_YML_MATRIX_COMPILER=clang10
export GH_YML_MATRIX_COMPILER=oneapi
export GH_YML_MATRIX_COMPILER=icc

Set a branch name

This build/test will get reported to cdash, so pick a branch name that can help you identify what you were testing when you look at the results in cdash, e.g.:

export GITHUB_REF_NAME="mpich_perf_testing_ch4_ofi"

Do the build/test

The remaining steps set a bunch of variables, and eventually do the configure/build/test steps as they would be done in gha:

cd /builds
export GITHUB_WORKSPACE=/builds
export GITHUB_PATH=/dev/null
export GITHUB_JOB=test
export GITHUB_EVENT_NAME="test_pull_request"
export GH_YML_BASE_OS=ubuntu
export RUNNER_TEMP="/"
export GH_YML_MATRIX_OS=ubuntu20.04
export GH_YML_JOBNAME=${GH_YML_MATRIX_OS}-${GH_YML_MATRIX_COMPILER}-${GH_YML_MATRIX_PARALLEL}
gha/scripts/ci/gh-actions/linux-setup.sh
cp /.local/bin/ninja /usr/bin/ninja
gha/scripts/ci/gh-actions/run.sh update
gha/scripts/ci/gh-actions/run.sh configure
gha/scripts/ci/gh-actions/run.sh build
gha/scripts/ci/gh-actions/run.sh test

[eisenhauer]

@scottwittenburg So, we've sorted the problem and I'm going to merge this PR, but the nature of the problem maybe points to other issues. When we have an MPI-enabled build, we build both serial and mpi versions of the ADIOS library (and of many of the tests in testing/engines/bp). However, we only ever build one version of the SST runtime (the bulk of the SST engine), which is its own library largely for historical reasons. So even if we have a serial application, built and linked with a serial version of the ADIOS library, SST doesn't know that. Specifically, in this circumstance SST is built with and depends upon MPI, and it also considers @vicentebolea 's MPI data plane a viable possibility. The deadlock I was seeing was happening because the MPI dataplane was trying to initialize MPI deep in the data transport, having noticed that it hadn't been initialized at the application level (this didn't go well). The fix for this PR was to always use the MPI version of the test if it had been built, even if we were only doing a 1 to 1 SST test.

But I'm wondering if this is sufficient. Because ADIOS always depends upon SST (unless disabled) and SST always depends upon MPI if it's present, building a non-MPI version of the higher level ADIOS library to avoid an MPI dependency seems moot because it'll inherit that from SST. Should we be building a serial and mpi version of SST also? Is it time to abandon the SST-is-a-separate-library thing (no good reason to keep it that way)? Or is there some advantage in the current situation because a one-rank application might use the MPI data plane to connect to an MPI application? (If so, we should test this and make sure it works rather than deadlocking on some platforms.). Anyhow, something to discuss, perhaps when @vicentebolea is back. I believe that Chuck did a lot of the work behind building Serial and MPI versions of ADIOS, and I'm not sure of the reasoning behind not extending that to SST (and the MPI data plane was not a wrinkle that existed at the time).

[caitlinross]

@vicentebolea this is the PR we're talking about