try using external find for automake and autoconf

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Dockerfile

@@ -54,6 +54,7 @@ RUN git clone --depth 1 -b vsoch/db-17-splice-july-25 https://github.com/vsoch/s
     spack config add 'modules:default:enable::[]' && \
     spack config add config:source_cache:/cache/spack-cache && \
     /spack/bin/spack compiler find && \
+    spack external find automake autoconf && \
     mkdir -p /results
 
 # Add scripts

README.md

@@ -57,6 +57,9 @@ $ python scripts/submit_jobs.py ./splices --dry-run --single
 # Generate docker run commands to manually test (with default paths)
 $ python scripts/submit_jobs.py ./splices --docker --dry-run
 
+# One command per top level package
+$ python scripts/submit_jobs.py ./splices --docker --dry-run --single > commands.txt
+
 # Limit to 10
 $ python scripts/submit_jobs.py ./splices --docker --dry-run -N 10
 
@@ -67,6 +70,14 @@ $ python scripts/submit_jobs.py ./splices --docker --spack-opt ./spack-opt --cac
 and of course you can shell into any container with the same binds to do the same.
 
 
+### Generating subset with tests
+
+1. I modified experiments.yaml -> experiments_with_tests.yaml in scripts/generate_commands.py
+2. `docker build -t ghcr.io/buildsi/spliced-experiment .`
+2. `mkdir -p with_tests/splices`
+3. `docker run -v $PWD/cache:/cache -v $PWD/with_tests/splices:/splices -it ghcr.io/buildsi/spliced-experiment:latest spack python /code/scripts/generate_experiments.py /code/experiments_with_tests.yaml /splices/`
+
+
 ## Running the Experiment
 
 ### Clone the repository

manual-runs.md

@@ -6,27 +6,31 @@ Welcome to the new game of...
 
 This is a manual test of installing the main packages (to be spliced) - if this fails, there is no point attempting them in the experiment. Spack MUST build and splice them to get any kind of result. From the below, I've derived the following list of packages we can run/test in GitHub actions:
 
-
- - [ ] qthreads/numactl ([run]())
- - [ ] hdf5/pkgconf ([run]())
- - [ ] hdf5/zlib ([run]())
- - [ ] raja/blt ([run]())
- - [ ] raja/camp ([run]())
- - [ ] upcxx/python ([run]())
- - [ ] arborx/kokkos ([run]())
- - [ ] legion/zlib ([run]())
- - [ ] bolt/argobots ([run]())
- - [ ] bolt/autoconf ([run]())
- - [ ] bolt/automake ([run]())
- - [ ] bolt/libtool ([run]())
- - [ ] superlu/openblas ([run]())
- - [ ] darshan-util/autoconf ([run]())
- - [ ] darshan-util/automake ([run]())
- - [ ] darshan-util/libtool ([run]())
- - [ ] darshan-util/m4 ([run]())
- - [ ] darshan-util/zlib ([run]())
- - [ ] swig/pcre ([run]())
- - [ ] swig/pkgconf([run]())
+ - [x] qthreads/numactl ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2762890515))
+ - [x] hdf5/pkgconf ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2764243645))
+ - [x] hdf5/zlib ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2764243977))
+ - [x] raja/blt ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2762946012))
+ - [x] raja/camp ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2762946312))
+ - [x] arborx/kokkos ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2766904278))
+ - [x] bolt/argobots ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2766496849))
+ - [x] bolt/autoconf ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2766495801))
+ - [x] bolt/automake ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2766494500))
+ - [x] superlu/openblas ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2767052014))
+ - [x] darshan-util/autoconf ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2767479049))
+ - [x] darshan-util/automake ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2767480400))
+ - [x] darshan-util/libtool ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2767481780))
+ - [x] darshan-util/m4 ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2767482219))
+ - [x] darshan-util/zlib ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2767482631))
+ - [x] swig/pcre ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2767484624))
+ - [x] swig/pkgconf([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2767485056))
+
+
+The following packages were attempted but killed the worker (meaning likely were using too much memory):
+
+
+ - [x] upcxx/python ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2764793482))
+ - [x] bolt/libtool ([run](https://github.com/buildsi/splice-experiment-runs/actions/runs/2764794583))
+ - [x] legion/zlib ([run](https://github.com/buildsi/splice-experiment-runs/runs/7592992926?check_suite_focus=true)) is Python and doesn't have any libs/binaries that are ELF.
 
 
 A checkbox means we have run -> artifacts -> results.

scripts/generate_experiments.py

@@ -71,6 +71,10 @@ def main(yaml_file, outdir):
 
 if __name__ == "__main__":
     yaml_file = os.path.join(here, "experiments.yaml")
+
+    # Yaml file is first argument (after script name)
+    if len(sys.argv) == 3:
+        yaml_file = sys.argv.pop(1)
     if len(sys.argv) != 2:
         sys.exit("please provide the output directory for experiment yamls.")
     outdir = sys.argv[1]