-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nek5000 Bakeoff problems #1
Conversation
Build script only builds the tools needed for the bakeoff problems: genmap and genbox
Need to move these to the build directory
TODO: * use the correct build parameters in makenek file * Run the tests * get the output * polish the scripts
tests/nek5000_bps/boxes/genbb
Outdated
@@ -0,0 +1,10 @@ | |||
cp $1.box ttt.box | |||
../../../product-sources/Nek5000/bin/genbox << EOF |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This ../../../
thing is not pretty. genbb
is executed inside boxes.sh
which is called from tests/nek5000_bps/bp1/bp1.sh
.
I can't think of a way to pass variables in go.sh
to genbb
script
without making them environment variables. Other option is to copy
all of these snippets to one .sh
file and create them as functions, say
inside bp1.sh
.
tests/nek5000_bps/boxes/genbb
Outdated
box | ||
.1 | ||
EOF | ||
mvn box $1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line will give an error if run on a machine which does not
have Nek5000/bin
directory in its path. I will fix this soon.
non-mpi compilers separately: CC, CXX, FC. Update the nek5000.sh to build in a separate directory based on the config + compiler and to use the Fortran and C compilers, FC and CC. Still need to update the compilers in machine-configs/{vulcan,ray}.sh.
Move it as a function inside the boxes.sh
Moved everything to bp1.sh script. There is a temporary workaround in build_and_run_tests in bp1.sh script. For some reason I can't access NEK5K_DIR variable defined in go.sh.
Also, managed to get rid of the temporary work-around by adding test_required_packages at the end.
I am going to soon expand this to run on other environments as well. This is just a test to see if everything works in the Nek5000 side.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I exported some of the variables in this commit so that they are accessible from makenek script.
tests/nek5000_bps/bp1/bp1.sh
Outdated
do | ||
cd b$j | ||
cp ../nek5000 . | ||
$NEK5K_DIR/bin/nekbmpi b$j 4 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just a test. I will update this with more general commands.
and improved the restructuring of the code.
I think now the executable can be run in both mac and linux. Does Vulcan use |
It looks like it uses slurm - but I've never used it. If you have access to LLNL confluence pages there is more information here https://lc.llnl.gov/confluence/display/BGQ/Running+jobs. |
You can just use the |
Also, I removed $CFLAGS and $FLAGS from being transferred into maketools file. We use maketools to build genmap and genbox in the front-end nodes in case of machines like vulcan. So, they don't need the flags which we use to compile a Nek5000 case.
We need to figure out a submission script for ray as well.
Not tested if they work fine. I am going to test them in a linux machine.
Is there a way to change the number of nodes allocated in the Currently, when I check the status of my jobs in Vulcan using I tried sbatch: error: Batch job submission failed: Node count specification invalid I tried different number of nodes as well. Any idea why this happens? If I don't use |
Added post-processing script plots the basic graph of DOFS/s vs DOFS. This can be used to generate the graphs till the python plotting is implemented. Below is the command to draw the graphs: ./go.sh --config <cfg> --compiler <cmplr> -pp tests/nek5000/bp1/pp_bp1.sh -pp (--post-process) sets the environment for post-processing script to run.
@thilinarmtb, sbatch $MPIEXEC_OPTS $MPIEXEC_NP $num_proc_node ./submit.sh $1 $2 I tried that and I get the same error I get when I run the job interactively with ./submit.sh $1 $2 There seems to be some issue with the executable. Here is what I get:
@ikarlin, any idea what this means? Or who we can ask about this error? Veselin Edit: The issue with the executable was that the combined size of all 16 tasks was more than 16 GB - the |
Otherwise nekbuilds with higher lx1 values do not get built.
Added post-processing script plots the basic graph of DOFS/s vs DOFS. ./go.sh --config <cfg> --compiler <cmplr> -pp tests/nek5000/bp1/pp_bp1.sh
|
I was able to run in Vulcan using |
I know about that issue with Even with this issue resolved (which is just a build issue), the problem is that the produced executable does not work, regardless of how you run it with SLURM - either interactively, with When you say "I was able to run in Vulcan using xlc compilers.", do you mean "run the scripts" or actually "run the executable produced by the Nek5000 build system"? -Veselin |
Thanks for the reply @v-dobrev. I was actually able to run the executables in Vulcan and they produced the |
Did you push that version to the repository? I tried the current version (20b2d3a) and I get a ./go.sh -c vulcan -m xlc -r tests/nek5000_bps/bp1/bp1.sh |
…or configs defineing the variable 'node_virt_mem_lim'.
Thanks for approving changes @tzanio ! Thanks everyone for all the help. |
This PR is an effort to add Nek5000 bake-off problems to the
CEED benchmarks repo.
Current status
As of now, users can use,
./go.sh --config vulcan --compiler gcc --build "nek5000" ./go.sh --config vulcan --compiler gcc --run tests/nek5000_bps/bp1/bp1.sh
to clone Nek5000, build the required tools (genmap and genbox)
and then build the executable for different
lx1
(order) values.This will create a separate directory for separate
lx1
values andcopy the box geometries into each of these directories. Next thing
is to
run the executable inside each of the box geometries andcollectthe data. That is really easy. We can change the range of
lx1
andthe elements in generated box geometries easily as well. For the time
being user has to edit the scripts to change these two parameters.
Currently, this PR uses values from themakenek
file to build theexecutable.makenek
file is the build script for Nek5000. So, I needto figure out how to change the parameters inmakenek
file dependingon the machine configuration we use.There are two approaches. We can keep a singlemakenek
file andedit it withsed
depending on the machine configuration we wantor create differentmakenek
files for different machines and use themdepending on the--config
parameter. I think latter is better. Let meknow if there is a better way.TODO
the machine for building the executables
- [ ] Get the output data from the generated log files after the runand plot the graphs.
- [ ] Polish the script files (Current script files in this repo are prettynice :) )