-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Segmentation fault when running more than one turbine #1068
Comments
This definitely feels like it could be due to using old versions of the stack. You mentioned using From the attached files you shared, it definitely looks like the error is in openfast. @psakievich, @jrood-nrel is there a way they can use vanilla spack here? I am a bit surprised it went with an old version of openfast. Or should they try the exawind-manager route? |
@justhawk98 what spack version are you using? I would prefer we fix this in spack if that is the issue rather than pushing more people to exawind-manager. Using $ spack solve amr-wind@main ^openfast@3.2.1
==> Error: concretization failed for the following reasons:
1. Cannot select a single "version" for package "openfast"
2. Cannot satisfy 'openfast@3.5:'
3. Cannot satisfy 'openfast@3.2.1'
4. Cannot satisfy 'openfast@2.6.0:3.4.1'
5. Cannot satisfy 'openfast@3.5:'
required because amr-wind depends on openfast@3.5: when @2:+openfast
required because amr-wind@main ^openfast@3.2.1 requested explicitly
6. Cannot satisfy 'openfast@3.2.1'
required because amr-wind@main ^openfast@3.2.1 requested explicitly
7. Cannot satisfy 'openfast@3.5:' and 'openfast@3.2.1
required because amr-wind depends on openfast@3.5: when @2:+openfast
required because amr-wind@main ^openfast@3.2.1 requested explicitly
required because amr-wind@main ^openfast@3.2.1 requested explicitly
8. Cannot satisfy 'openfast@3.2.1' and 'openfast@3.5:
required because amr-wind depends on openfast@3.5: when @2:+openfast
required because amr-wind@main ^openfast@3.2.1 requested explicitly
required because amr-wind@main ^openfast@3.2.1 requested explicitly So I suspect this is an older version of spack before we updated all the package requirements. |
@justhawk98 you need to use |
For what it's worth, Justin messaged me a few weeks back before AMR-Wind 2.0 and the OpenFAST 3.5 changeover, and he was having this same problem at that time |
Hi @justhawk98, I noticed that you're running two openfast turbines but have only 1 rank/mpi process for the entire simulation. I believe that amr-wind will let this happen (both openfast turbine instances will end up on the same processor), but it is probably not good practice. Lawrence |
Thanks for the help and suggestions everyone. Currently, we are using spack 0.19.2. It's looking like the best option is to recompile with an updated OpenFAST. I'll reach out to our research computing office and see what they think is the best way to do that. Also I wanted to ask, are there any flags we're missing that would be good to turn on? Or any that would be better to turn off? |
@justhawk98 if you are stuck on Spack Alternatively for openfast you could give the spec |
@justhawk98 in terms of flags, I don't think we typically run with openmp. I would try turning that off on both. We recently had to turn that off in openfast when it was accidentally turned on. I believe the issues was segfaults. So that could be your issue... |
And even if it is off, it could be still picking it up: OpenFAST/openfast#2229. We've asked that it be completely disabled so (very) recent commits of openfast should be safe(r) |
I was having the same issue when running small tests in my laptop with OpenMP. I sorted it out running with MPI. For instance: |
Hi everyone, thank you for all your help. The issue has been resolved. Here's a report on how I fixed the issue in case anyone comes across this in the future. TL;DR : I had to recompile OpenFAST and AMR-Wind from source with OpenMP off for both and MPI on for AMR-Wind. REPORT: terminate called after throwing an instance of 'std::runtime_error' But even with this build, a single turbine run would still finish without issue. OpenFAST compilation from openfast/build (As you can see the only additional flag was one to install to a specific directory): AMR-Wind: Once again, thank you everybody for your help. I couldn't have fixed this without you. p.s. |
Bug description
I'm running into a segmentation fault when trying to simulate multiple turbines. If I run just one, everything works fine and the program runs to completion. The error occurs during initialization of the 2nd instance of OpenFAST. Is there something I'm missing in my input file or .fst files? I'm using OpenFAST 3.2.1. Could the OpenFAST version be the issue?
Steps to reproduce
I've included a .zip file which contains the AMR-Wind input file, the FAST files for the turbines used, and the terminal output which includes the segmentation fault. The simulation uses only the FreeStream and Actuator physics models so anyone should be able to run my case without any precursor ABL simulations.
Steps to reproduce the behavior:
Compiler used
Operating system
Hardware:
Machine details ():
It's a slurm system. Spack was used for package management of both AMR-Wind and OpenFAST. Ran on the Brigham Young University supercomputer.
Input file attachments and .txt file of the terminal output
error_report.zip
Expected behavior
AMR-Wind information
Additional Content
I mentioned I'm using AMR-Wind and OpenFAST via spack. Here's the "spack find -lv" for AMR-Wind and OpenFAST which shows the flags used during compilation. Is there a flag I need to compile with to enable multiple turbines?
spack find -lv amr-wind
-- linux-rhel7-haswell / gcc@11.4.0 -----------------------------
jewntlv amr-wind@main
ascentcuda+hypreipomasa+mpi+netcdf+openfast+openmp~rocm+shared+tests build_system=cmake build_type=RelWithDebInfospack find -lv openfast
-- linux-rhel7-haswell / gcc@11.4.0 -----------------------------
2ohvfco openfast@3.2.1+cxx+dll-interface+double-precision
iponetcdf+openmp+pic+shared build_system=cmake build_type=RelWithDebInfoThe text was updated successfully, but these errors were encountered: