Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfaults during initialization when using the cpp interface on certain compilers #1345

Open
psakievich opened this issue Nov 23, 2022 · 0 comments

Comments

@psakievich
Copy link
Contributor

psakievich commented Nov 23, 2022

Bug description
We are consistently seeing segfaults in nalu-wind's CI during the openfast initialization process in our unit tests. These tests just read in a token input file, initialize openfast and then test some data types in nalu-wind.

An example github action report can be seen here: https://github.com/Exawind/nalu-wind/actions/runs/3527523887/jobs/5916699183

# output from logs
2022-11-22T22:58:53.4218576Z [ RUN      ] ActuatorBulkFastTests.NGP_initializeActuatorBulk
2022-11-22T22:58:53.4218581Z 
2022-11-22T22:58:53.4218691Z  **************************************************************************************************
2022-11-22T22:58:53.4218770Z  OpenFAST
2022-11-22T22:58:53.4218776Z 
2022-11-22T22:58:53.4218923Z  Copyright (C) 2022 National Renewable Energy Laboratory
2022-11-22T22:58:53.4219045Z  Copyright (C) 2022 Envision Energy USA LTD
2022-11-22T22:58:53.4219052Z 
2022-11-22T22:58:53.4219254Z  This program is licensed under Apache License Version 2.0 and comes with ABSOLUTELY NO WARRANTY.
2022-11-22T22:58:53.4219412Z  See the "LICENSE" file distributed with this software for details.
2022-11-22T22:58:53.4219588Z  **************************************************************************************************
2022-11-22T22:58:53.4219595Z 
2022-11-22T22:58:53.4219759Z  OpenFAST-v3.3.0-1-g4e85fa62
2022-11-22T22:58:53.4219846Z  Compile Info:
2022-11-22T22:58:53.4219986Z   - Compiler: GCC version 9.4.0
2022-11-22T22:58:53.4220127Z   - Architecture: 64 bit
2022-11-22T22:58:53.4220259Z   - Precision: double
2022-11-22T22:58:53.4220375Z   - OpenMP: No
2022-11-22T22:58:53.4220503Z   - Date: Nov 22 2022
2022-11-22T22:58:53.4220624Z   - Time: 01:52:08
2022-11-22T22:58:53.4220695Z  Execution Info:
2022-11-22T22:58:53.4220822Z   - Date: 11/22/2022
2022-11-22T22:58:53.4220951Z   - Time: 14:58:53-0800
2022-11-22T22:58:53.4220957Z 
2022-11-22T22:58:53.4221061Z  OpenFAST input file heading:
2022-11-22T22:58:53.4221227Z      FAST Certification Test #01: NREL 5.0 MW Baseline Wind Turbine (Onshore)
2022-11-22T22:58:53.4221234Z 
2022-11-22T22:58:53.4221322Z  Running ElastoDyn.
2022-11-22T22:58:53.4221511Z  Nodal outputs section of ElastoDyn input file not found or improperly formatted.
2022-11-22T22:58:53.4221597Z  Running AeroDyn.
2022-11-22T22:58:53.4221782Z  Warning: Turning off Unsteady Aerodynamics because UA parameters are not included in airfoil
2022-11-22T22:58:53.4221928Z  (airfoil has likely has constant polars). (node 1, blade 1)
2022-11-22T22:58:53.4222118Z  Warning: Turning off Unsteady Aerodynamics because UA parameters are not included in airfoil
2022-11-22T22:58:53.4222258Z  (airfoil has likely has constant polars). (node 1, blade 2)
2022-11-22T22:58:53.4222448Z  Warning: Turning off Unsteady Aerodynamics because UA parameters are not included in airfoil
2022-11-22T22:58:53.4222586Z  (airfoil has likely has constant polars). (node 1, blade 3)
2022-11-22T22:58:53.4222592Z 
2022-11-22T22:58:53.4222713Z ===================================================================================
2022-11-22T22:58:53.4222862Z =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
2022-11-22T22:58:53.4223015Z =   PID 38339 RUNNING AT 2ea5735c7933
2022-11-22T22:58:53.4223099Z =   EXIT CODE: 139
2022-11-22T22:58:53.4223209Z =   CLEANING UP REMAINING PROCESSES
2022-11-22T22:58:53.4223335Z =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
2022-11-22T22:58:53.4223452Z ===================================================================================
2022-11-22T22:58:53.4223637Z YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
2022-11-22T22:58:53.4223787Z This typically refers to a problem with your application.
2022-11-22T22:58:53.4223908Z Please see the FAQ page for debugging suggestions

I tried to run this same test on my laptop using apple-clang@13 + gcc@11 and I don't see this.

To Reproduce
Clone the docker image: docker pull ecpe4s/exawind-snapshot:latest
From inside the container run:

# load bash shell
bash -c
# load spack-manager
source spack-manager/start.sh
# setup a nalu-wind build with openfast
quick-develop --name segfault --spec nalu-wind@master+openfast
# install environment
spack install
# dive into the build environment
build-env-dive nalu-wind
# run the tests and see the segfault
./unittestX --gtest_filter=ActuatorBulkFastTests.NGP_initializeActuatorBulk

@tasmith4 and I have spent time trying to debug this segfault in the past on SNL hardware and couldn't find a good solution.
We saw the segfault disappear in debug builds.

Expected behavior
I expect the code to initialize without segfaulting on any platform

Screenshots, if applicable

OpenFAST Version
Running with the latest master branch v3.3.0. Sample output is above in the log

System Information (please complete the following information):

  • OS: ubuntu 20.04 and REHL7
  • Compiler: GNU 9.3 and 9.4
  • Compiler settings: <e.g. CMake flags or other settings>

Additional context
Exawind/nalu-wind#979
Pinging @jrood-nrel @rafmudaf

@psakievich psakievich changed the title Random segfaults when using the cpp interface Segfaults during initialization when using the cpp interface on certain compilers Nov 23, 2022
psakievich added a commit to Exawind/nalu-wind that referenced this issue Nov 23, 2022
OpenFAST is segfaulting. I can't reproduce on my laptop.  Turning it off in the CI again.

OpenFAST/openfast#1345
psakievich added a commit to Exawind/nalu-wind that referenced this issue Nov 23, 2022
OpenFAST is segfaulting. I can't reproduce on my laptop.  Turning it off in the CI again.

OpenFAST/openfast#1345
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant