Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Debug testing failure #6

Closed
moosebuild opened this issue Oct 6, 2021 · 2 comments
Closed

Debug testing failure #6

moosebuild opened this issue Oct 6, 2021 · 2 comments
Assignees
@cticenhour
Labels
c: infrastructure p: critical t: defect Code defect or bug

Comments

@moosebuild
Copy link

In GitLab by @cticenhour on Oct 6, 2021, 11:40

See idaholab/valhalla#6 for original issue. Description repeated below.

Bug Description

Max time parameter is violated on test melt_pool_fluid.fluid on CIVET

Steps to Reproduce

Inconsistent - seen mainly on CIVET for around the last month (February 2021) or so: https://moosebuild.hpc.inl.gov/job/309917/

Impact

Submodule updates not being merged to master.
@moosebuild
Copy link
Author

In GitLab by @jiangwen84 on Oct 6, 2021, 20:53

mentioned in commit d56bd5b

@moosebuild
Copy link
Author

In GitLab by @cticenhour on Oct 6, 2021, 21:08

For SQA completeness, and just in case the CIVET link above fails after Valhalla is archived and removed from regular testing, I am recording the test failure log below:

Click to expand: 
//: singularity exec --no-home /data/CIVET_CI/singularity/CentOS-8 /tmp/civet_CentOS-8_build_1/tmph8vj04hy
meowbuild2 @ 169G remaining space
AMD EPYC 7702P 64-Core Processor  x1

Loaded modules 'mpich cmake python vtk'
METHODS=dbg
METHOD=dbg
Date: Thu Feb 25 15:27:14 MST 2021
Machine: meowbuild2.hpc.inl.gov
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 8.0.1905 (Core) 
Release:	8.0.1905
Codename:	Core
Cannot find MOOSE Package version.
Build Root: /opt/civet/build_1
Trigger: Push devel
Step: Test valhalla (debug) (7)
MODULESHOME: /usr/share/Modules
Currently Loaded Modulefiles:
 1) gcc-9.3.0-gcc-8.2.1-y6xo224                   47) vtk/8.2.0-gcc(default)  
 2) mpich-3.3.2-gcc-9.3.0-gyitxt7                 
 3) mpich/3.3.2-gcc-9.3.0(default)                
 4) cmake                                         
 5) python-3.7.8-gcc-9.3.0-p6j5c4q                
 6) py-astroid-2.3.3-gcc-9.3.0-souudos            
 7) py-latexcodec-1.0.4-gcc-9.3.0-ret3luc         
 8) py-mock-3.0.5-gcc-9.3.0-t55rjga               
 9) py-pylint-2.3.1-gcc-9.3.0-ggwb2hy             
10) py-soupsieve-1.9.3-gcc-9.3.0-hv4cofa          
11) py-beautifulsoup4-4.8.0-gcc-9.3.0-hgvgo72     
12) py-lazy-object-proxy-1.4.3-gcc-9.3.0-5kvyeib  
13) py-mpmath-1.1.0-gcc-9.3.0-u3f46y4             
14) py-pyparsing-2.4.2-gcc-9.3.0-sjx5icv          
15) py-sympy-1.4-gcc-9.3.0-lalehri                
16) py-bottleneck-1.2.1-gcc-9.3.0-gxzprev         
17) py-livereload-2.6.3-gcc-9.3.0-vrkeqhs         
18) py-numexpr-2.7.0-gcc-9.3.0-mbclwu7            
19) py-python-dateutil-2.8.0-gcc-9.3.0-vajhctu    
20) py-certifi-2020.6.20-gcc-9.3.0-c4bbtsf        
21) py-lxml-4.5.2-gcc-9.3.0-kelhhtp               
22) py-numpy-1.19.4-gcc-9.3.0-qcjultr             
23) py-pytz-2020.1-gcc-9.3.0-e42jful              
24) py-cycler-0.10.0-gcc-9.3.0-4ljkfru            
25) py-mako-1.0.4-gcc-9.3.0-p6mnftn               
26) py-pandas-1.1.4-gcc-9.3.0-4bgox7j             
27) py-pyyaml-5.3.1-gcc-9.3.0-6z2zrrg             
28) py-tornado-6.0.3-gcc-9.3.0-mj3ltdw            
29) py-cython-0.29.21-gcc-9.3.0-oollyzy           
30) py-markupsafe-1.1.1-gcc-9.3.0-wpuhyb5         
31) py-pillow-7.2.0-gcc-9.3.0-vjwta4l             
32) py-setuptools-50.3.2-gcc-9.3.0-apj44bx        
33) py-typed-ast-1.4.0-gcc-9.3.0-qjsr672          
34) py-isort-4.3.20-gcc-9.3.0-d7vrhk3             
35) py-matplotlib-3.3.2-gcc-9.3.0-hhqaixb         
36) py-pybtex-0.21-gcc-9.3.0-e6voas5              
37) py-setuptools-scm-4.1.2-gcc-9.3.0-ibkyexk     
38) py-wrapt-1.11.2-gcc-9.3.0-ye4mqkl             
39) py-kiwisolver-1.1.0-gcc-9.3.0-r652ydj         
40) py-mccabe-0.6.1-gcc-9.3.0-c26vumf             
41) py-pylatexenc-2.8-gcc-9.3.0-5kyxmzp           
42) py-six-1.14.0-gcc-9.3.0-iqvj3xp               
43) py-pyflakes-2.2.0-gcc-9.3.0-3a74qvv           
44) python/3.7.8(default)                         
45) zlib-1.2.11-gcc-9.3.0-kma42ln                 
46) vtk-8.2.0-gcc-9.3.0-7yft3rf                   
Base commit: git@hpcgitlab.hpc.inl.gov:idaholab/valhalla.git:devel:757bb3b10a6d893c359e5f8513d787a028d7c153
Head commit: git@hpcgitlab.hpc.inl.gov:idaholab/valhalla.git:devel:025888614d822c17e74dab9d257b3b18aab9a0c0

BUILD_ROOT/valhalla/: ./run_tests --pedantic-checks -j 16 -l 16 --longest-jobs=10
    SKIP test:melt_pool_level_set.level_set [METHOD!=OPT]
    SKIP examples/melt_pool_3d.melt_pool_3d [METHOD!=OPT]
    SKIP examples/melt_pool_2d.melt_pool_2d [METHOD!=OPT]
 RUNNING test:gradient_regularization.gradient_regularization
 RUNNING test:melt_pool_fluid.fluid
 RUNNING test:melt_pool_heat.heat
 RUNNING test:curvature_regularization.curvature_regularization
      OK test:gradient_regularization.gradient_regularization [FINISHED]
      OK test:curvature_regularization.curvature_regularization [FINISHED]
      OK test:melt_pool_heat.heat [FINISHED]
 RUNNING test:melt_pool_heat.thermal_material
      OK test:melt_pool_heat.thermal_material [FINISHED]
test:melt_pool_fluid.fluid: Working Directory: /opt/civet/build_1/valhalla/test/tests/melt_pool_fluid
test:melt_pool_fluid.fluid: Running command: /opt/civet/build_1/valhalla/valhalla-dbg -i fluid.i --no-gdb-backtrace
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: The following total 13 aux variables:
test:melt_pool_fluid.fluid:   delta_function
test:melt_pool_fluid.fluid:   enthalpy
test:melt_pool_fluid.fluid:   heaviside_function
test:melt_pool_fluid.fluid:   liquid_mass_fraction
test:melt_pool_fluid.fluid:   liquid_volume_fraction
test:melt_pool_fluid.fluid:   melt_pool_mass_rate
test:melt_pool_fluid.fluid:   mu
test:melt_pool_fluid.fluid:   permeability
test:melt_pool_fluid.fluid:   rho
test:melt_pool_fluid.fluid:   solid_mass_fraction
test:melt_pool_fluid.fluid:   solid_volume_fraction
test:melt_pool_fluid.fluid:   specific_heat
test:melt_pool_fluid.fluid:   thermal_conductivity
test:melt_pool_fluid.fluid: are added for automatic output by MaterialOutputAction.
test:melt_pool_fluid.fluid: Initializing equation system ..........................................                    [ 43.47 s]
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Framework Information:
test:melt_pool_fluid.fluid: MOOSE Version:           git commit 6fac045ef1 on 2021-02-25
test:melt_pool_fluid.fluid: LibMesh Version:         ed76100f41840c34c146dd0a7f1ff5410370b489
test:melt_pool_fluid.fluid: PETSc Version:           3.14.2
test:melt_pool_fluid.fluid: SLEPc Version:           3.14.0
test:melt_pool_fluid.fluid: Current Time:            Thu Feb 25 15:28:04 2021
test:melt_pool_fluid.fluid: Executable Timestamp:    Thu Feb 25 15:27:11 2021
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Parallelism:
test:melt_pool_fluid.fluid:   Num Processors:          1
test:melt_pool_fluid.fluid:   Num Threads:             1
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Mesh: 
test:melt_pool_fluid.fluid:   Parallel Type:           replicated
test:melt_pool_fluid.fluid:   Mesh Dimension:          2
test:melt_pool_fluid.fluid:   Spatial Dimension:       2
test:melt_pool_fluid.fluid:   Nodes:                   
test:melt_pool_fluid.fluid:     Total:                 2601
test:melt_pool_fluid.fluid:     Local:                 2601
test:melt_pool_fluid.fluid:   Elems:                   
test:melt_pool_fluid.fluid:     Total:                 2500
test:melt_pool_fluid.fluid:     Local:                 2500
test:melt_pool_fluid.fluid:   Num Subdomains:          1
test:melt_pool_fluid.fluid:   Num Partitions:          1
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Nonlinear System:
test:melt_pool_fluid.fluid:   Num DOFs:                18207
test:melt_pool_fluid.fluid:   Num Local DOFs:          18207
test:melt_pool_fluid.fluid:   Variables:               "temp" { "grad_ls" "velocity" } { "p" "curvature" } 
test:melt_pool_fluid.fluid:   Finite Element Types:    "LAGRANGE" "LAGRANGE_VEC" "LAGRANGE" 
test:melt_pool_fluid.fluid:   Approximation Orders:    "FIRST" "FIRST" "FIRST" 
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Auxiliary System:
test:melt_pool_fluid.fluid:   Num DOFs:                35101
test:melt_pool_fluid.fluid:   Num Local DOFs:          35101
test:melt_pool_fluid.fluid:   Variables:               "ls" { "delta_function" "enthalpy" "heaviside_function" "liquid_mass_fraction" 
test:melt_pool_fluid.fluid:                              "liquid_volume_fraction" ... "rho" "solid_mass_fraction" "solid_volume_fraction" 
test:melt_pool_fluid.fluid:                              "specific_heat" "thermal_conductivity" } 
test:melt_pool_fluid.fluid:   Finite Element Types:    "LAGRANGE" "MONOMIAL" 
test:melt_pool_fluid.fluid:   Approximation Orders:    "FIRST" "CONSTANT" 
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Execution Information:
test:melt_pool_fluid.fluid:   Executioner:             Transient
test:melt_pool_fluid.fluid:   TimeStepper:             ConstantDT
test:melt_pool_fluid.fluid:   Solver Mode:             NEWTON
test:melt_pool_fluid.fluid:   MOOSE Preconditioner:    SMP
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: LEGACY MODES ENABLED:
test:melt_pool_fluid.fluid:  This application uses the legacy material output option: material properties are output only on TIMESTEP_END, not INITIAL. To remove this message, set 'use_legacy_material_output' to false in this application. If there are gold output files that contain material property output for which output occurs on INITIAL, then these will generate diffs due to zero values being stored, and these tests should be re-golded.
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Projecting initial condition ........                                                      [  9.59 s]
test:melt_pool_fluid.fluid: Computing initial stateful property values ......                                          [  7.69 s]
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Time Step 0, time = 0
test:melt_pool_fluid.fluid: Outputting exodus                                                                          [  1.44 s]
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Time Step 1, time = 0.0001, dt = 0.0001
test:melt_pool_fluid.fluid: Computing initial residual ...                                                             [  4.50 s]
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Performing automatic scaling calculation
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid:  0 Nonlinear |R| = 3.131386e+04
test:melt_pool_fluid.fluid:  1 Nonlinear |R| = 2.145463e+08
test:melt_pool_fluid.fluid:  2 Nonlinear |R| = 3.201286e-04
test:melt_pool_fluid.fluid:  3 Nonlinear |R| = 3.187874e-04
test:melt_pool_fluid.fluid: 
 TIMEOUT test:melt_pool_fluid.fluid FAILED (TIMEOUT) [FINISHED]


Final Test Results:
--------------------------------------------------------------------------------------------------------------
    SKIP test:melt_pool_level_set.level_set [METHOD!=OPT]
    SKIP examples/melt_pool_3d.melt_pool_3d [METHOD!=OPT]
    SKIP examples/melt_pool_2d.melt_pool_2d [METHOD!=OPT]
      OK test:gradient_regularization.gradient_regularization [FINISHED]
      OK test:curvature_regularization.curvature_regularization [FINISHED]
      OK test:melt_pool_heat.heat [FINISHED]
      OK test:melt_pool_heat.thermal_material [FINISHED]
 TIMEOUT test:melt_pool_fluid.fluid FAILED (TIMEOUT) [FINISHED]
--------------------------------------------------------------------------------------------------------------
Ran 5 tests in 300.2 seconds. Average test time 101.3 seconds, maximum test time 300.0 seconds.
4 passed, 3 skipped, 0 pending, 1 FAILED

10 longest running jobs:
--------------------------------------------------------------------------------------------------------------
[300.0s]  TIMEOUT test:melt_pool_fluid.fluid FAILED (TIMEOUT) [FINISHED]
[56.83s]       OK test:melt_pool_heat.thermal_material [FINISHED]
[54.54s]       OK test:melt_pool_heat.heat [FINISHED]
[50.93s]       OK test:curvature_regularization.curvature_regularization [FINISHED]
[44.24s]       OK test:gradient_regularization.gradient_regularization [FINISHED]
There are no race conditions.


***************************************************************************
Your exit_code is: 1
Possible intermittent failure condition detected! - Retrying failed tests...

BUILD_ROOT/valhalla/: ./run_tests -j 16 -l 16 --longest-jobs=10 --failed-tests --ignore prereq
 RUNNING test:melt_pool_fluid.fluid
test:melt_pool_fluid.fluid: Working Directory: /opt/civet/build_1/valhalla/test/tests/melt_pool_fluid
test:melt_pool_fluid.fluid: Running command: /opt/civet/build_1/valhalla/valhalla-dbg -i fluid.i --no-gdb-backtrace
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: The following total 13 aux variables:
test:melt_pool_fluid.fluid:   delta_function
test:melt_pool_fluid.fluid:   enthalpy
test:melt_pool_fluid.fluid:   heaviside_function
test:melt_pool_fluid.fluid:   liquid_mass_fraction
test:melt_pool_fluid.fluid:   liquid_volume_fraction
test:melt_pool_fluid.fluid:   melt_pool_mass_rate
test:melt_pool_fluid.fluid:   mu
test:melt_pool_fluid.fluid:   permeability
test:melt_pool_fluid.fluid:   rho
test:melt_pool_fluid.fluid:   solid_mass_fraction
test:melt_pool_fluid.fluid:   solid_volume_fraction
test:melt_pool_fluid.fluid:   specific_heat
test:melt_pool_fluid.fluid:   thermal_conductivity
test:melt_pool_fluid.fluid: are added for automatic output by MaterialOutputAction.
test:melt_pool_fluid.fluid: Initializing equation system ...........................................                   [ 44.05 s]
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Framework Information:
test:melt_pool_fluid.fluid: MOOSE Version:           git commit 6fac045ef1 on 2021-02-25
test:melt_pool_fluid.fluid: LibMesh Version:         ed76100f41840c34c146dd0a7f1ff5410370b489
test:melt_pool_fluid.fluid: PETSc Version:           3.14.2
test:melt_pool_fluid.fluid: SLEPc Version:           3.14.0
test:melt_pool_fluid.fluid: Current Time:            Thu Feb 25 15:33:06 2021
test:melt_pool_fluid.fluid: Executable Timestamp:    Thu Feb 25 15:27:11 2021
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Parallelism:
test:melt_pool_fluid.fluid:   Num Processors:          1
test:melt_pool_fluid.fluid:   Num Threads:             1
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Mesh: 
test:melt_pool_fluid.fluid:   Parallel Type:           replicated
test:melt_pool_fluid.fluid:   Mesh Dimension:          2
test:melt_pool_fluid.fluid:   Spatial Dimension:       2
test:melt_pool_fluid.fluid:   Nodes:                   
test:melt_pool_fluid.fluid:     Total:                 2601
test:melt_pool_fluid.fluid:     Local:                 2601
test:melt_pool_fluid.fluid:   Elems:                   
test:melt_pool_fluid.fluid:     Total:                 2500
test:melt_pool_fluid.fluid:     Local:                 2500
test:melt_pool_fluid.fluid:   Num Subdomains:          1
test:melt_pool_fluid.fluid:   Num Partitions:          1
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Nonlinear System:
test:melt_pool_fluid.fluid:   Num DOFs:                18207
test:melt_pool_fluid.fluid:   Num Local DOFs:          18207
test:melt_pool_fluid.fluid:   Variables:               "temp" { "grad_ls" "velocity" } { "p" "curvature" } 
test:melt_pool_fluid.fluid:   Finite Element Types:    "LAGRANGE" "LAGRANGE_VEC" "LAGRANGE" 
test:melt_pool_fluid.fluid:   Approximation Orders:    "FIRST" "FIRST" "FIRST" 
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Auxiliary System:
test:melt_pool_fluid.fluid:   Num DOFs:                35101
test:melt_pool_fluid.fluid:   Num Local DOFs:          35101
test:melt_pool_fluid.fluid:   Variables:               "ls" { "delta_function" "enthalpy" "heaviside_function" "liquid_mass_fraction" 
test:melt_pool_fluid.fluid:                              "liquid_volume_fraction" ... "rho" "solid_mass_fraction" "solid_volume_fraction" 
test:melt_pool_fluid.fluid:                              "specific_heat" "thermal_conductivity" } 
test:melt_pool_fluid.fluid:   Finite Element Types:    "LAGRANGE" "MONOMIAL" 
test:melt_pool_fluid.fluid:   Approximation Orders:    "FIRST" "CONSTANT" 
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Execution Information:
test:melt_pool_fluid.fluid:   Executioner:             Transient
test:melt_pool_fluid.fluid:   TimeStepper:             ConstantDT
test:melt_pool_fluid.fluid:   Solver Mode:             NEWTON
test:melt_pool_fluid.fluid:   MOOSE Preconditioner:    SMP
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: LEGACY MODES ENABLED:
test:melt_pool_fluid.fluid:  This application uses the legacy material output option: material properties are output only on TIMESTEP_END, not INITIAL. To remove this message, set 'use_legacy_material_output' to false in this application. If there are gold output files that contain material property output for which output occurs on INITIAL, then these will generate diffs due to zero values being stored, and these tests should be re-golded.
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Projecting initial condition ........                                                      [  9.62 s]
test:melt_pool_fluid.fluid: Computing initial stateful property values ......                                          [  7.67 s]
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Time Step 0, time = 0
test:melt_pool_fluid.fluid: Outputting exodus                                                                          [  1.44 s]
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Time Step 1, time = 0.0001, dt = 0.0001
test:melt_pool_fluid.fluid: Computing initial residual ...                                                             [  4.46 s]
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid: Performing automatic scaling calculation
test:melt_pool_fluid.fluid: 
test:melt_pool_fluid.fluid:  0 Nonlinear |R| = 3.131386e+04
test:melt_pool_fluid.fluid:  1 Nonlinear |R| = 2.145463e+08
test:melt_pool_fluid.fluid:  2 Nonlinear |R| = 3.201286e-04
test:melt_pool_fluid.fluid:  3 Nonlinear |R| = 3.187874e-04
test:melt_pool_fluid.fluid: 
 TIMEOUT test:melt_pool_fluid.fluid FAILED (TIMEOUT) [FINISHED,previous results: TIMEOUT]


Final Test Results:
--------------------------------------------------------------------------------------------------------------
 TIMEOUT test:melt_pool_fluid.fluid FAILED (TIMEOUT) [FINISHED,previous results: TIMEOUT]
--------------------------------------------------------------------------------------------------------------
Ran 1 tests in 300.2 seconds. Average test time 300.0 seconds, maximum test time 300.0 seconds.
0 passed, 0 skipped, 0 pending, 1 FAILED

10 longest running jobs:
--------------------------------------------------------------------------------------------------------------
[300.0s]  TIMEOUT test:melt_pool_fluid.fluid FAILED (TIMEOUT) [FINISHED,previous results: TIMEOUT]
ERROR: exiting with code 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cticenhour cticenhour

Labels
c: infrastructure p: critical t: defect Code defect or bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@moosebuild @cticenhour