Petsc Floating Point Exception with large deltaT

[jcc242]

Hello,

Thank you for making this amazing library. I compiled the library on my system for OpenFOAM 8. I started from the of90 version of RheoTool and had to make a few changes to the source of RheoTool. I've tested several of the segregated solvers and was able to recover the validation results.

Then I tested the coupled PNP rheoEFoam solver: tutorials/rheoEFoam/ICEO/NernstPlanckCoupled. When I use the default deltaT, I get a floating point exception with Petsc:

[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR: INSTEAD the line number of the start of the function
[0]PETSC ERROR: is given.
[0]PETSC ERROR: [0] MatMult_SeqAIJ line 1298 /blues/gpfs/home/ssergien/soft/petsc-3.8.0/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: [0] MatMult line 2299 /blues/gpfs/home/ssergien/soft/petsc-3.8.0/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.8.0, Sep, 26, 2017
[0]PETSC ERROR: Unknown Name on a linux-gnu-intel named knl-0266 by jchristopher Mon Feb 6 15:26:55 2023
[0]PETSC ERROR: Configure options --prefix=/soft/bebop/petsc/3.8.0 PETSC_ARCH=linux-gnu-intel --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort CFLAGS="-axCOMMON-AVX512,MIC-AVX512,CORE-AVX512,CORE-AVX2,AVX -fPIC" CXXFLAGS="-axCOMMON-AVX512,MIC-AVX512,CORE-AVX512,CORE-AVX2,AVX -fPIC" FFLAGS="-axCOMMON-AVX512,MIC-AVX512,CORE-AVX512,CORE-AVX2,AVX -fPIC" --with-metis-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/metis-5.1.0-rw7ulwytxwujvf5pbv77m3iugqx67lz5 --with-parmetis-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/parmetis-4.0.3-okja4ke2nxurblhoqdwtkyikg43gpce7 --with-hdf5-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/hdf5-1.10.1-vzncl5hvkrdioypigebpug7uw3hbto5z --with-blaslapack-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/intel-mkl-2017.3.196-jyjmyutcol3veyt4bazftyqkxxzmfd7o/pkg/mkl/lib/intel64_lin --with-netcdf-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/netcdf-4.4.1.1-prsuusl3t5bfhbamqgobax2fxzvez6d2 --with-boost-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/boost-1.63.0-yyrgc3rfpmaw7i32yrmu64lwzmb2crsk --with-superlu_dist-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/superlu-dist-5.1.3-dljju447aeatnvhvmdizfjq7vxtwt7r2 --with-fftw-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/fftw-3.3.7-bsjsf45ay6jwxautz7n3kcmztbt7bndw --with-scalapack-lib="-L/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/intel-mkl-2017.3.196-jyjmyutcol3veyt4bazftyqkxxzmfd7o/pkg/mkl/lib/intel64_lin -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-mumps-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/mumps-5.1.1-6pxszht7dou6e4l3xzqjwrrbaltryjp3
[0]PETSC ERROR: #1 User provided function() line 0 in unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59
:
system msg for write_line failure : Bad file descriptor

This is using CentOS 7.0 using Petsc 3.8.0 installed by the system admin. I had to compile OpenFOAM and RheoTool with the Intel 17.04 compiler. When I decrease the deltaT to less than 4.e-5, then the solver progresses.

Have you seen this error before and know how to solve with a larger deltaT? Perhaps a different preconditioner or linear solve approach is required?

Thank you!