Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem of checkpoint and restart #3975

Closed
HLQzZ opened this issue Jun 8, 2023 · 3 comments
Closed

Problem of checkpoint and restart #3975

HLQzZ opened this issue Jun 8, 2023 · 3 comments
Labels
bug: affects latest release Bug also exists in latest release version bug Something isn't working question Further information is requested

Comments

@HLQzZ
Copy link

HLQzZ commented Jun 8, 2023

Dear developers:
I encountered a problem while using checkpoint and restart. Everything was fine before step 900,000, and the calculating example can be successfully restarted. However, after step 900,000, when I try to use restart, the program only runs for 593 steps and then throws an error, as shown below. How can I resolve this issue?
Thanks!

STEP 900591 starts ...
STEP 900591 ends. TIME = 1.26004565e-09 DT = 1.399131959e-15
Evolve time = 97.65934283 s; This step = 0.189179198 s; Avg. per step = 0.1652442349 s

STEP 900592 starts ...
STEP 900592 ends. TIME = 1.260047049e-09 DT = 1.399131959e-15
Evolve time = 97.84968653 s; This step = 0.190343705 s; Avg. per step = 0.1652866327 s

STEP 900593 starts ...
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
Segfault
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: /usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file

/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file/usr/bin/addr2line:
'/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.72.0 file for details
See Backtrace.24.0 file for details
See Backtrace.74.0 file for details
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.154.0 file for details
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.152.0 file for details
See Backtrace.155.0 file for details
See Backtrace.153.0 file for details
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.218.0 file for details
See Backtrace.209.0 file for details
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.216.0 file for details
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.211.0 file for details
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.65.0 file for details
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.67.0 file for details
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.11.0 file for details
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.8.0 file for details
See Backtrace.10.0 file for details
/usr/bin/addr2line: '/public3/home/sc31928/software/WarpX-development/two-bunch-40/warpx.3d.MPI.OMP.DP.PDP.OPMD.QED': No such file
See Backtrace.9.0 file for details
application called MPI_Abort(comm=0x84000004, 11) - process 72
application called MPI_Abort(comm=0x84000004, 11) - process 24
application called MPI_Abort(comm=0x84000002, 11) - process 74

In Backrace file:
Host Name: h1901.para.cstc
=== If no file names and line numbers are shown below, one can run
addr2line -Cpfie my_exefile my_line_address
to convert my_line_address (e.g., 0x4a6b) into file name and line number.
Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
One can use
readelf -wl my_exefile | grep my_line_address'
to find out the offset for that line.

0: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x8ea9f6]

1: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x8ec91d]

2: /lib64/libc.so.6(+0x36400) [0x2b82d9041400]

3: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x7f0640]

4: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x7e0e92]

5: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x7964ce]

6: /public3/soft/anaconda/miniconda3-phonopy-phono3py/lib/libgomp.so.1(GOMP_parallel+0x42) [0x2b82d6fbee84]
GOMP_parallel
/home/conda/feedstock_root/build_artifacts/ctng-compilers_1578638331887/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libgomp/parallel.c:173:1

7: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x7acccd]

8: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x6b7dcb]

9: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x609b76]

10: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x60a9b4]

11: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x60d660]

12: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x45b70b]

13: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2b82d902d555]

14: warpx.3d.MPI.OMP.DP.PDP.OPMD.QED() [0x4680a4]

===== TinyProfilers ======
main()
REG::WarpX::Evolve()
WarpX::Evolve()
WarpX::Evolve::step
WarpX::OneStep_nosub()
PhysicalParticleContainer::Evolve()
WarpXParticleContainer::DepositCurrent::CurrentDeposition
@ax3l ax3l added the question Further information is requested label Jun 12, 2023
@ax3l
Copy link
Member

ax3l commented Jun 12, 2023 •
edited
Loading

Hi @HLQzZ,

Thank you for the report!

Can you please compile in debug mode and post the backtrace file that is created on segfault?
Here is out debugging workflow to get to the bottom of this:
https://warpx.readthedocs.io/en/latest/usage/workflows/debugging.html

Which version of WarpX did you use? Versions are shown on the command line output directly after starting the simulation.

If you can share, the inputs file would be useful to see as well so we know which features you used and might need an update for checkpoint-restart support.

@ax3l ax3l added bug Something isn't working bug: affects latest release Bug also exists in latest release version labels Jun 12, 2023
@HLQzZ
Copy link
Author

HLQzZ commented Jun 19, 2023

I think I have solved this problem, thanks!

@ax3l
Copy link
Member

ax3l commented Jun 19, 2023

Awesome, glad to read! Just for documentation purposes, what did you change to solve your problem? :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug: affects latest release Bug also exists in latest release version bug Something isn't working question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ax3l @RemiLehe @HLQzZ