Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Random failure in Unit.Evolution.Systems.GeneralizedHarmonic.Characteristics #1308

Closed
wthrowe opened this issue Jan 31, 2019 · 8 comments · Fixed by #1620
Closed

Random failure in Unit.Evolution.Systems.GeneralizedHarmonic.Characteristics #1308

wthrowe opened this issue Jan 31, 2019 · 8 comments · Fixed by #1620
Assignees
@prayush
Labels
bug

Comments

@wthrowe
Copy link
Member

wthrowe commented Jan 31, 2019 

Charm++: standalone mode (not using charmrun)
Charm++> Running in Multicore mode:  1 threads
Converse/Charm++ Commit ID: v6.8.2
[0] isomalloc.c> Disabling isomalloc because isomalloc disabled by conv-mach
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 unique compute nodes (32-way SMP).
Charm++> cpu topology info is gathered in 0.000 seconds.
SpECTRE Build Information:
Version:                      0.0.0
Compiled on host:             ungoliant
Compiled in directory:        /home/wthrowe/spectre-work
Source directory is:          /home/wthrowe/spectre
Compiled on git branch:       develop
Compiled with git hash:       9616ac1b545d48735581c9a881ae7ddb79c0f705
Linked on:                    Thu Jan 31 14:21:51 2019

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RunTests is a Catch v2.5.0 host application.
Run with -? for options

-------------------------------------------------------------------------------
Unit.Evolution.Systems.GeneralizedHarmonic.Characteristics
-------------------------------------------------------------------------------
/home/wthrowe/spectre/tests/Unit/Evolution/Systems/GeneralizedHarmonic/Test_Characteristics.cpp:503
...............................................................................

/home/wthrowe/spectre/tests/Unit/TestingFramework.hpp:137: FAILED:
  CHECK( a == appx(b) )
with expansion:
  -6.2856077844 == Approx( -6.2856077844 )
with messages:
  seed: 479364081
  function: char_field_uzero
  /home/wthrowe/spectre/tests/Unit/Pypp/CheckWithRandomValues.hpp:222: result =
  = pypp::call<ResultType>( module_name, function_name, std::get<ArgumentIs>
  (args)..., forward_to_pypp<ResultType>(std::get<MemberArgsIs>(member_args),
  used_for_size)...)
  a: T(0,0,0)=(5.848515898819056019e+04,4.574266272797355778e+05,-9.
  615472439722281706e+04,1.580058267155152862e+05,-4.509070357106604497e+05,-1.
  774134599593203748e+05,3.985918004222508898e+04,-4.580962612235626921e+02,-2.
  281025273701576589e+04,-1.866233262533290836e+05,-8.772396622437198630e+02,1.
  195063785246795160e+05,4.032941446045924677e+05,6.855596154870730243e+04,-2.
  979837620005070676e+03,-3.632027390544380069e+03,-1.709982839100827505e+03,-
  1.401915449730193650e+05,2.026653997725884255e+04,2.111310934315297345e+05)
<lots of numbers omitted>
  a: (-3.497608995597991743e+05,1.814888823864474252e+05,1.108199560069938889e+
  05,2.518720361409107281e+05,-5.504227813799985597e+04,-1.044813875036323880e+
  05,1.282915103985938731e+04,1.064096746204912876e+02,-2.410437563539280745e+
  04,1.572217804190125025e+05,-1.132422428463703181e+04,-9.854870727899871417e+
  04,4.142831554732488585e+05,1.580409339389271627e+05,1.090890793719776673e+
  06,-6.285607784362095884e+00,2.184942850230274416e+04,-3.227785413232819410e+
  05,1.225238626494904202e+04,-5.974468892584493151e+05)
  b: (-3.497608995597991161e+05,1.814888823864474252e+05,1.108199560069939034e+
  05,2.518720361409108154e+05,-5.504227813799987052e+04,-1.044813875036323880e+
  05,1.282915103985938913e+04,1.064096746204912733e+02,-2.410437563539280745e+
  04,1.572217804190125025e+05,-1.132422428463703181e+04,-9.854870727899869962e+
  04,4.142831554732488585e+05,1.580409339389271918e+05,1.090890793719776440e+
  06,-6.285607784371444851e+00,2.184942850230274053e+04,-3.227785413232819410e+
  05,1.225238626494903474e+04,-5.974468892584493151e+05)
  a: -6.285607784362095884e+00
  b: -6.285607784371444851e+00

/home/wthrowe/spectre/tests/Unit/TestingFramework.hpp:137: FAILED:
  CHECK( a == appx(b) )
with expansion:
  -6.2856077844 == Approx( -6.2856077844 )
with messages:
  seed: 479364081
  function: char_field_uzero
  /home/wthrowe/spectre/tests/Unit/Pypp/CheckWithRandomValues.hpp:222: result =
  = pypp::call<ResultType>( module_name, function_name, std::get<ArgumentIs>
  (args)..., forward_to_pypp<ResultType>(std::get<MemberArgsIs>(member_args),
  used_for_size)...)
  a: T(0,0,0)=(5.848515898819056019e+04,4.574266272797355778e+05,-9.
  615472439722281706e+04,1.580058267155152862e+05,-4.509070357106604497e+05,-1.
  774134599593203748e+05,3.985918004222508898e+04,-4.580962612235626921e+02,-2.
  281025273701576589e+04,-1.866233262533290836e+05,-8.772396622437198630e+02,1.
  195063785246795160e+05,4.032941446045924677e+05,6.855596154870730243e+04,-2.
  979837620005070676e+03,-3.632027390544380069e+03,-1.709982839100827505e+03,-
  1.401915449730193650e+05,2.026653997725884255e+04,2.111310934315297345e+05)
<lots more numbers omitted>
  a: (-3.497608995597991743e+05,1.814888823864474252e+05,1.108199560069938889e+
  05,2.518720361409107281e+05,-5.504227813799985597e+04,-1.044813875036323880e+
  05,1.282915103985938731e+04,1.064096746204912876e+02,-2.410437563539280745e+
  04,1.572217804190125025e+05,-1.132422428463703181e+04,-9.854870727899871417e+
  04,4.142831554732488585e+05,1.580409339389271627e+05,1.090890793719776673e+
  06,-6.285607784362095884e+00,2.184942850230274416e+04,-3.227785413232819410e+
  05,1.225238626494904202e+04,-5.974468892584493151e+05)
  b: (-3.497608995597991161e+05,1.814888823864474252e+05,1.108199560069939034e+
  05,2.518720361409108154e+05,-5.504227813799987052e+04,-1.044813875036323880e+
  05,1.282915103985938913e+04,1.064096746204912733e+02,-2.410437563539280745e+
  04,1.572217804190125025e+05,-1.132422428463703181e+04,-9.854870727899869962e+
  04,4.142831554732488585e+05,1.580409339389271918e+05,1.090890793719776440e+
  06,-6.285607784371444851e+00,2.184942850230274053e+04,-3.227785413232819410e+
  05,1.225238626494903474e+04,-5.974468892584493151e+05)
  a: -6.285607784362095884e+00
  b: -6.285607784371444851e+00

===============================================================================
test cases:     1 |     0 passed | 1 failed
assertions: 68212 | 68210 passed | 2 failed

(Since you have to touch the file anyway, you could also remove the three unused data_size variables.
But that's really a separate issue.)
@wthrowe wthrowe added the bug label Jan 31, 2019
@prayush prayush mentioned this issue Feb 4, 2019
Merged
9 tasks
@prayush
Copy link
Contributor

prayush commented Feb 4, 2019

Fix submitted through #1308

@prayush prayush mentioned this issue Feb 4, 2019
Closed
@kidder kidder mentioned this issue Feb 6, 2019
Merged
9 tasks
@kidder
Copy link
Member

kidder commented May 2, 2019

@prayush I just encountered this again:

/work/spectre/tests/Unit/Evolution/Systems/GeneralizedHarmonic/Test_Characteristics.cpp:484
...............................................................................
/work/spectre/tests/Unit/TestingFramework.hpp:137: FAILED:
  CHECK( a == appx(b) )
with expansion:
  9.7316589239 == Approx( 9.7316589206 )
with messages:
  seed: 2834253179
  function: char_field_uzero
<omitting lots of numbers>
a: (-2.646459324098662473e+06,-6.623881911211770494e+05,-3.
  460668056899430603e+07,-1.881593526739295945e+07,-1.169702334863626771e+07,-
  1.062692606329838745e+07,9.214647117815436795e+05,3.335254708680156991e+07,2.
  337583580758304894e+07,2.120499109940140694e+07,9.731658923871229661e+00,3.
  485446470259945840e+06,2.849507590567802265e+07,-5.449359050569123030e+07,5.
  447775697295981646e+07,3.929529811376248300e+07,5.406056532068995386e+07,-2.
  363233877441792190e+07,-3.324039849719322287e+06,-3.746671256504593790e+07)
  b: (-2.646459324098662939e+06,-6.623881911211772822e+05,-3.
  460668056899430603e+07,-1.881593526739295945e+07,-1.169702334863626771e+07,-
  1.062692606329839304e+07,9.214647117815440288e+05,3.335254708680156991e+07,2.
  337583580758304521e+07,2.120499109940140322e+07,9.731658920645713806e+00,3.
  485446470259946771e+06,2.849507590567802265e+07,-5.449359050569123030e+07,5.
  447775697295979410e+07,3.929529811376247555e+07,5.406056532068993896e+07,-2.
  363233877441792190e+07,-3.324039849719323218e+06,-3.746671256504593790e+07)
  a: 9.731658923871229661e+00
  b: 9.731658920645713806e+00

@wthrowe
Copy link
Member Author

wthrowe commented May 21, 2019 

-------------------------------------------------------------------------------
Unit.Evolution.Systems.GeneralizedHarmonic.Characteristics
-------------------------------------------------------------------------------
/home/wthrowe/spectre/tests/Unit/Evolution/Systems/GeneralizedHarmonic/Test_Characteristics.cpp:484
...............................................................................

/home/wthrowe/spectre/tests/Unit/TestingFramework.hpp:137: FAILED:
  CHECK( a == appx(b) )
with expansion:
  -36.1379028198 == Approx( -36.1379028127 )
with messages:
  seed: 3397765079
  function: char_field_uzero
<snip>
  a: -3.613790281976936569e+01
  b: -3.613790281271133864e+01

@kidder
Copy link
Member

kidder commented May 24, 2019 

/work/spectre/tests/Unit/TestingFramework.hpp:137: FAILED:
  CHECK( a == appx(b) )
with expansion:
  -4.7335822595 == Approx( -4.7335822595 )
with messages:
  seed: 838041581
  function: char_field_uplus
<snip>
 a: -4.733582259461172725e+00
  b: -4.733582259519380386e+00

@kidder
Copy link
Member

kidder commented May 24, 2019 

/work/spectre/tests/Unit/TestingFramework.hpp:137: FAILED:
  CHECK( a == appx(b) )
with expansion:
  -0.8272465828 == Approx( -0.8272465829 )
with messages:
  seed: 1224482130
  function: char_field_uplus
<snip>
  a: -8.272465828376900365e-01
  b: -8.272465828896997664e-01

@wthrowe
Copy link
Member Author

wthrowe commented Jun 14, 2019

3503003868

@nilsdeppe
Copy link
Member

Another one: 627095092

@geoffrey4444 @prayush these random failures need to be fixed ASAP please. They're impeding development on other parts of the code with their overall frequency...

@nilsdeppe nilsdeppe mentioned this issue Jul 17, 2019
Merged
7 tasks
@prayush prayush mentioned this issue Jul 22, 2019
Merged
9 tasks
@prayush
Copy link
Contributor

prayush commented Jul 22, 2019

@nilsdeppe @wthrowe @kidder just added a fix, see #1620

nilsdeppe added a commit that referenced this issue Jul 24, 2019
@nilsdeppe
Merge pull request #1620 from prayush/fix_rnd_gh_failures
a8a8d3d 
Fix for #1308 #1315
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prayush prayush

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix for #1308 #1315 prayush/spectre
4 participants
@wthrowe @kidder @prayush @nilsdeppe