Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make get_put_monitor_eqc pass deterministically #857

Merged
merged 3 commits into from
Mar 3, 2014

Conversation

andrewjstone
Copy link
Contributor

  • Change check_state to be the optional invariant callback.
  • Just return true from postcondition since it was only used to check
    the state.
  • remove unneccessary timer:sleep/1 calls
  • Poll for folsom stat changes instead of relying on timer:sleep. This will fix
    the test on slow machines such as the builders. It also makes the test
    run a lot faster as it no longer sleeps arbitrarily.
  • Move test initialization out of the initial_state callback
  • Log the metric that failed for invariants
  • Poll active stats during get and put fsm shutdown/exit since the
    count needs to be decremented
  • swap ordering of monitoring the pid and sending it a kill msg in
    end_and_wait/2

 * Change check_state to be the optional invariant callback.
 * Just return true from postcondition since it was only used to check
   the state.
 * remove unneccessary `timer:sleep/1` calls
 * Poll for folsom stat changes instead of relying on timer:sleep. This will fix
   the test on slow machines such as the builders. It also makes the test
   run a lot faster as it no longer sleeps arbitrarily.
 * Move test initialization out of the initial_state callback
 * Log the metric that failed for invariants
 * Poll active stats during get and put fsm shutdown/exit since the
   count needs to be decremented
 * swap ordering of monitoring the pid and sending it a kill msg in
   `end_and_wait/2`
@jrwest
Copy link
Contributor

jrwest commented Mar 1, 2014

Huge improvement but not 100% there yet. Still seeing a regular failure due to a race when running 10k iterations. the test passes using eqc:check every time afterwards.

Failed Metric = {riak_kv,node,gets,fsm,active}
QC result: {invariant,
               {'EXIT',
                   {{assertEqual_failed,
                        [{module,get_put_monitor_eqc},
                         {line,163},
                         {expression,"Val"},
                         {expected,3},
                         {value,4}]},
                    [{get_put_monitor_eqc,'-invariant/1-fun-0-',2,
                         [{file,"test/get_put_monitor_eqc.erl"},{line,163}]},
                     {get_put_monitor_eqc,'-invariant/1-lc$^0/1-0-',1,
                         [{file,"test/get_put_monitor_eqc.erl"},{line,163}]},
                     {get_put_monitor_eqc,invariant,1,
                         [{file,"test/get_put_monitor_eqc.erl"},{line,157}]},
                     {eqc_statem,run_commands,2,
                         [{file,"../src/eqc_statem.erl"},{line,704}]},
                     {get_put_monitor_eqc,'-prop/0-fun-0-',1,
                         [{file,"test/get_put_monitor_eqc.erl"},
                          {line,50}]}]}}}
Shrinking.(1 times)
[{set,{var,1},{call,get_put_monitor_eqc,get_fsm_started,[]}},
 {set,{var,2},{call,get_put_monitor_eqc,put_fsm_started,[]}},
 {set,{var,3},{call,get_put_monitor_eqc,put_fsm_noproc,[]}},
 {set,{var,4},{call,get_put_monitor_eqc,put_fsm_exit_normal,[put,{var,2}]}},
 {set,{var,5},{call,get_put_monitor_eqc,get_fsm_started,[]}},
 {set,{var,6},{call,get_put_monitor_eqc,get_fsm_started,[]}},
 {set,{var,7},{call,get_put_monitor_eqc,get_fsm_noproc,[]}}]

?FORALL(Cmds, commands(?MODULE), begin
crypto:start(),
application:start(folsom),
reset_test_state(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its good to do clean up after the iteration runs, also. That way the last iteration doesn't leave around artifacts that mess with tests that run later in the suite.

 * process errors also decrement the active process count, so poll for
   those
 * syncrhonously run the noproc tests to avoid races
 * Run tests for 5000 iterations
 * Cleanup non-idomatic assertions in invariant/1
@andrewjstone
Copy link
Contributor Author

I used eqc_statem:pretty_commands/4 instead of ?WHENFAIL

@jrwest
Copy link
Contributor

jrwest commented Mar 3, 2014

+1 merge. 110k iterations passed. nice work.

andrewjstone added a commit that referenced this pull request Mar 3, 2014
Make get_put_monitor_eqc pass deterministically
@andrewjstone andrewjstone merged commit 29f9d04 into develop Mar 3, 2014
@seancribbs seancribbs deleted the bugfix/ajs/fix-get_put_monitor_eqc branch April 1, 2015 23:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants