Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shrinker adjustments #277

Merged
merged 7 commits into from May 2, 2023
Merged

Shrinker adjustments #277

merged 7 commits into from May 2, 2023

Conversation

jmid
Copy link
Collaborator

@jmid jmid commented May 1, 2023

This PR fixes #241 causing the QCheck.Shrink.int* shrinkers to emit duplicates (read: wasted effort) for certain inputs.

I was re-reminded of this while attempting to improve the Shrink.string shrinker, as it uses Shrink.char which again uses Shrink.int underneath and hence inherits the bug.

The bug is fixed by slightly expanding the condition after the while loop in all 3 int{,32,64} cases.
The expect test updates nicely eliminates quite a few WTF? cases from the unit test suite.

While looking at the code, I realized that Shrink.list_spine could also emit duplicate shrinking candidates as a base-case inside its recursive approach, so I added a small fix to that too.

To document that Shrink.string may still be too exhaustive I've added unit test examples of it.
80c8c88 contains a commit with the current behaviour and f37621a then improves it.
As the test shows, there is still room for improvement.

Finally with the shrinker benchmark from #177 restored in #274 I ran it to measure the difference.
Time-wise it is a small improvement to the QCheck(1) shrinkers, but there are occasional drops in the number of shrink attempts which may matter when testing more expensive properties.
Some highlight examples include:

  • bind list_size constant drops from 230 to 207 shrink attempts
  • lists shorter than 432 drops from 20417 to 18727 shrink attempts
  • fail_pred_map_commute drops from 507 to 444 shrink attempts
  • fold_left fold_right uncurried drops from 724 to 645 shrink attempts

Here's a selection before (only the Q1 columns are relevant):

                                                         iteration seed 1234                   iteration seed 8743                   iteration seed 6789               total
Shrink test name                                  Q1/s  #succ/#att   Q2/s  #succ/#att   Q1/s  #succ/#att   Q2/s  #succ/#att   Q1/s  #succ/#att   Q2/s  #succ/#att    Q1/s   Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
...
bytes have unique chars                           0.000   14/46      0.000   14/29      0.000   11/42      0.000   12/22      0.000    4/25      0.000    7/11      0.000  0.000
strings have unique chars                         0.000   14/46      0.000   14/29      0.000   11/42      0.000   12/22      0.000    4/25      0.000    7/11      0.000  0.000
pairs have different components                   0.000    0/6       0.000    0/6       0.000    0/4       0.000    0/8       0.000    0/4       0.000    0/8       0.000  0.000
pairs are ordered                                 0.000  554/10546   0.000   88/1052    0.000  577/11088   0.000   93/1225    0.000  646/13180   0.000   86/949     0.001  0.001
triples have pair-wise different components       0.000    4/34      0.000    3/12      0.000    3/9       0.000    3/3       0.000    6/10      0.000    3/3       0.000  0.000
triples are ordered                               0.000  184/185     0.000    3/4       0.000  572/9940    0.000   95/1288    0.000  184/247     0.000   95/1227    0.000  0.001
quadruples are ordered                            0.000  247/248     0.000    4/5       0.001  910/16505   0.000   90/981     0.000  247/310     0.000   96/1244    0.001  0.000
forall (a, b) in nat: a < b                       0.000    5/7       0.000    3/6       0.000    8/14      0.000    2/6       0.000    8/14      0.000    2/6       0.000  0.000
forall (a, b, c) in nat: a < b < c                0.000   14/18      0.000    5/13      0.000   14/22      0.000    4/10      0.000   14/19      0.000    3/3       0.000  0.000
forall (a, b, c, d) in nat: a < b < c < d         0.000   17/20      0.000    4/4       0.000   18/25      0.000    4/4       0.000   18/18      0.000    4/4       0.000  0.000
bind list_size constant                           0.000   52/230     0.000   12/25      0.000   49/227     0.000    9/19      0.000   43/192     0.000   10/19      0.000  0.000
lists shorter than 10                             0.000   49/315     0.000   15/24      0.000   50/315     0.000   17/34      0.000   36/236     0.000   13/28      0.000  0.000
lists shorter than 432                            0.250 1738/20737   1.098  413/448     0.157 1667/19905   0.403  419/461     0.258 1712/20417   0.938  422/471     0.665  2.439
lists have unique elems                           0.000   10/22      0.000   10/21      0.000    3/10      0.000    5/9       0.000    2/14      0.000    5/9       0.000  0.000
fail_pred_map_commute                             0.000   79/507     0.010  244/1213    0.000   76/357     0.001  102/320     0.000   76/353     0.006  203/905     0.000  0.017
fold_left fold_right uncurried                    0.004   41/121     0.081  434/807     0.001   68/724     0.001    6/191     0.000   46/269     0.000    6/101     0.006  0.081
fold_left fold_right uncurried fun last           0.000   25/74      0.001   37/151     0.000   19/63      0.000   23/92      0.000   20/62      0.000   12/43      0.000  0.001
...
                                                                                                                                                                    1.221 19.536

Here's the same selection after:

                                                         iteration seed 1234                   iteration seed 8743                   iteration seed 6789               total
Shrink test name                                  Q1/s  #succ/#att   Q2/s  #succ/#att   Q1/s  #succ/#att   Q2/s  #succ/#att   Q1/s  #succ/#att   Q2/s  #succ/#att    Q1/s   Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
...
bytes have unique chars                           0.000   14/45      0.000   14/29      0.000   11/41      0.000   12/22      0.000    4/22      0.000    7/11      0.000  0.000
strings have unique chars                         0.000   14/45      0.000   14/29      0.000   11/41      0.000   12/22      0.000    4/22      0.000    7/11      0.000  0.000
pairs have different components                   0.000    0/4       0.000    0/6       0.000    0/4       0.000    0/8       0.000    0/4       0.000    0/8       0.000  0.000
pairs are ordered                                 0.000  554/10543   0.000   88/1052    0.000  577/11085   0.000   93/1225    0.000  646/13122   0.000   86/949     0.001  0.001
triples have pair-wise different components       0.000    4/24      0.000    3/12      0.000    3/9       0.000    3/3       0.000    6/10      0.000    3/3       0.000  0.000
triples are ordered                               0.000  184/185     0.000    3/4       0.000  572/9938    0.000   95/1288    0.000  184/247     0.000   95/1227    0.000  0.000
quadruples are ordered                            0.000  247/248     0.000    4/5       0.001  910/16447   0.000   90/981     0.000  247/310     0.000   96/1244    0.001  0.000
forall (a, b) in nat: a < b                       0.000    5/7       0.000    3/6       0.000    8/13      0.000    2/6       0.000    8/13      0.000    2/6       0.000  0.000
forall (a, b, c) in nat: a < b < c                0.000   14/18      0.000    5/13      0.000   14/20      0.000    4/10      0.000   14/18      0.000    3/3       0.000  0.000
forall (a, b, c, d) in nat: a < b < c < d         0.000   17/20      0.000    4/4       0.000   18/23      0.000    4/4       0.000   18/18      0.000    4/4       0.000  0.000
bind list_size constant                           0.000   52/207     0.000   12/25      0.000   49/204     0.000    9/19      0.000   43/171     0.000   10/19      0.000  0.000
lists shorter than 10                             0.000   49/279     0.000   15/24      0.000   50/282     0.000   17/34      0.000   36/210     0.000   13/28      0.000  0.000
lists shorter than 432                            0.236 1738/19018   0.963  413/448     0.133 1667/18254   0.346  419/461     0.207 1712/18727   0.831  422/471     0.576  2.140
lists have unique elems                           0.000   10/21      0.000   10/21      0.000    3/9       0.000    5/9       0.000    2/11      0.000    5/9       0.000  0.001
fail_pred_map_commute                             0.000   79/444     0.009  244/1213    0.000   76/357     0.001  102/320     0.000   76/291     0.006  203/905     0.000  0.016
fold_left fold_right uncurried                    0.004   41/121     0.075  434/807     0.001   68/645     0.001    6/191     0.000   46/259     0.000    6/101     0.006  0.075
fold_left fold_right uncurried fun last           0.000   25/73      0.001   37/151     0.000   19/57      0.000   23/92      0.000   20/62      0.000   12/43      0.000  0.001
...
                                                                                                                                                                    1.108 16.773

@jmid jmid merged commit aeb9bd0 into c-cube:main May 2, 2023
11 checks passed
@jmid jmid deleted the shrinker-adjustments branch May 2, 2023 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

QCheck int shrinker produces duplicates
1 participant