Benchmark Reader and inline its requests.

[patrickt]

• Depends on Remove Codensity-related benchmarks. #346.

Fixes #345.

Benchmark results follow:

With -O1

Before any optimizations:

benchmarked Control.Carrier.Reader/ask/10
time                 9.877 ns   (9.380 ns .. 10.43 ns)
                     0.985 R²   (0.971 R² .. 0.997 R²)
mean                 10.46 ns   (10.17 ns .. 11.26 ns)
std dev              1.444 ns   (741.9 ps .. 2.451 ns)
variance introduced by outliers: 75% (severely inflated)

benchmarked Control.Carrier.Reader/ask/100
time                 47.77 ns   (46.73 ns .. 48.80 ns)
                     0.998 R²   (0.997 R² .. 0.999 R²)
mean                 47.67 ns   (47.39 ns .. 48.05 ns)
std dev              1.146 ns   (928.0 ps .. 1.382 ns)

benchmarked Control.Carrier.Reader/ask/1000
time                 330.7 ns   (303.9 ns .. 353.1 ns)
                     0.984 R²   (0.978 R² .. 0.998 R²)
mean                 310.2 ns   (306.4 ns .. 315.9 ns)
std dev              15.17 ns   (10.61 ns .. 21.86 ns)
variance introduced by outliers: 27% (moderately inflated)

benchmarked Control.Carrier.Reader/local/10
time                 10.04 ns   (9.919 ns .. 10.14 ns)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 10.02 ns   (9.968 ns .. 10.09 ns)
std dev              205.6 ps   (164.1 ps .. 277.0 ps)

benchmarked Control.Carrier.Reader/local/100
time                 48.32 ns   (47.30 ns .. 49.70 ns)
                     0.996 R²   (0.994 R² .. 0.998 R²)
mean                 47.85 ns   (47.48 ns .. 48.27 ns)
std dev              1.325 ns   (1.041 ns .. 1.875 ns)
variance introduced by outliers: 11% (moderately inflated)

benchmarked Control.Carrier.Reader/local/1000
time                 310.3 ns   (303.8 ns .. 318.1 ns)
                     0.996 R²   (0.994 R² .. 0.999 R²)
mean                 312.0 ns   (309.3 ns .. 316.0 ns)
std dev              11.23 ns   (8.171 ns .. 17.49 ns)
variance introduced by outliers: 18% (moderately inflated)

Adding INLINE everywhere:

benchmarked Control.Carrier.Reader/ask/10
time                 9.747 ns   (9.649 ns .. 9.867 ns)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 9.747 ns   (9.687 ns .. 9.831 ns)
std dev              250.9 ps   (193.4 ps .. 347.7 ps)
variance introduced by outliers: 11% (moderately inflated)

benchmarked Control.Carrier.Reader/ask/100
time                 47.14 ns   (45.48 ns .. 49.72 ns)
                     0.987 R²   (0.970 R² .. 0.997 R²)
mean                 47.84 ns   (47.22 ns .. 48.80 ns)
std dev              2.469 ns   (1.622 ns .. 3.721 ns)
variance introduced by outliers: 30% (moderately inflated)

benchmarked Control.Carrier.Reader/ask/1000
time                 315.8 ns   (302.9 ns .. 330.6 ns)
                     0.988 R²   (0.981 R² .. 0.995 R²)
mean                 316.5 ns   (311.5 ns .. 327.6 ns)
std dev              23.56 ns   (16.18 ns .. 37.31 ns)
variance introduced by outliers: 46% (moderately inflated)

benchmarked Control.Carrier.Reader/local/10
time                 9.599 ns   (9.260 ns .. 9.838 ns)
                     0.992 R²   (0.982 R² .. 0.997 R²)
mean                 10.21 ns   (10.02 ns .. 10.61 ns)
std dev              812.9 ps   (509.7 ps .. 1.261 ns)
variance introduced by outliers: 52% (severely inflated)

benchmarked Control.Carrier.Reader/local/100
time                 46.47 ns   (44.86 ns .. 48.61 ns)
                     0.988 R²   (0.981 R² .. 0.996 R²)
mean                 49.60 ns   (48.58 ns .. 51.17 ns)
std dev              4.818 ns   (3.576 ns .. 6.936 ns)
variance introduced by outliers: 60% (severely inflated)

benchmarked Control.Carrier.Reader/local/1000
time                 294.5 ns   (283.1 ns .. 307.2 ns)
                     0.991 R²   (0.985 R² .. 0.997 R²)
mean                 313.5 ns   (306.1 ns .. 323.5 ns)
std dev              31.43 ns   (19.41 ns .. 47.06 ns)
variance introduced by outliers: 64% (severely inflated)

Changing those INLINEs to INLINABLEs:

benchmarked Control.Carrier.Reader/ask/10
time                 10.02 ns   (9.678 ns .. 10.34 ns)
                     0.995 R²   (0.991 R² .. 0.999 R²)
mean                 9.776 ns   (9.699 ns .. 9.875 ns)
std dev              297.9 ps   (222.8 ps .. 422.9 ps)
variance introduced by outliers: 13% (moderately inflated)

benchmarked Control.Carrier.Reader/ask/100
time                 45.43 ns   (44.83 ns .. 46.10 ns)
                     0.998 R²   (0.997 R² .. 0.999 R²)
mean                 46.09 ns   (45.78 ns .. 46.52 ns)
std dev              1.293 ns   (1.068 ns .. 1.806 ns)
variance introduced by outliers: 11% (moderately inflated)

benchmarked Control.Carrier.Reader/ask/1000
time                 323.7 ns   (314.4 ns .. 336.2 ns)
                     0.986 R²   (0.969 R² .. 0.999 R²)
mean                 306.1 ns   (301.6 ns .. 314.4 ns)
std dev              19.80 ns   (13.56 ns .. 32.49 ns)
variance introduced by outliers: 39% (moderately inflated)

benchmarked Control.Carrier.Reader/local/10
time                 10.40 ns   (10.06 ns .. 10.66 ns)
                     0.992 R²   (0.986 R² .. 0.997 R²)
mean                 10.48 ns   (10.33 ns .. 10.77 ns)
std dev              682.9 ps   (398.2 ps .. 1.138 ns)
variance introduced by outliers: 41% (moderately inflated)

benchmarked Control.Carrier.Reader/local/100
time                 49.73 ns   (47.69 ns .. 52.46 ns)
                     0.987 R²   (0.978 R² .. 0.995 R²)
mean                 49.21 ns   (48.51 ns .. 50.83 ns)
std dev              3.673 ns   (1.984 ns .. 6.185 ns)
variance introduced by outliers: 46% (moderately inflated)

benchmarked Control.Carrier.Reader/local/1000
time                 318.8 ns   (310.0 ns .. 335.7 ns)
                     0.982 R²   (0.958 R² .. 0.997 R²)
mean                 310.9 ns   (306.1 ns .. 319.9 ns)
std dev              19.42 ns   (10.20 ns .. 33.75 ns)

With -O2

With INLINE:

benchmarked Control.Carrier.Reader/ask/10
time                 28.31 ns   (27.76 ns .. 28.85 ns)
                     0.997 R²   (0.995 R² .. 0.999 R²)
mean                 28.75 ns   (28.48 ns .. 29.06 ns)
std dev              1.012 ns   (727.3 ps .. 1.508 ns)
variance introduced by outliers: 16% (moderately inflated)

benchmarked Control.Carrier.Reader/ask/100
time                 290.0 ns   (273.4 ns .. 312.0 ns)
                     0.969 R²   (0.940 R² .. 0.997 R²)
mean                 272.5 ns   (268.3 ns .. 281.2 ns)
std dev              20.21 ns   (11.20 ns .. 36.37 ns)
variance introduced by outliers: 48% (moderately inflated)

benchmarked Control.Carrier.Reader/ask/1000
time                 2.311 μs   (2.285 μs .. 2.349 μs)
                     0.998 R²   (0.997 R² .. 1.000 R²)
mean                 2.363 μs   (2.348 μs .. 2.388 μs)
std dev              63.84 ns   (48.78 ns .. 83.32 ns)
variance introduced by outliers: 11% (moderately inflated)

benchmarked Control.Carrier.Reader/local/10
time                 28.83 ns   (28.35 ns .. 29.22 ns)
                     0.998 R²   (0.996 R² .. 0.999 R²)
mean                 28.74 ns   (28.48 ns .. 29.14 ns)
std dev              1.009 ns   (589.8 ps .. 1.523 ns)
variance introduced by outliers: 16% (moderately inflated)

benchmarked Control.Carrier.Reader/local/100
time                 285.8 ns   (275.7 ns .. 304.1 ns)
                     0.966 R²   (0.924 R² .. 0.995 R²)
mean                 275.2 ns   (268.4 ns .. 286.3 ns)
std dev              27.90 ns   (16.98 ns .. 40.98 ns)
variance introduced by outliers: 63% (severely inflated)

benchmarked Control.Carrier.Reader/local/1000
time                 2.446 μs   (2.390 μs .. 2.523 μs)
                     0.990 R²   (0.973 R² .. 0.999 R²)
mean                 2.468 μs   (2.437 μs .. 2.527 μs)
std dev              140.8 ns   (83.78 ns .. 238.9 ns)
variance introduced by outliers: 36% (moderately inflated)

With INLINABLE:

benchmarked Control.Carrier.Reader/ask/10
time                 33.39 ns   (30.38 ns .. 36.10 ns)
                     0.935 R²   (0.881 R² .. 0.977 R²)
mean                 33.31 ns   (31.69 ns .. 36.39 ns)
std dev              6.752 ns   (3.154 ns .. 11.40 ns)
variance introduced by outliers: 87% (severely inflated)

benchmarked Control.Carrier.Reader/ask/100
time                 306.3 ns   (272.7 ns .. 337.1 ns)
                     0.953 R²   (0.911 R² .. 0.989 R²)
mean                 293.5 ns   (284.8 ns .. 307.7 ns)
std dev              36.90 ns   (18.22 ns .. 65.44 ns)
variance introduced by outliers: 73% (severely inflated)

benchmarked Control.Carrier.Reader/ask/1000
time                 2.492 μs   (2.389 μs .. 2.630 μs)
                     0.974 R²   (0.946 R² .. 0.993 R²)
mean                 2.746 μs   (2.665 μs .. 2.858 μs)
std dev              321.8 ns   (246.6 ns .. 423.9 ns)
variance introduced by outliers: 71% (severely inflated)

benchmarked Control.Carrier.Reader/local/10
time                 27.69 ns   (26.91 ns .. 28.70 ns)
                     0.987 R²   (0.974 R² .. 0.994 R²)
mean                 31.68 ns   (30.93 ns .. 32.68 ns)
std dev              2.748 ns   (2.209 ns .. 3.429 ns)
variance introduced by outliers: 56% (severely inflated)

benchmarked Control.Carrier.Reader/local/100
time                 293.8 ns   (277.3 ns .. 310.3 ns)
                     0.986 R²   (0.980 R² .. 0.996 R²)
mean                 268.3 ns   (264.7 ns .. 274.9 ns)
std dev              14.90 ns   (10.46 ns .. 20.70 ns)
variance introduced by outliers: 33% (moderately inflated)

benchmarked Control.Carrier.Reader/local/1000
time                 2.421 μs   (2.340 μs .. 2.486 μs)
                     0.985 R²   (0.970 R² .. 0.994 R²)
mean                 2.558 μs   (2.480 μs .. 2.705 μs)
std dev              363.6 ns   (250.6 ns .. 531.5 ns)
variance introduced by outliers: 80% (severely inflated)

[ocharles]

Am I reading this right that it gets worse with -O2? The last benchmark has:

benchmarked Control.Carrier.Reader/ask/1000
time                 2.492 μs   (2.389 μs .. 2.630 μs)
                     0.974 R²   (0.946 R² .. 0.993 R²)
mean                 2.746 μs   (2.665 μs .. 2.858 μs)
std dev              321.8 ns   (246.6 ns .. 423.9 ns)
variance introduced by outliers: 71% (severely inflated)

But your first benchmark has

benchmarked Control.Carrier.Reader/ask/1000
time                 330.7 ns   (303.9 ns .. 353.1 ns)
                     0.984 R²   (0.978 R² .. 0.998 R²)
mean                 310.2 ns   (306.4 ns .. 315.9 ns)
std dev              15.17 ns   (10.61 ns .. 21.86 ns)
variance introduced by outliers: 27% (moderately inflated)

So it looks like things have worse, not better (from 310.2ns to 2.746μs)

[patrickt]

@ocharles Yeah, @robrix and I were looking at this. It’s unclear what’s going on, but I’m hesitant to dig in until we have concrete evidence that this is affecting real-world apps—µbenchmarks aren’t to be trusted, on principle.

[robrix]

@patrickt: I’d rather not close out #345 as fixed until we’ve INLINEd every effect’s operators. I don’t think we need to benchmark all of them, tho; would you mind INLINEing the others?

Just read your comments on #345, mea culpa 😊

[robrix]

Also, I don’t think the INLINE vs. INLINABLE thing is really conclusive; given the variance, I think it could go either way. Unlike the -O2 thing, I think this does warrant some investigation of the core; what’s your take?

[robrix]

(It occurs to me that we could benchmark them against each other directly by duplicating the constructors in separate modules or something, too; that’d be another tactic we could try.)

[robrix]

Upon reflection, I’m pretty ok with INLINE. These are tiny definitions and I think it’s reasonable to be opinionated about inlining them.