Roadmap

[japaric]

Subject to change without prior notice

Functionality:

• Classification of outliers
• Draw bootstrap distribution of statistics (mean,median,SD,MAD)
• Draw bootstrap distribution of the relative change of the {mean,median,slope}
• Check if the relative change of the (mean,median) is "significant"
  • Add threshold noise (percentage) field to the Criterion struct
  • If the whole confidence interval is below minus threshold noise,
    there has been a improvement
  • If the whole confidence interval is above plus threshold noise,
    there has been a regression
  • fail! if both the mean and median have regressed
• Benchmark external programs
  • Decide on the interface that the external program must implement
    • for each line in stdin; niters = parse line; print ns_elapsed for niters
  • Hack up an implementation
  • Refactor/DRY the implementation
• Summarize a bench group
  • Plot {mean,median,slope} vs input + error bars (ci)
    • add error bars for confidence intervals
    • automagically trigger log scale when appropriate
  • Draw a trendline
    • Show the formula of the trendline + R^2
    • Draw the confidence interval of the trendline
• Implement #[criterion] with procedural macros
  • Try to replicate functionality of the --bench flag
  • Or hide the benchs as tests behind [cfg(bench)]
• Use euler problems as a testing ground
• Document the analysis process
• Collect sample with a different number of iterations on each measurement
  • Number of iterations follow an arithmetic progression
  • Store iteration/total time pairs
• Linear regression
• Factor out statistics related code to a library

Directory hierarchy:

.criterion
`-- fib
    |-- 10
    |   |-- base
    |   |   |-- MAD.svg
    |   |   |-- SD.svg
    |   |   |-- estimates.json
    |   |   |-- mean.svg
    |   |   |-- median.svg
    |   |   |-- outliers.json
    |   |   |-- pdf.svg
    |   |   |-- regression.svg
    |   |   |-- sample.json
    |   |   `-- slope.svg
    |   |-- both
    |   |   |-- pdf.svg
    |   |   `-- regression.svg
    |   |-- change
    |   |   |-- estimates.json
    |   |   |-- mean.svg
    |   |   |-- median.svg
    |   |   `-- t-test.svg
    |   `-- new
    |       `-- (...)  # Same as `base`
    |-- {15,20,5}
    |   `-- (...)  # Same as `10`
    `-- summary
        |-- base
        |   |-- means.svg
        |   |-- medians.svg
        |   `-- slopes.svg
        `-- new
            `-- (...)  # Same as `base`

Plots/Data dumps:

• sample.json: Iteration/total time pair stored as u64
• estimates.json: Estimate of several statistics
• pdf.svg
  • sample PDF
  • sample points
  • color points depending on their "outlierness"
  • fence lines
• outliers.json: Store outlier count/fences
• both/pdf.svg: base and new PDF on the same graph
• both/regression.svg: base and new linear regressions on the same graph
• {base,new}/{mean,median,SD,MAD}.svg
  • PDF of the bootstrap distribution
  • Colored CI area
  • Point estimate
• change/{mean,median}.svg
  • PDF of the bootstrap distribution
  • Colored CI area
  • Point estimate
  • Colored threshold noise
• summary/{base,new}/{mean,median}.svg
  • point estimates
  • error bars (CI)
  • trend line + formula

Also, plots must have:

• Title
• X/Y labels
• properly scaled X/Y axis
• Legend (if necessary)

Wishlist:

• plots should use:
  • "nice" colors
  • "nice" line styles
  • A "nice" font
• JSON files
  • should be "pretty" encoded

[japaric]

The rest of issues are now being tracked individually: #11, #12, #13, #14, #15.