Enhance the tbl property of a parametricbootstrap result

[dmbates]

• allow _generatesyms with string for tag
• incorporate correlation estimates in table of parameter estimates from bootstrap

[codecov]

Codecov Report

Patch coverage is 100.00% of modified lines.

Files Changed	Coverage
src/bootstrap.jl	100.00%
src/profile/utilities.jl	100.00%

📢 Thoughts on this report? Let us know!.

[github-actions]

Benchmark Report for /home/runner/work/MixedModels.jl/MixedModels.jl

Job Properties

• Time of benchmarks:
  • Target: 29 Aug 2023 - 19:44
  • Baseline: 29 Aug 2023 - 19:50
• Package commits:
  • Target: 28f12f
  • Baseline: 6412fe
• Julia commits:
  • Target: bed2cd
  • Baseline: bed2cd
• Julia command flags:
  • Target: None
  • Baseline: -Cnative,-J/opt/hostedtoolcache/julia/1.9.3/x64/lib/julia/sys.so,-g1,-O3,-e,using Pkg; Pkg.update(); Pkg.add(["BenchmarkTools", "StatsModels"])
• Environment variables:
  • Target: None
  • Baseline: None

Results

A ratio greater than 1.0 denotes a possible regression (marked with ❌), while a ratio less
than 1.0 denotes a possible improvement (marked with ✅). Only significant results - results
that indicate possible regressions or improvements - are shown below (thus, an empty table means that all
benchmark results remained invariant between builds).

ID	time ratio	memory ratio
["crossed", "insteval:1"]	0.54 (5%) ✅	0.36 (1%) ✅
["crossed", "kb07:1"]	0.00 (5%) ✅	0.00 (1%) ✅
["crossed", "ml1m:1"]	0.90 (5%) ✅	0.77 (1%) ✅
["crossed", "mrk17_exp1:1"]	0.01 (5%) ✅	0.18 (1%) ✅
["crossedvector", "d3:1"]	0.81 (5%) ✅	0.51 (1%) ✅
["crossedvector", "mrk17_exp1:2"]	0.91 (5%) ✅	0.65 (1%) ✅
["nested", "pastes:2"]	0.97 (5%)	0.98 (1%) ✅
["singlevector", "sleepstudy:3"]	0.00 (5%) ✅	0.00 (1%) ✅

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

• ["crossed"]
• ["crossedvector"]
• ["nested"]
• ["singlevector"]

Julia versioninfo

Target

Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
      Ubuntu 22.04.3 LTS
  uname: Linux 5.15.0-1041-azure #48-Ubuntu SMP Tue Jun 20 20:34:08 UTC 2023 x86_64 x86_64
  CPU: Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz: 
              speed         user         nice          sys         idle          irq
       #1  2095 MHz       6425 s          0 s        233 s        486 s          0 s
       #2  2095 MHz       1435 s          0 s        291 s       5371 s          0 s
  Memory: 6.7694854736328125 GB (4858.171875 MB free)
  Uptime: 719.17 sec
  Load Avg:  1.0  1.07  0.76
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake-avx512)
  Threads: 1 on 2 virtual cores

Baseline

Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
      Ubuntu 22.04.3 LTS
  uname: Linux 5.15.0-1041-azure #48-Ubuntu SMP Tue Jun 20 20:34:08 UTC 2023 x86_64 x86_64
  CPU: Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz: 
              speed         user         nice          sys         idle          irq
       #1  2095 MHz       8093 s          0 s        373 s       2635 s          0 s
       #2  2095 MHz       4751 s          0 s        415 s       5895 s          0 s
  Memory: 6.7694854736328125 GB (4904.3046875 MB free)
  Uptime: 1115.78 sec
  Load Avg:  1.1  1.35  1.04
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake-avx512)
  Threads: 1 on 2 virtual cores

[dmbates]

I believe the wonderful Benchmark comparisons are artificial. See Discussion #712

[dmbates]

@palday I think this is ready to go now. I can add to the documentation later. The general idea is to extract the tbl property of a bootstrap result. Plots of, e.g., empirical density estimates can be made using the wide approach in AlgebraOfGraphics. I will document that later.

[dmbates]

@palday I have updated the documentation but not yet switched the documentation to use AlgebraOfGraphics.jl. As an example of how to use AlgebraOfGraphics on the table of parametricbootstrap results

import CairoMakie
using AlgebraOfGraphics, MixedModels
using MixedModels: dataset

CairoMakie.activate!(; type="svg")
contrasts = Dict{Symbol, Any}(:subj => Grouping());

m01 = let f = @formula reaction ~ 1 + days + (1 + days|subj)
    fit(MixedModel, f, dataset(:sleepstudy); contrasts)
end

m01samp = parametricbootstrap(10_000, m01; hide_progress=true);
tbl = m01samp.tbl

draw(
    let σs = collect(filter(startswith('σ'), string.(propertynames(tbl))))
        data(tbl) *
        mapping(
            σs .=> "Bootstrap replicates of standard deviation parameters",
            color=dims(1) => renamer(σs),
        ) *
        density()
    end
)

Note that if we import CairoMakie instead of using then we do not need to qualify visualization functions like density.

Do you think we should create a function in MixedModelsMakie to encapsulate the operations in the draw call?

[palday]

Do you think we should create a function in MixedModelsMakie to encapsulate the operations in the draw call?

Yes, though potentially AoG support should be in a package extension, in case somebody is using only the Makie and not AoG components.

We could also add support for plotting the sigmas to ridgeplot.

[dmbates]

Alternatively, and more simply, we could save a copy of lambda, create the table and copy the saved lambda back into the bootstrap struct.

I think this is the way to go.

Done.

[palday]

looks good -- I think this counts as a 'feature', so can you

• bump the minor version number
• add a NEWS entry

?

[dmbates]

@palday Would you take one more look and squash and merge if this seems okay to you?