Performance regression

[jmboehm]

I have replicated the problem I was having in #95 in a smaller example.

On

DataFrames v0.19.4
FixedEffectModels v0.7.4
FixedEffects v0.1.2

running

using Random, DataFrames, FixedEffectModels, BenchmarkTools

Random.seed!(0)
n = 5_000_000
df = DataFrame(y = rand(Float64, n), x = rand(Float64, n), z = rand(Float64, n), fe1 = rand(collect(0:1_000), n), fe2 = rand(collect(0:1_000), n))
df[:fe1] = categorical(df[:fe1]);
df[:fe2] = categorical(df[:fe2]);
@btime rr1s1 = reg(df, @model(y ~ x , fe = fe1*z + fe2*z), save = :residuals)

yields an benchmark time of 2.223s

On the other hand, on

DataFrames v0.20.2
FixedEffectModels v0.10.5
FixedEffects v0.7.2

running

using Random, DataFrames, FixedEffectModels, BenchmarkTools
Random.seed!(0)
n = 5_000_000
df = DataFrame(y = rand(Float64, n), x = rand(Float64, n), z = rand(Float64, n), fe1 = rand(collect(0:1_000), n), fe2 = rand(collect(0:1_000), n))
df[:fe1] = categorical(df[:fe1]);
df[:fe2] = categorical(df[:fe2]);
@btime rr1s1 = reg(df, @formula(y ~ x + fe(fe1)*z + fe(fe2)*z), save = :residuals)

yields a benchmark time of 2.95s.

Both are with nthreads of 4 (it's similar with 1).

I've also tried this on larger examples (500_000 and 10_000 levels of fe1 and fe2 respectively) and the time differences are similar, 350s vs 495s.

Any sense of what the reason could be?

[matthieugomez]

It's really hard to understand a performance drop by something as small as 30%.
I have checked and it is not related to the demeaning operation.
I have tried to improve a bit the construction of fixed effects for the case of interacted fixed effects, I think it improves the performance a bit, see #98
On my computer, after doing these changes, I no longer get differences in performance for the following case

using Random, DataFrames, FixedEffectModels
Random.seed!(0)
n = 50_000_000
df = DataFrame(y = rand(Float64, n), x = rand(Float64, n), z = rand(Float64, n), fe1 = rand(collect(0:5_000), n), fe2 = rand(collect(0:5_000), n))
df.fe1 = categorical(df.fe1)
df.fe2 = categorical(df.fe2)
@time rr1s1 = reg(df, @formula(y ~ x + fe(fe1)*z + fe(fe2)*z))

[matthieugomez]

You started with a regression that took 1/2h in past versions, and more than a night with the current version. Is that still the case?

[jmboehm]

I can confirm that there are not performance differences for the example anymore. Unfortunately, the regression that on 0.7.4 takes 1/2h is still not done after 6h with matthieugomez/FixedEffectModels.jl#master.

If you want to look more into it, I can send you the data (but no worries if not -- I'll use the old version as a workaround).

[matthieugomez]

Yes I can look into it.

[matthieugomez]

Ok, it should be solved with the new version of FixedEffects v0.7.3. reopen if not.