Error with v0.11.0

[amrods]

With the new v0.11.0 I get an error when optimizing a function, I didn't have that error with v0.10.7:

ERROR: LoadError: MethodError: no method matching Cholesky{Float64, Matrix{Float64}}(::Cholesky{Float64, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(γ = 1:2,)}, Axis{(γ = 1:2,)}}}})
Closest candidates are:
  Cholesky{T, S}(::Any, ::Any, ::Any) where {T, S<:(AbstractMatrix{T} where T)} at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/cholesky.jl:84

I will try to track it down later today.

[amrods]

Here is a mwe that shows the problem:

using ComponentArrays
using Optim

f(x; a=1, b=100) = (a - x[1])^2 + b*(x[2] - x[1]^2)^2
optimize(f, ComponentArray(z = [10.0, 10.0]), Newton())

ERROR: MethodError: no method matching LinearAlgebra.Cholesky{Float64, Matrix{Float64}}(::LinearAlgebra.Cholesky{Float64, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(z = 1:2,)}, Axis{(z = 1:2,)}}}})
Closest candidates are:
  LinearAlgebra.Cholesky{T, S}(::Any, ::Any, ::Any) where {T, S<:(AbstractMatrix{T} where T)} at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/cholesky.jl:84
Stacktrace:
 [1] convert(#unused#::Type{LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, f::LinearAlgebra.Cholesky{Float64, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(z = 1:2,)}, Axis{(z = 1:2,)}}}})
   @ LinearAlgebra /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/factorization.jl:56
 [2] setproperty!(x::Optim.NewtonState{ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(z = 1:2,)}}}, Float64, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, f::Symbol, v::LinearAlgebra.Cholesky{Float64, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(z = 1:2,)}, Axis{(z = 1:2,)}}}})
   @ Base ./Base.jl:34
 [3] update_state!(d::TwiceDifferentiable{Float64, ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(z = 1:2,)}}}, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(z = 1:2,)}, Axis{(z = 1:2,)}}}, ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(z = 1:2,)}}}}, state::Optim.NewtonState{ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(z = 1:2,)}}}, Float64, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, method::Newton{LineSearches.InitialStatic{Float64}, LineSearches.HagerZhang{Float64, Base.RefValue{Bool}}})
   @ Optim ~/.julia/packages/Optim/uwNqi/src/multivariate/solvers/second_order/newton.jl:67
 [4] optimize(d::TwiceDifferentiable{Float64, ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(z = 1:2,)}}}, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(z = 1:2,)}, Axis{(z = 1:2,)}}}, ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(z = 1:2,)}}}}, initial_x::ComponentVector{Float64}, method::Newton{LineSearches.InitialStatic{Float64}, LineSearches.HagerZhang{Float64, Base.RefValue{Bool}}}, options::Optim.Options{Float64, Nothing}, state::Optim.NewtonState{ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(z = 1:2,)}}}, Float64, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}})
   @ Optim ~/.julia/packages/Optim/uwNqi/src/multivariate/optimize/optimize.jl:57
 [5] optimize
   @ ~/.julia/packages/Optim/uwNqi/src/multivariate/optimize/optimize.jl:35 [inlined]
 [6] #optimize#87
   @ ~/.julia/packages/Optim/uwNqi/src/multivariate/optimize/interface.jl:142 [inlined]
 [7] top-level scope
   @ REPL[4]:1

[jonniedie]

Uh, this is weird. I'm getting an error, but not that one. What version of Optim are you on?

[amrods]

Optim v1.3.0

[jonniedie]

Actually, would you mind doing a full ]st? I can't reproduce this even with a fresh environment of ComponentArrays v0.11.0 and Optim v1.3.0.

[amrods]

Hmm, you are correct. I must have pasted the wrong error from another window. I still encounter the first error with Cholesky:

ERROR: LoadError: MethodError: no method matching Cholesky{Float64, Matrix{Float64}}(::Cholesky{Float64, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(γ = 1:2,)}, Axis{(γ = 1:2,)}}}})
Closest candidates are:
  Cholesky{T, S}(::Any, ::Any, ::Any) where {T, S<:(AbstractMatrix{T} where T)} at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/cholesky.jl:84
Stacktrace:
 [1] convert(#unused#::Type{Cholesky{Float64, Matrix{Float64}}}, f::Cholesky{Float64, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(γ = 1:2,)}, Axis{(γ = 1:2,)}}}})
   @ LinearAlgebra /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/factorization.jl:56
 [2] setproperty!(x::Optim.NewtonState{ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(γ = 1:2,)}}}, Float64, Cholesky{Float64, Matrix{Float64}}}, f::Symbol, v::Cholesky{Float64, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(γ = 1:2,)}, Axis{(γ = 1:2,)}}}})
   @ Base ./Base.jl:34
 [3] update_state!(d::TwiceDifferentiable{Float64, ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(γ = 1:2,)}}}, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(γ = 1:2,)}, Axis{(γ = 1:2,)}}}, ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(γ = 1:2,)}}}}, state::Optim.NewtonState{ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(γ = 1:2,)}}}, Float64, Cholesky{Float64, Matrix{Float64}}}, method::Newton{InitialStatic{Float64}, HagerZhang{Float64, Base.RefValue{Bool}}})
   @ Optim ~/.julia/packages/Optim/uwNqi/src/multivariate/solvers/second_order/newton.jl:67
 [4] optimize(d::TwiceDifferentiable{Float64, ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(γ = 1:2,)}}}, ComponentMatrix{Float64, Matrix{Float64}, Tuple{Axis{(γ = 1:2,)}, Axis{(γ = 1:2,)}}}, ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(γ = 1:2,)}}}}, initial_x::ComponentVector{Float64}, method::Newton{InitialStatic{Float64}, HagerZhang{Float64, Base.RefValue{Bool}}}, options::Optim.Options{Float64, Nothing}, state::Optim.NewtonState{ComponentVector{Float64, Vector{Float64}, Tuple{Axis{(γ = 1:2,)}}}, Float64, Cholesky{Float64, Matrix{Float64}}})
   @ Optim ~/.julia/packages/Optim/uwNqi/src/multivariate/optimize/optimize.jl:57
 [5] optimize
   @ ~/.julia/packages/Optim/uwNqi/src/multivariate/optimize/optimize.jl:35 [inlined]
 [6] #optimize#87
   @ ~/.julia/packages/Optim/uwNqi/src/multivariate/optimize/interface.jl:142 [inlined]
 [7] ol_psmle(dataca::ComponentVector{Float64})
   @ Main ~/proj1/code/ologit.jl:42
 [8] auxiliarymodel(dataca::ComponentVector{Float64})
   @ Main ~/proj1/code/estimation.jl:80
 [9] top-level scope
   @ ~/proj1/code/estimation.jl:94
in expression starting at /Users/amrods/proj1/code/estimation.jl:94

I'm having trouble finding a simple example, but before updating CA I didn't have it.

[jonniedie]

Now that ComponentArrays follow a more standard interface for broadcasting and axes, there are a lot of places that use similar to create new arrays that used to fall back to creating standard Arrays now (correctly) create ComponentArrays. So there are probably some issues in Optim and other packages that aren't as generic as they should be (but never had problems with ComponentArrays before). This is one example in NLSolversBase that is too rigid to handle arbitrary array types (this is the error I'm hitting, so it must be that something in your environment is blocking NLSolversBase from updating).

With that said, the error you're hitting is something different. It should be able to handle that conversion. I'll have to look into it.

[amrods]

Yeah, there's a PR for that error that hasn't been merged [https://github.com/JuliaNLSolvers/NLSolversBase.jl/pull/140]. I also am trying to track the other issue.

[jonniedie]

Oh cool. A step ahead of me.

As for the Cholesky thing, though, I just don't get why LinearAlgebra isn't handling that conversion correctly. I guess I can add it myself, but I feel like that's a little annoying. I'll see what I can do.

[jonniedie]

This will get it to work for now. I'm not sure I want to do this for every type of factorization, though:

Base.convert(::Type{Cholesky{T,Matrix{T}}}, x::Cholesky{T,<:ComponentArray}) where {T} = Cholesky(copy(getdata(x.factors)), x.uplo, x.info)

[amrods]

Is this problem arising because of some non-generic code in LinearAlgebra as you described above?

[jonniedie]

Yeah sorta. The problem is LinearAlgebra calls the convert function to convert between Cholesky factorization of different array type parameterizations but doesn’t actually provide a method for that conversion.

[amrods]

meta question, how do you track where the error originates, do you run a debugger and see where it stops?

[jonniedie]

Sometimes I use the debugger, but it almost never works like it is supposed to so I'll usually just give up and try something else. In this case, the debugger wasn't even starting because the Julia Language Server in VSCode was having trouble for some reason. I just ended up calling @which or @edit and reading through the code.

[jonniedie]

Well I just ended up pushing the dumb fix for Cholesky only. Hopefully that's all that's needed for Optim stuff at least.