Did Optimization v5 break Optimization v4 somehow?

[oschulz]

Since Optimization v5 was released, I can't install Optimization v4 anymore (at least not cleanly). Starting with a fresh, empty Julia depot:

pkg> add Optimization@4
  Installing known registries into `/tmp/jldepot/.julia`
       Added `General` registry to /tmp/jldepot/.julia/registries
    Updating registry at `/tmp/jldepot/.julia/registries/General.toml`
   Resolving package versions...
   Installed Adapt ─────────────────────── v4.4.0
   [...]
Precompiling project...
  ? OptimizationBase
  ? Optimization
  71 dependencies successfully precompiled in 51 seconds. 28 already precompiled.
  2 dependencies failed but may be precompilable after restarting julia
  2 dependencies had output during precompilation:
┌ Optimization
│  WARNING: Method definition (::Type{OptimizationBase.IncompatibleOptimizerError})(String) in module OptimizationBase at /tmp/jldepot/.julia/packages/OptimizationBase/BGm33/src/solve.jl:5 overwritten on the same line (check for duplicate calls to `include`).
│  ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
└  
┌ OptimizationBase
│  WARNING: Method definition (::Type{OptimizationBase.IncompatibleOptimizerError})(String) in module OptimizationBase at /tmp/jldepot/.julia/packages/OptimizationBase/BGm33/src/solve.jl:5 overwritten on the same line (check for duplicate calls to `include`).
│  ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
└  

julia> using Optimization
Precompiling Optimization...
Info Given Optimization was explicitly requested, output will be shown live 
WARNING: Method definition (::Type{OptimizationBase.IncompatibleOptimizerError})(String) in module OptimizationBase at /tmp/jldepot/.julia/packages/OptimizationBase/BGm33/src/solve.jl:5 overwritten on the same line (check for duplicate calls to `include`).
ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
  ? OptimizationBase
  ? Optimization
[ Info: Precompiling Optimization [7f7a1694-90dd-40f0-9382-eb1efda571ba] 
WARNING: Method definition (::Type{OptimizationBase.IncompatibleOptimizerError})(String) in module OptimizationBase at /tmp/jldepot/.julia/packages/OptimizationBase/BGm33/src/solve.jl:5 overwritten on the same line (check for duplicate calls to `include`).
ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
┌ Info: Skipping precompilation due to precompilable error. Importing Optimization [7f7a1694-90dd-40f0-9382-eb1efda571ba].
└   exception = Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.
Precompiling OptimizationBase...
Info Given OptimizationBase was explicitly requested, output will be shown live 
WARNING: Method definition (::Type{OptimizationBase.IncompatibleOptimizerError})(String) in module OptimizationBase at /tmp/jldepot/.julia/packages/OptimizationBase/BGm33/src/solve.jl:5 overwritten on the same line (check for duplicate calls to `include`).
ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
  ? OptimizationBase
[ Info: Precompiling OptimizationBase [bca83a33-5cc9-4baa-983d-23429ab6bcbb] 
WARNING: Method definition (::Type{OptimizationBase.IncompatibleOptimizerError})(String) in module OptimizationBase at /tmp/jldepot/.julia/packages/OptimizationBase/BGm33/src/solve.jl:5 overwritten on the same line (check for duplicate calls to `include`).
ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
┌ Info: Skipping precompilation due to precompilable error. Importing OptimizationBase [bca83a33-5cc9-4baa-983d-23429ab6bcbb].
└   exception = Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.
┌ Warning: Replacing docs for `CommonSolve.solve :: Tuple{SciMLBase.OptimizationProblem, Any, Vararg{Any}}` in module `OptimizationBase`
└ @ Base.Docs docs/Docs.jl:243
┌ Warning: Replacing docs for `CommonSolve.init :: Tuple{SciMLBase.OptimizationProblem, Any, Vararg{Any}}` in module `OptimizationBase`
└ @ Base.Docs docs/Docs.jl:243
┌ Warning: Replacing docs for `CommonSolve.solve! :: Tuple{SciMLBase.AbstractOptimizationCache}` in module `OptimizationBase`
└ @ Base.Docs docs/Docs.jl:243

Happens with Julia v1.11 and v1.12.

Pinning OptimizationBase down to v2.12.0 seems to help, at first:

pkg> add Optimization@4 OptimizationBase@2.12
    Updating registry at `/tmp/jldepot/.julia/registries/General.toml`
   Resolving package versions...
   Installed OptimizationBase ─ v2.12.0
    Updating `/tmp/jldepot/.julia/environments/v1.11/Project.toml`
⌅ [bca83a33] + OptimizationBase v2.12.0
    Updating `/tmp/jldepot/.julia/environments/v1.11/Manifest.toml`
⌅ [bca83a33] ↓ OptimizationBase v2.13.3 ⇒ v2.12.0
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
Precompiling project...
  2 dependencies successfully precompiled in 7 seconds. 99 already precompiled.

But then:

pkg> add OptimizationOptimJL
   Resolving package versions...
   Installed PtrArrays ────────────── v1.3.0
   [...]
Precompiling project...
  ✗ OptimizationOptimJL
  40 dependencies successfully precompiled in 34 seconds. 103 already precompiled.
  1 dependency errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

julia> using OptimizationOptimJL
Precompiling OptimizationOptimJL...
Info Given OptimizationOptimJL was explicitly requested, output will be shown live 
ERROR: LoadError: MethodError: no method matching init(::SciMLBase.OptimizationProblem{true, SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, OptimizationOptimJL.var"#obj_f#38", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, Tuple{Matrix{Float64}, Vector{Float64}}, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, Base.Pairs{Symbol, Union{}, Tuple{}, @NamedTuple{}}}, ::Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.HagerZhang{Float64, Base.RefValue{Bool}}, Returns{Nothing}}; maxiters::Int64, maxtime::Int64)
The function `init` exists, but no method is defined for this combination of argument types.
[...]
Failed to precompile OptimizationOptimJL [36348300-93cb-4f02-beb5-3c3902f8871e] to "/tmp/jldepot/.julia/compiled/v1.11/OptimizationOptimJL/jl_Wj2NTk".
[...]

Somehow Optimization v4 seems to be in dependency hell now.

[ChrisRackauckas]

Sorry should be handled now. 🤞 I think I backported every required commit.

[oschulz]

Hey Chris, I still get the error above when I try to do

pkg> add Optimization@4 OptimizationBase

on an empty repo. Do I need to wait for any releases? Currently this tries to install Optimization v4.8.0 and OptimizationBase v2.13.3 (and fails).

[alecloudenback]

Also seeing this. Is it a matter of just waiting for the fixes to propagate on General and then up the packages in my environment?

[ChrisRackauckas]

I just noticed the OptimizationBase patch didn't get released 😅 didn't bump the version number

[ChrisRackauckas]

Releasing this as v2.14.0 since I messed up the version numbers, let's see if this works...

[ChrisRackauckas]

Okay the release went through, let me know if OptimizationBase v2.14 is good.

[oschulz]

All good now for me - thanks a lot for the quick fix, Chris!