`JET.@test_opt` fails on Julia >= 1.12 due to `AssertionError`s

[devmotion]

I was just rerunning a few tests with JET.@test_opt that involve ForwardDiff.gradient and ForwardDiff.jacobian calls on Julia 1.12.0-rc3.

It seems that the JET analysis became a bit better (stricter?) as tests pass on Julia 1.11.7. On Julia 1.12.0-rc3 I see errors such as:

  ││││││││││││││┌ gradient(f::Pumas.var"#f#_penalty_hessian_full##0"{…}, x::Vector{…}, cfg::ForwardDiff.GradientConfig{…}, ::Val{…}) @ ForwardDiff /build/.julia/packages/ForwardDiff/X74OO/src/gradient.jl:22
  │││││││││││││││┌ chunk_mode_gradient(f::Pumas.var"#f#_penalty_hessian_full##0"{…}, x::Vector{…}, cfg::ForwardDiff.GradientConfig{…}) @ ForwardDiff /build/.julia/packages/ForwardDiff/X74OO/src/gradient.jl:109
  ││││││││││││││││┌ _assert_tostring(msg::String) @ Base ./error.jl:247
  │││││││││││││││││ runtime dispatch detected: %1::Any(msg::String)::Any
  ││││││││││││││││└────────────────────
  │││││││││││││││┌ chunk_mode_gradient(f::Pumas.var"#f#_penalty_hessian_full##0"{…}, x::Vector{…}, cfg::ForwardDiff.GradientConfig{…}) @ ForwardDiff /build/.julia/packages/ForwardDiff/X74OO/src/gradient.jl:109
  ││││││││││││││││ runtime dispatch detected: AssertionError(%416::Any)::AssertionError
  │││││││││││││││└────────────────────
  ││││││││││┌ jacobian(f::Pumas.var"#452#453"{…}, x::SubArray{…}, cfg::ForwardDiff.JacobianConfig{…}, ::Val{…}) @ ForwardDiff /build/.julia/packages/ForwardDiff/X74OO/src/jacobian.jl:24
  │││││││││││┌ chunk_mode_jacobian(f::Pumas.var"#452#453"{…}, x::SubArray{…}, cfg::ForwardDiff.JacobianConfig{…}) @ ForwardDiff /build/.julia/packages/ForwardDiff/X74OO/src/jacobian.jl:172
  ││││││││││││ runtime dispatch detected: AssertionError(%317::Any)::AssertionError
  │││││││││││└────────────────────

[KristofferC]

Can you show the exact invocations you run

[devmotion]

No, it's JET.@test_opt f(....) of a private function f. But the only problematic part is when it calls ForwardDiff.jacobian and ForwardDiff.gradient shown above, so it should be easily reproducible with just ForwardDiff.jacobian and ForwardDiff.gradient calls.

In any case, IMO it would be beneficial to avoid unnecessary non-lazy string interpolations and replace the @assert statements with proper errors (ArgumentErrors?).

[devmotion]

Simple MWE:

julia> using ForwardDiff, JET

julia> @test_opt ForwardDiff.gradient(only, [1.0], ForwardDiff.GradientConfig(only, [1.0], ForwardDiff.Chunk{1}()))
JET-test failed at REPL[8]:1
  Expression: #= REPL[8]:1 =# JET.@test_opt ForwardDiff.gradient(only, [1.0], ForwardDiff.GradientConfig(only, [1.0], ForwardDiff.Chunk{1}()))
  ═════ 2 possible errors found ═════
  ┌ gradient(f::typeof(only), x::Vector{…}, cfg::ForwardDiff.GradientConfig{…}) @ ForwardDiff /Users/david/.julia/packages/ForwardDiff/2nvFM/src/gradient.jl:17
  │┌ gradient(f::typeof(only), x::Vector{…}, cfg::ForwardDiff.GradientConfig{…}, ::Val{…}) @ ForwardDiff /Users/david/.julia/packages/ForwardDiff/2nvFM/src/gradient.jl:22
  ││┌ chunk_mode_gradient(f::typeof(only), x::Vector{…}, cfg::ForwardDiff.GradientConfig{…}) @ ForwardDiff /Users/david/.julia/packages/ForwardDiff/2nvFM/src/gradient.jl:116
  │││┌ _assert_tostring(msg::String) @ Base ./error.jl:247
  ││││ runtime dispatch detected: %1::Any(msg::String)::Any
  │││└────────────────────
  ││┌ chunk_mode_gradient(f::typeof(only), x::Vector{…}, cfg::ForwardDiff.GradientConfig{…}) @ ForwardDiff /Users/david/.julia/packages/ForwardDiff/2nvFM/src/gradient.jl:116
  │││ runtime dispatch detected: AssertionError(%385::Any)::AssertionError
  ││└────────────────────

ERROR: There was an error during testing

julia> @test_opt ForwardDiff.jacobian(identity, [1.0], ForwardDiff.JacobianConfig(identity, [1.0], ForwardDiff.Chunk{1}()))
JET-test failed at REPL[9]:1
  Expression: #= REPL[9]:1 =# JET.@test_opt ForwardDiff.jacobian(identity, [1.0], ForwardDiff.JacobianConfig(identity, [1.0], ForwardDiff.Chunk{1}()))
  ═════ 2 possible errors found ═════
  ┌ jacobian(f::typeof(identity), x::Vector{…}, cfg::ForwardDiff.JacobianConfig{…}) @ ForwardDiff /Users/david/.julia/packages/ForwardDiff/2nvFM/src/jacobian.jl:19
  │┌ jacobian(f::typeof(identity), x::Vector{…}, cfg::ForwardDiff.JacobianConfig{…}, ::Val{…}) @ ForwardDiff /Users/david/.julia/packages/ForwardDiff/2nvFM/src/jacobian.jl:24
  ││┌ chunk_mode_jacobian(f::typeof(identity), x::Vector{…}, cfg::ForwardDiff.JacobianConfig{…}) @ ForwardDiff /Users/david/.julia/packages/ForwardDiff/2nvFM/src/jacobian.jl:172
  │││┌ _assert_tostring(msg::String) @ Base ./error.jl:247
  ││││ runtime dispatch detected: %1::Any(msg::String)::Any
  │││└────────────────────
  ││┌ chunk_mode_jacobian(f::typeof(identity), x::Vector{…}, cfg::ForwardDiff.JacobianConfig{…}) @ ForwardDiff /Users/david/.julia/packages/ForwardDiff/2nvFM/src/jacobian.jl:172
  │││ runtime dispatch detected: AssertionError(%213::Any)::AssertionError
  ││└────────────────────

ERROR: There was an error during testing

julia> versioninfo()
Julia Version 1.12.0-rc3
Commit 7522b240144 (2025-09-26 07:42 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 10 × Apple M2 Pro
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m2)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 6 virtual cores)

(jl_dz1DwL) pkg> st
Status `/private/var/folders/n6/98_7bm0j0hb57zv3l3tj8sxh0000gn/T/jl_dz1DwL/Project.toml`
  [f6369f11] ForwardDiff v1.2.1 `https://github.com/JuliaDiff/ForwardDiff.jl.git#master`
  [c3a54625] JET v0.10.8

[devmotion]

For comparison, on Julia 1.11.7 (note that JET 0.10 only supports Julia >= 1.12):

julia> using ForwardDiff, JET

julia> @test_opt ForwardDiff.gradient(only, [1.0], ForwardDiff.GradientConfig(only, [1.0], ForwardDiff.Chunk{1}()))
Test Passed

julia> @test_opt ForwardDiff.jacobian(identity, [1.0], ForwardDiff.JacobianConfig(identity, [1.0], ForwardDiff.Chunk{1}()))
Test Passed

julia> versioninfo()
Julia Version 1.11.7
Commit f2b3dbda30a (2025-09-08 12:10 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  CPU: 10 × Apple M2 Pro
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m2)
Threads: 1 default, 0 interactive, 1 GC (on 6 virtual cores)

(jl_KykCss) pkg> st
Status `/private/var/folders/n6/98_7bm0j0hb57zv3l3tj8sxh0000gn/T/jl_KykCss/Project.toml`
  [f6369f11] ForwardDiff v1.2.1 `https://github.com/JuliaDiff/ForwardDiff.jl.git#master`
⌅ [c3a54625] JET v0.9.20
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated`