Add Enzyme tests

[devmotion]

I had tried to run Enzyme tests, and it seems currently quite a few tests fail/error.

[yebai]

The fact that it doesn't segfault in so many tests is a good sign.

Related: TuringLang/Turing.jl#1887

[wsmoses]

@devmotion can you rerun on latest Enzyme main?

[wsmoses]

fyi the test failures appear to be due to currently unimplemented derivatives of lapack functions. probably should disable those tests.

[wsmoses]

@devmotion @yebai at this point, everything in this PR works except:

• differentiating through Lapack functions is not yet supported by Enzyme
• your test code uses an active return for a function which returns a vector (this must be duplicated).

Moreover all the errors are relatively nice.

at this point there aren't Enzyme bugs here and merging this is now on your end (presumably marking the code using lapack as unsupported atm, and fixing your test handler to not use an active vector for that input)

[devmotion]

Thank you, I'll try to have a look at and fix the remaining issues this week.

[devmotion]

The latest commit added a way to mark some Enzyme tests as broken (and I also introduced a bug in the test setup 😄) but it also reveals that there are many more issues - so far only a tiny initial testset was run but no tests of the log densities of the uni-, multi- and matrixvariate distributions we're actually interested in.

[devmotion]

@wsmoses Is this an enzyme bug or did I call autodiff incorrectly? https://github.com/TuringLang/DistributionsAD.jl/actions/runs/5872312798/job/15923610267?pr=254#step:6:303

[wsmoses]

Definitely seems like a bug (that we need a minimal reproducer for)

…

On Wed, Aug 16, 2023 at 4:26 PM David Widmann ***@***.***> wrote: @wsmoses <https://github.com/wsmoses> Is this an enzyme bug or did I call autodiff incorrectly? https://github.com/TuringLang/DistributionsAD.jl/actions/runs/5872312798/job/15923610267?pr=254#step:6:303 — Reply to this email directly, view it on GitHub <#254 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJTUXBUFJ5TP4AJUKMV3ADXVRY3HANCNFSM6AAAAAA2LPVOW4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[yebai]

One slightly annoying thing I found about the Enzyme error message is printing a large amount of LLVM code when compilation fails (see here for an example). This is useful when developing Enzyme (at least in the early stages) but not very helpful for users of Enzyme. Very few people can read such LLVM code and figure out why Enzyme couldn't compile. They also add clutter to CI logs. I suggest disabling such LLVM messages by default but providing an API for developers to activate them when needed.

[wsmoses]

@devmotion did you have a chance to isolate that 1.6 error that crashes Julia (so we can ensure it does not)?

[wsmoses]

@devmotion can you repush (its not letting me rerun tests for some reason)

[wsmoses]

@devmotion hm I still can't retrigger this, if you can have the tests run again.

[yebai]

@wsmoses you should have the priveledges to trigger CI (re-)runs -- can you try again?

[mhauru]

I reran CI against the latest Enzyme.jl v0.12.19. They are now failing with either (mac os)

Assertion failed: (vec.back() == currentBlock), function addReverseBlock, file /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp, line 8952.

or (ubuntu)

 julia: /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:8952: llvm::BasicBlock* GradientUtils::addReverseBlock(llvm::BasicBlock*, const llvm::Twine&, bool, bool): Assertion `vec.back() == currentBlock' failed.

See the failed CI runs for details.

@wsmoses, is this an issue you are aware of?

[wsmoses]

Nope not aware of please open MWE

[mhauru]

Gathering links to some related Enzyme issues:
EnzymeAD/Enzyme.jl#1585
EnzymeAD/Enzyme.jl#1593
EnzymeAD/Enzyme.jl#1594
EnzymeAD/Enzyme.jl#1595
EnzymeAD/Enzyme.jl#1596

[wsmoses]

@yebai Looks like the issues here are that Enzyme actually passes tests it was marked as failing?

[yebai]

yes, most tests are passing with most recent Enzyme.

[yebai]

@devmotion do you know why we are testing Enzyme only on Others tests?

[yebai]

Suggested change

	# Disable Enzyme warnings
	Enzyme.API.typeWarning!(false)
	# Enable runtime activity (workaround)
	Enzyme.API.runtimeActivity!(true)

[wsmoses]

Typewarning shouldnt need to be set [it is presently the default]

[yebai]

this commit is for removing them

[wsmoses]

Ah, you may still need the runtimeactivity one for now [and looks like tests concur]

[yebai]

I think Enzyme might still struggle with a few MvNomal tests, which was revealed after 8b9fa77

[wsmoses]

open issues and we'll get those sorted!

[wsmoses]

Oh yeah runtime activity isn't supported yet for forward mode blas, so my recommendation here is to turn on runtime activity more selectively for which tests need it

[mhauru]

Some open issues that are causing test failures here. Some might just need marking as broken, others seem more in need of fixing:

• Missing Rmath derivatives EnzymeAD/Enzyme.jl#1620 (marked as broken)
• Implement besseli EnzymeAD/Enzyme.jl#1603 (marked as broken)
• xlogy grad wrong at 0 EnzymeAD/Enzyme.jl#1601
• Cannot deduce type for QuadGK EnzymeAD/Enzyme.jl#1599
• Cholesky solve segfault EnzymeAD/Enzyme.jl#1649
• unwrapMode != UnwrapMode::LegalFullUnwrap with Julia <1.8 EnzymeAD/Enzyme.jl#1629
• Indeterministic illegal instruction: 4 EnzymeAD/Enzyme.jl#1626
• Turing model crashes without runtimeActivity EnzymeAD/Enzyme.jl#1623
• Illegal type analysis - BigFloat EnzymeAD/Enzyme.jl#1621

See also:

• Simpler error messages for missing gradient rules EnzymeAD/Enzyme.jl#1625
• Raise Julia errors rather than crashing on Assertion failures EnzymeAD/Enzyme.jl#1634

I was in the process of checking if I could find more issues in this test suite, but got distracted by TuringLang/Turing.jl#2240 and related issues. Will get back to this once I'm done minimising what I've found there.