Insane precompile times in Metalhead models

[theabhirath]

Xref FluxML/Metalhead.jl#105 (comment). When running gradient tests on some Metalhead models, the times taken for the first gradient are off the charts. Specifically, the function gradtest was run on the models in Metalhead#master:

function gradtest(model, input)
  y, pb = Zygote.pullback(() -> model(input), Flux.params(model))
  gs = pb(ones(Float32, size(y)))

  # if we make it to here with no error, success!
  return true
end

and the times taken for the tests were:

The ViT model can be ignored because gradtest wasn't run on it, but it seems to be taking quite a large amount of time for precompilation of the other models...

cc @CarloLucibello

[DhairyaLGandhi]

Definitely a regression from v0.4, and v0.5.

Honestly surprised no one has complained about it yet. I've been running into this often too. Thanks for the summary!

+1 for enabling the gradtests, we really need those in our testing suite.

[CarloLucibello]

Let's close this since it is a duplicate of #1126, sorry I didn't notice earlier