Generator timing issue in Julia 1.5

[GearsAD]

I'm seeing the generator graphs take a long time, this should be a quick operation.

Timing is around 15 minutes for a small graph, which seems strange.

MRE and Timing

using DistributedFactorGraphs
using IncrementalInference, RoME

ENV["JULIA_DEBUG"] = "IncrementalInference"
##

@info "Setting up graph..."
# create temp in memory graph that we will tee up to server via cfg
@time hex = generateCanonicalFG_Hexagonal()

1002.582511 seconds (1.01 G allocations: 58.696 GiB, 1.22% gc time)

Environment

I'm using Julia 1.5, my packages:

Status `~/.julia/environments/v1.5/Project.toml`
  [f0fec3d5] AprilTags v0.9.0
  [a81c6b42] Compose v0.9.2
  [070d9d8b] Diana v0.2.0 `https://github.com/neomatrixcode/Diana.jl.git#master`
  [b5cc3c7e] DistributedFactorGraphs v0.16.0
  [ffbed154] DocStringExtensions v0.8.5
  [c91e804a] Gadfly v1.3.4
  [cd3eb016] HTTP v0.8.19
  [904591bb] IncrementalInference v0.25.1 `~/.julia/dev/IncrementalInference`
  [682c06a0] JSON v0.21.2
  [9b8beb19] JSONWebTokens v1.1.0
  [f3e6a059] NavAbilitySDK v0.1.0 `~/Ganbatte/NVASDK.jl`
  [d2adbeaf] Neo4j v2.0.0
  [91fb55c2] RoME v0.16.0
  [cbff2730] Unmarshal v0.4.3

[Affie]

If it's only in julia v1.5 we should perhaps drop it as soon as v1.7 comes out. Julia v1.6 will be the next LTS

[dehann]

thats weird, i tried it in Julia 1.6.3 without the debug printout:

@time fg = generateCanonicalFG_Hexagonal()
1.395839 seconds (13.61 M allocations: 1.009 GiB, 10.54% gc time)

Will try it again with debug printout, but dont think itll slow down as much as you seeing. I also going to guess it's likely some type stability issue in Julia 1.5 vs 1.6

[dehann]

aside, can do 2-for-1 by fixing #490 at the same time

[GearsAD]

Looks consistent, at least on my machine:

886.191347 seconds (1.01 G allocations: 58.696 GiB, 1.40% gc time)

I'll try Julia 1.6

[GearsAD]

Yes, at least just with my 2 sample points it does look like a 1.5 vs 1.6 issue (some type instability). With a fresh 1.6 environment it's one/two orders of magnitude faster (probably faster in general because it was a first run)

 32.761157 seconds (90.25 M allocations: 5.628 GiB, 3.92% gc time, 95.16% compilation time)