Link-time error after changing a dependency

[gergoerdi]

On Stack 2.5.1, I am using the following resolver file:

name: clash-HEAD
resolver: lts-16.21

packages:
  - github: gergoerdi/clash-compiler
    commit: d83b5cc58816654c84537b0b049b32ce899e179e
    subdirs:
      - clash-ghc
      - clash-prelude
      - clash-lib

  - concurrent-supply-0.1.8
  - ghc-typelits-extra-0.4.1

flags:
    clash-prelude:
      multiple-hidden: false

After a full stack build in my project, I then want to change the version of Clash used, so I change the git commit hash to 62411778020b09433744d267acd5aa3deb85b04f. Subsequently, stack build fails at link time with an error message referencing a symbol that has been removed from Clash between the two Git commits:

17:47:46 [cactus@galaxy space-invaders-book2]$ stack build
Building all executables for `clashilator' once. After a successful build of all of them, only specified executables will be rebuilt.
clash-shake        > configure (lib)
clash-shake        > Configuring clash-shake-0.0.0...
clash-shake        > build (lib)
clashilator        > configure (lib + exe)
clash-shake        > Preprocessing library for clash-shake-0.0.0..
clash-shake        > Building library for clash-shake-0.0.0..
clash-shake        > [2 of 3] Compiling Clash.Shake.Xilinx
clashilator        > Configuring clashilator-0.1.0...
clashilator        > build (lib + exe)
retroclash-lib     > configure (lib)
clashilator        > Preprocessing library for clashilator-0.1.0..
clashilator        > Building library for clashilator-0.1.0..
clashilator        > [1 of 3] Compiling Clash.Clashilator
retroclash-lib     > Configuring retroclash-lib-0.0.0...
clashilator        > <command line>: /home/cactus/sdk/stack/snapshots/x86_64-linux-tinfo6/ec9124c0759d4a86157169cae10f6eb8fa7c593cc5806c7a1c85f1b2d4175440/8.8.4/lib/x86_64-linux-ghc-8.8.4/libHSclash-ghc-1.3.0-J8pqPdvV3DI3lqfzkVaQji-ghc8.8.4.so: undefined symbol: clashzmpreludezm1zi3zi0zmBPZZ7kiQNy99ISBL1Pghy2m_ClashziAnnotationsziPrimitive_zdfDataPrimitiveGuard5_info
retroclash-lib     > build (lib)         
retroclash-lib     > Preprocessing library for retroclash-lib-0.0.0..
retroclash-lib     > Building library for retroclash-lib-0.0.0..
clash-shake        > <command line>: /home/cactus/sdk/stack/snapshots/x86_64-linux-tinfo6/ec9124c0759d4a86157169cae10f6eb8fa7c593cc5806c7a1c85f1b2d4175440/8.8.4/lib/x86_64-linux-ghc-8.8.4/libHSclash-ghc-1.3.0-J8pqPdvV3DI3lqfzkVaQji-ghc8.8.4.so: undefined symbol: clashzmpreludezm1zi3zi0zmBPZZ7kiQNy99ISBL1Pghy2m_ClashziAnnotationsziPrimitive_zdfDataPrimitiveGuard5_info
retroclash-lib     > <command line>: /home/cactus/sdk/stack/snapshots/x86_64-linux-tinfo6/ec9124c0759d4a86157169cae10f6eb8fa7c593cc5806c7a1c85f1b2d4175440/8.8.4/lib/x86_64-linux-ghc-8.8.4/libHSclash-ghc-1.3.0-J8pqPdvV3DI3lqfzkVaQji-ghc8.8.4.so: undefined symbol: clashzmpreludezm1zi3zi0zmBPZZ7kiQNy99ISBL1Pghy2m_ClashziAnnotationsziPrimitive_zdfDataPrimitiveGuard5_info
Progress 3/6                             

--  While building package clash-shake-0.0.0 (scroll up to its section to see the error) using:
      /home/cactus/sdk/stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.4 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.0.1.0 build lib:clash-shake --ghc-options " -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1


--  While building package retroclash-lib-0.0.0 (scroll up to its section to see the error) using:
      /home/cactus/sdk/stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.4 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.0.1.0 build lib:retroclash-lib --ghc-options " -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1


--  While building package clashilator-0.1.0 (scroll up to its section to see the error) using:
      /home/cactus/sdk/stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.4 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.0.1.0 build lib:clashilator exe:clashilator --ghc-options " -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1

I tried removing .stack-work and */.stack-work, but the problem persists. It seems like I would need to nuke ~/.stack/, just like in the bad old Cabal days.

[qrilka]

@gergoerdi would you mind creating a simple project to reproduce this? In that case I could take a look into what's going on here.
Thanks for reporting this anyway.

[gergoerdi]

Sure, I will try to make a small self-contained way of reproducing it (hopefully by tomorrow). In the meantime, https://github.com/gergoerdi/clash-issue-1536/tree/memory-map-th should be the full code, including the stack.yaml and clash-HEAD.yaml files.

[qrilka]

Minimizing preproduction will help a lot, thank you.

[gergoerdi]

OK I have uploaded a fully self-contained, reproducing repo to https://github.com/gergoerdi/stack-issue-5473

1. stack build
2. cp clash-HEAD2.yaml clash-HEAD.yaml
3. stack build

[qrilka]

Yeah, great, reproduced it locally

[qrilka]

@gergoerdi it looks like you've found a hole in Pantry design :)
the problem is that clash-ghc directory has the same contents for both commits but its dependencies differ. As contents doesn't change (and the deps have the same versions) Stack reuses already built version which results in the failure you're getting.
Not sure yet what to do about this but there is a workaround which doesn't require nuking ~/.stack - use stack exec -- ghc-pkg unregister clash-ghc

[gergoerdi]

So wouldn't a straightforward fix here be to include, in the identifier of a dependency, a hash of (the hash of) all its dependencies?

[qrilka]

Nix-like way (when all inputs in theory determine the resulting output) makes sense but that's not quite how Stack works. In principle when checking already compiled libraries deps are also taken into account but it looks like in this case for some reason package identifiers (from ghc-pkg) for cabal-lib and cabal-prelude stay the same. So probably there's some lurking bug - I'll try to dig deeper into it.

[runeksvendsen]

@qrilka Thank you for looking into this. I'm hitting this bug as well, and I reported it as #5501 (before I found this issue).

I've experienced this a couple of times now, and I very often update git dependencies, so it happens fairly rarely. I was quite happy that I was able to reproduce it in a Docker container (see #5501). This reproduction may provide an extra data point in figuring out what's going on. But I'm happy to see that this issue has a more minimal reproduction.

For both reproductions, the bug is triggered when a git dependency commit hash is changed to a version where an export is added to an existing module. This is the diff between the dependency versions that causes the error for this issue's reproduction: gergoerdi/clash-compiler@d83b5cc...6241177, and this is the diff between the dependency version that causes the error for my reproduction: runeksvendsen/bellman-ford@a089bfe...45bfb65. In both cases the linker error refers to the added exports (modules Clash.Annotations.Primitive and Data.Graph.Digraph respectively).

[qrilka]

@runeksvendsen as I've written above it looks like Stack doesn't see the update and reusing old library clearly causes problems. Unfortunately I didn't have time yet to look more into it.

[qrilka]

After coming back to this ticket now I see that stack exec -- ghc-pkg unregister clash-ghc I advised above doesn't work - after unregistering Stack picks up incorrect version as precompiled. Will dig more into why that happens.

[qrilka]

@gergoerdi I had another look into the details of this and it appears that for some reason Cabal produces the same package-id for 2 different source trees (and having the same package version in cabal file) and it seems like it doesn't contradict the docs in GHC sources - https://gitlab.haskell.org/ghc/ghc/-/blob/master/compiler/GHC/Unit.hs#L42 but at the same time Cabal's definition is slightly different (UnitId is basically ComponentId if we ignore Backback for which Stack doesn't have support yet) - https://hackage.haskell.org/package/Cabal-3.4.0.0/docs/Distribution-Types-ComponentId.html#t:ComponentId
Stack by itself relies on package ids reported by Cabal so I'm not sure what could be a fix for this scenario.
@snoyberg do you know what definition of package id made Stack rely on uniqueness of package ids?

[snoyberg]

I have no idea. Package ID is one of those concepts that seems to have changed significantly over time, and I have no idea what we can or cannot rely upon in the Cabal or GHC worlds anymore.

[chreekat]

Looks like I just ran into this as well, ironically while building a repro for a different bug in a different library: https://github.com/chreekat/r2-repro/

I say "Uncomment to try the fix", but that doesn't work because I just get a bunch of linker errors. :P