Template Haskell fails when signatures stack up

[dwincort]

Hi,

There is definitely some weird interplay between Template Haskell and backpack that I don't totally understand. Even when the two seem completely unrelated, I'm getting errors. Consider the following example:

Example

Let's assume this is the cabal file:

name: th-backpack-failure
version: 1.0
build-type: Simple
cabal-version: >= 2.0

library
    build-depends: base, impl, sig-with-th
    hs-source-dirs: th-backpack-failure
    mixins: sig-with-th requires (Sig as Impl)
    signatures: Unused
    exposed-modules: Go

library sig-with-th
    build-depends: base
    hs-source-dirs: sig-with-th
    signatures: Sig
    exposed-modules: THFuns

library impl
    build-depends: base
    hs-source-dirs: impl
    exposed-modules: Impl

The individual files are quite simple.
impl/Impl.hs:

module Impl where

sig-with-th/Sig.hsig

signature Sig where

sig-with-th/THFuns.hs

module THFuns where
thfun _ = return []

th-backpack-failure/Go.hs

{-# LANGUAGE TemplateHaskell #-}
module Go where
import THFuns

thfun ''Int

th-backpack-failure/Unused.hsig

signature Unused where

Tests

Building the above fails with an error message that (I think) indicates that Template Haskell and Backpack don't play nicely together. But, if you remove the line signatures: Unused from the cabal file, it builds just fine. Alternatively, you can remove the lines mixins: sig-with-th requires (Sig as Impl) and signatures: Sig and it builds just fine too.

Another work-around is to add a new library stanza to the cabal file, like so:

library intermediary
    build-depends: base, sig-with-th, impl
    reexported-modules: THFuns
    mixins: sig-with-th requires (Sig as Impl)

Then, change the main library stanza by removing the mixins line and depending on intermediary instead of sig-with-th and impl. This builds just fine too.

Thoughts

It seems like signatures are being piled together in a way that doesn't support template haskell. In other words, I really wanted it to be the case that when the main library instantiated sig-with-th, that import would be treated as if there were no backpack components at all (after all, all the backpack holes have been filled in). Then, any Template Haskell imports from that import are totally fine, even if I add a signature to the main library.

The intermediary option is a definite work-around, but it would be better to not have to use such tedious boilerplate.

[ezyang]

Yeah, this should work. What was the error message you got?

[dwincort]

Preprocessing library for th-backpack-failure-1.0..
Building library instantiated with Unused = <Unused>
for th-backpack-failure-1.0..
[1 of 2] Compiling Go               ( th-backpack-failure/Go.hs, /var/folders/mk/7vfmd_rj6hj448vqccjtxrcwk8zdtj/T/ghc23250_0/ghc_1.o )
ghc: ^^ Could not load '_thzmbackpackzmfailurezm1zi0zminplacezmsigzmwithzmthzm8D6FyzzfGpcE9fVvBiKCfEy_THFuns_thfun_closure', dependency unresolved. See top entry above.


ByteCodeLink.lookupCE
During interactive linking, GHCi couldn't find the following symbol:
  thzmbackpackzmfailurezm1zi0zminplacezmsigzmwithzmthzm8D6FyzzfGpcE9fVvBiKCfEy_THFuns_thfun_closure
This may be due to you not asking GHCi to load extra object files,
archives or DLLs needed by your current session.  Restart GHCi, specifying
the missing library using the -L/path/to/object/dir and -lmissinglibname
flags, or simply by naming the relevant files on the GHCi command line.
Alternatively, this link failure might indicate a bug in GHCi.
If you suspect the latter, please send a bug report to:
  glasgow-haskell-bugs@haskell.org

[ezyang]

OK, I have a guess as to what's going on (we're not correctly populating the dependent libraries list when we typecheck indefinite packages). I won't be able to look at this in detail until the weekend. This is probably a GHC bug, rather than a Cabal bug.

[ezyang]

Nevermind, it's a Cabal bug.

Source component graph:
    component lib:sig-with-th
    component lib:impl
    component lib
        dependency lib:impl
        dependency lib:sig-with-th
Configured component graph:
    component bkpcabal5634-0.1.0.0-4XmXZPVW0Kd19Mr0GgJNJd-sig-with-th
        include base-4.12.0.0
    component bkpcabal5634-0.1.0.0-LvAMsiEP2WdAOf3SBSykQ1-impl
        include base-4.12.0.0
    component bkpcabal5634-0.1.0.0-1cmoWSZWkMZJ6bVOoH2uZL
        include bkpcabal5634-0.1.0.0-4XmXZPVW0Kd19Mr0GgJNJd-sig-with-th requires (Sig as Impl)
        include base-4.12.0.0
        include bkpcabal5634-0.1.0.0-LvAMsiEP2WdAOf3SBSykQ1-impl
Linked component graph:
    unit bkpcabal5634-0.1.0.0-4XmXZPVW0Kd19Mr0GgJNJd-sig-with-th[Sig=<Sig>]
        include base-4.12.0.0
        THFuns=bkpcabal5634-0.1.0.0-4XmXZPVW0Kd19Mr0GgJNJd-sig-with-th[Sig=<Sig>]:THFuns
    unit bkpcabal5634-0.1.0.0-LvAMsiEP2WdAOf3SBSykQ1-impl
        include base-4.12.0.0
        Impl=bkpcabal5634-0.1.0.0-LvAMsiEP2WdAOf3SBSykQ1-impl:Impl
    unit bkpcabal5634-0.1.0.0-1cmoWSZWkMZJ6bVOoH2uZL[Unused=<Unused>]
        include bkpcabal5634-0.1.0.0-4XmXZPVW0Kd19Mr0GgJNJd-sig-with-th[Sig=bkpcabal5634-0.1.0.0-LvAMsiEP2WdAOf3SBSykQ1-impl:Impl]
        include base-4.12.0.0
        include bkpcabal5634-0.1.0.0-LvAMsiEP2WdAOf3SBSykQ1-impl
        Go=bkpcabal5634-0.1.0.0-1cmoWSZWkMZJ6bVOoH2uZL[Unused=<Unused>]:Go
Ready component graph:
    definite bkpcabal5634-0.1.0.0-LvAMsiEP2WdAOf3SBSykQ1-impl
        depends base-4.12.0.0
    indefinite bkpcabal5634-0.1.0.0-4XmXZPVW0Kd19Mr0GgJNJd-sig-with-th
        depends base-4.12.0.0
    indefinite bkpcabal5634-0.1.0.0-1cmoWSZWkMZJ6bVOoH2uZL
        depends bkpcabal5634-0.1.0.0-4XmXZPVW0Kd19Mr0GgJNJd-sig-with-th
        depends base-4.12.0.0
        depends bkpcabal5634-0.1.0.0-LvAMsiEP2WdAOf3SBSykQ1-impl

For some reason, it doesn't decide to actually build the instantiated component.

[LightAndLight]

I've recently encountered an issue along these lines (maybe even the same problem). Here's my test repo https://github.com/LightAndLight/backpack-test.

I don't even have to use any template haskell features, the presence of the pragma is enough to cause the issue.

I've used this knowledge to get around the issue in my actual code. My current heuristic is "don't use template haskell in modules that (either directly or transitively) depend on a signature".

[LightAndLight]

This is actually really inconvenient as backpack is a promising method of abstraction for my current project.

It might be more productive for me to help fix this than find ways to get around it.

@ezyang If you have the time, would you be able to point me in the right direction for debugging these errors, and potentially upstreaming a fix?

[ezyang]

I have reviewed the code for linked component to ready component, and I think the problem is that we are not instantiating fully definite includes inside indefinite packages. Should be easy enough to fix.

[ezyang]

I threw up a patch but I have not tested it at all. Will test over the weekend.

[ezyang]

@LightAndLight I think your problem is a different one, as it's not fixed by the posted PR.

[LightAndLight]

@ezyang No worries