Explicit Splice Imports

[mpickering]

This proposal suggest a refinement of the import syntax to distinguish between imports which can and can't be used in top-level splices.

The gist is that an import of the form

import splice A

can only be used in top-level splices, and other imports can only be used outside of top-level splices.

Rendered

[Ericson2314]

I talked to @mpickering and I support this.

I think this proposal is insufficient to ensure the correct native vs foreign versions of packages is always used for cross compilation, but also that that's fine. This indeed unblocks better parallelism in the presence of TH, and lays a foundation both in terms of the implementation and GHC user mind-share for something like my #243 to do the rest for cross.

That proposal, with its GHC and Cabal portions, had proved to much for people to digest all at once, and likewise to much for me as author assure every morsel was completely baked, so I think approving a design incrementally like this is not only good, but necessary.

[goldfirere]

I'm in strong support of this directional of travel, but have a number of nitpicks/clarifying questions:

• Please define "top-level", as it has two possible interpretations: either splice-that-is-not-within-a-bracket or unnested-declaration-splice. I figured out that you mean the former, but I did not know this when starting my read of the proposal.
• Please define levels; the proposal refers to negative level but does not define this.
• Your note about Prelude: presumably this is not the case with -XNoImplicitPrelude, but it would be good to say.
• Presumably, an implicit Prelude hiding ( head ) also eliminates head from being in scope in splices? Would be good to say something like "If Prelude is imported implicitly, then import splice Prelude is also in effect, implicitly. Otherwise, any import of Prelude is duplicated, but with import splice."
• Could you include a short BNF of import statements? Doing so would, for example, say whether splice comes before or after qualified.
• Can splice go after the module name? This proposal would probably want it to.
• What happens with ambiguity? For example:

import A ( x )
import splice B ( x )

foo = $( x ) x

Is that allowed? It's unambiguous, but perhaps confusing. I think I would prefer names from both bare imports and import splices to be in scope, but there are usage restrictions on the names. That would mean that my example would be rejected as ambiguous. Others may disagree here.

• Relatedly, how does this interact with -XRebindableSyntax? If I have a splice-imported fromInteger and a regular-imported fromInteger, does the meaning of 0 in a splice differ from that outside the splice?
• How does this feature interact with re-exports? Presumably, you can't re-export a splice-imported thing.
• Please define (briefly) what a home module is. I think it's a module in the same package as the one being compiled, but I'm actually not sure.

Thanks -- this is such a nice simplification over #243, and I think it will be much easier to implement and understand.

[Ericson2314]

Otherwise, any import of Prelude is duplicated, but with import splice."

I would prefer not? It was my understanding that import Prelude wasn't special-cased except for it replacing the default implicit prelude. I view making one have to do the two separate imports as keeping the special-case as minimal as possible.

It's unambiguous, but perhaps confusing. I think I would prefer names from both bare imports and import splices to be in scope, but there are usage restrictions on the names.

So once we get to supporting cross, e.g. Int is already not the same Int anyways. In general, the stages become completely separate modules that just "happen to share the same module and have dependencies". I think this strong disambiguation rules might just confuse people as to how the phase separation works, rather than prevent confusion.

How does this feature interact with re-exports? Presumably, you can't re-export a splice-imported thing.

Good catch. I agree.

Please define (briefly) what a home module is. I think it's a module in the same package as the one being compiled, but I'm actually not sure.

For the record, it is.

---

Please define (briefly) what a home module is. I think it's a module in the same package as the one being compiled, but I'm actually not sure.

I'm fine landing this as a first step, but I highly doubt we're going to be able to avoid the extra stuff from #243 and I think it's important that we agree on the requirements long term, even if we just deciding on this for now.

For cross, this proposal solves the following:

module C where
import splice B (bar)

x = $(bar) -- foo is from the native platform's A

However, it doesn't help us write A

module B
import {- ?? -} A (foo)
bar = [| foo |] -- what platform?

If we follow the evaluation:

x = $(bar) = $([| foo |]) = foo

we see that foo has to come from the foreign platform. But there's no type of import for A in B that accomplishes that.

That means we at least need import, import quote, and import splice. This corresponds to the for-syntax and for-template shorthands in Racket (https://docs.racket-lang.org/reference/require.html?q=require).

Finally, if one wants to support $(... $(...) ...) and [| ... [| ... |] ... |], one ends up freely generating stages for each integer. That's the general case in Racket and #243.

[mpickering]

Thanks for that example @Ericson2314, I now understand what you were staying about cross-compilation on IRC.

Are there not deeper problems here though? If you import splice A, then that says we need module A to be build for the host. Then looking at the interface file for A you find that there is a definition you can use A.foo :: Q Exp, at that point there's no information that foo contains identifiers from the previous module which was already compiled for a specific platform.

In fact, I think the whole idea cross-stage persistence for top-level functions is built under the assumption that host = target. I think it's an open research design question about how it should work for heterogeneous cases. For example, you wouldn't expect CSP to work when generating a C program from a Haskell program.

[mpickering]

I'm in strong support of this directional of travel, but have a number of nitpicks/clarifying questions:

* Please define "top-level", as it has two possible interpretations: either splice-that-is-not-within-a-bracket or unnested-declaration-splice. I figured out that you mean the former, but I did not know this when starting my read of the proposal.

Done, I mean both.

* Please define levels; the proposal refers to negative level but does not define this.

Done

* Your note about `Prelude`: presumably this is not the case with `-XNoImplicitPrelude`, but it would be good to say.

Clarified.

* Presumably, an `implicit Prelude hiding ( head )` also eliminates `head` from being in scope in splices? Would be good to say something like "If `Prelude` is imported implicitly, then `import splice Prelude` is also in effect, implicitly. Otherwise, any `import` of `Prelude` is duplicated, but with `import splice`."

* Could you include a short BNF of `import` statements? Doing so would, for example, say whether `splice` comes before or after `qualified`.

I choose (at random) before.

* Can `splice` go _after_ the module name? [This proposal](https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0190-module-qualified-syntax.rst) would probably want it to.

I have no opinion about this. There are two things which make this awkward. 1. The extension is called ImportQualifiedPost, nothing to do with splice imports, would another extension have to be introduced? 2. The proposal already did nothing with package imports or source imports.

* What happens with ambiguity? For example:

import A ( x )
import splice B ( x )

foo = $( x ) x

Is that allowed? It's unambiguous, but perhaps confusing. I think I would prefer names from both bare imports and import splices to be in scope, but there are usage restrictions on the names. That would mean that my example would be rejected as ambiguous. Others may disagree here.

At least a warning would be expected I think.

* Relatedly, how does this interact with `-XRebindableSyntax`? If I have a splice-imported `fromInteger` and a regular-imported `fromInteger`, does the meaning of `0` in a splice differ from that outside the splice?

This is a good question. I think that is a similar question to your one about ambiguity. You definitely need to splice import a module if it's instance is used in a top-level splice, but whether you allow different instances to be in scope in splices as the rest of the module is a choice of design. It seems like it would be less confusing to insist on the same instances in scope in both places but there's no reason from an implementation perspective.

* How does this feature interact with re-exports? Presumably, you can't re-export a `splice`-imported thing.

That is disallowed.

* Please define (briefly) what a home module is. I _think_ it's a module in the same package as the one being compiled, but I'm actually not sure.

Done

Thanks -- this is such a nice simplification over #243, and I think it will be much easier to implement and understand.

[Ericson2314]

Glad it's clear now, @mpickering.

In fact, I think the whole idea cross-stage persistence for top-level functions is built under the assumption that host = target.

Yes I agree, but conversely.

I think it's an open research design question about how it should work for heterogeneous cases.

I think it's just sufficient to not have cross stage persistence. The Racket way should work just fine, and the way build time dependencies work in Nixpkgs is analogous to Racket. I would say the research is done.

[goldfirere]

I'm confused by "or" here. It seems that the first set includes the second, making the "or" part redundant. Or am I missing something?

[mpickering]

I don't know how to describe a declaration splice which is not explicitly spliced. For example

module M where

import Lens

data A ...

deriveLens 'A

The call to deriveLens 'A should be understood to be at a negative level (hence deriveLens should be splice imported) but it isn't surrounded by splices.

[goldfirere]

I don't see any text in the updated proposal clarifying the ambiguity questions around e.g. foo = $( x ) x.

Good point about instances -- I hadn't thought about those.

Let's examine this:

module X where
data X = MkX

module Normal where
import X
instance Show X where show _ = "normal"

module Splice where
import X
instance Show X where show _ = "splice"

module Bottom where
import X (X(..))
import splice X (X(..))
import Normal ()
import splice Splice ()
import splice Language.Haskell.TH.Lib ( stringE )

s1 = show MkX
s2 = $( stringE (show MkX) )

Should this be accepted? If we allow different sets of instances to be in scope in different stages, then "yes", with s1 being "normal" and s2 being "splice". But if we require the same set of instances to be in scope in both stages, then this is rejected with two possible Show X instances.

[Ericson2314]

FWIW in the cross case, @goldfirere's example is fine, even necessary. E.g.

module X where
  data X = MkX

module Instance where
  import X

  #if aarch64_HOST_OS
  instance Show X where show _ = "are you reading this on a phone?"
  #else
  instance Show X where show _ = "are you reading this on a desktop?"
  #endif

module Bottom where
  import X (X(..))
  import splice X (X(..))
  import Instance ()
  import splice Instance ()
  import splice Language.Haskell.TH.Lib ( stringE )

  s1 = show MkX
  s2 = $( stringE (show MkX) )

It's only when we wish to "observe" build == host and "unlock" features like cross stage persistence that we have new coherence obligations across stages.

[mpickering]

I think the conservative position here is to disallow all these examples of ambiguity we have been discussing. If in the future like @Ericson2314 suggests we want to relax the requirements, then it won't break any user code to make the check less stringent.

The main uses of levels in this proposal seem to only go negative, what is the use for positive levels?

Positive levels ensure that code generation doesn't depend on information which is not yet available. For example, positive levels
disallow:

foo = [| \x -> $(x) |]

[mpickering]

I updated the proposal now with comments about ambiguity.

[Ericson2314]

@mpickering That's fair, but if we are trying to be conservative we should probably straight up ban cross stage persistence too?

[mpickering]

@Ericson2314 Cross-stage persistence is to do with promoting identifiers upwards in levels, this proposal is only about splices, not about quotes.

[Ericson2314]

@mpickering well what about local definitions using splice imports that are then spliced? That is "morally OK" provided those definitions are used at runtime or exported, but doesn't that work today as cross-stage persistence?

(Maybe this comes with GHCi more than real code, I forget the ins and outs of today's staging restriction)

[mpickering]

An example would be useful? Do you mean..

foo x = $$([|| x ||])

There's no CSP there, x is used at the level it is bound.

[Ericson2314]

Nevermind, I think it's just ghci, which let me do

> id' a = a
> $(id' [| 1 |])

[simonpj]

Good stuff. Some thoughts

• Could you number the sections for easy reference, please? (There is a standard way to automate such numbering.)

• If I splice-import A then A must be compiled to executable code before importing it. But so must all modules that A depends on! This may seem obvious, but I really think it's worth stating explicitly.

• Under "Proposed change" I see "The splice modifier indicates to the compiler that identifiers imported from the module can only be used inside splices (1)." I think you have (1) and (2) back to front.

• Sometimes you say "can only be used inside splices" and sometimes "can only be used in top-level splices". I think you mean the latter. Worth being consistent.

• I wonder if you couldn't specify all this more neatly by saying that

  • Ordinary imports bind variables at level 0
  • Splice imports bind variables at level -1
  • The body of a module is processed starting at level 0
    Then the inability to refer to an ordinary import in a top-level splice becomes simply an ordinary staging error. And it makes sense: ordinary-imported modules can't be executed, and so are available "later" than splice-imported ones.

• That way of looking at the design would strongly suggest that splice-imported variables be available at level 0, i.e. outsides splices. This contradicts the proposal as written, but seems much nicer to me, because it rests on existing logic. I strongly urge this change, unless I have missed something vital.

• Under "Ambiguity of...", I think it would be easier to say

  • Resolution of scopes (often called "renaming") is blind to whether or not an identifier was imported with splice.
    And then give examples.

• "Splice imports can't be re-exported, unless they are also imported normally. Why can't splice-imports be re-exported? I think you might be worried about

  module M( f ) where { import splice S( f ) }
  module Y where { import M(f); $(f) }
  

  Here Y ordinary-imports f, but then uses it in a top-level splice. But that's illegal (stage error) so Y would be rejected. What goes wrong if you delete this prohibition?

[mpickering]

@simonpj Thanks for your comments. I have updated the proposal apart from your "strongly suggested change". Distinguishing between things which are only needed at build-time and things which are needed at runtime can further be exploited if we introduce the idea of build dependencies, which will not ultimately be linked into the final executable. I believe this is an important distinction to make and would prefer to keep the design as-is.

"Splice imports can't be re-exported, unless they are also imported normally. Why can't splice-imports be re-exported? I think you might be worried about

If you export a splice import in a situation like you describe, it turns from a build-time dependency into a runtime dependency. Splice imported things definitely won't exist at runtime (only the stuff they end up generating). Allowing a splice export would be similar to allowing:

module M (foo) where
import splice A

-- Stage error here
foo = $([| A.foo |])

[simonpj]

Distinguishing between things which are only needed at build-time and things which are needed at runtime can further be exploited if we introduce the idea of build dependencies, which will not ultimately be linked into the final executable. I believe this is an important distinction to make and would prefer to keep the design as-is.

If this is an important distinction, perhaps you can list it under "Motivation"? I had no idea that this was one of your goals, and I don't think the proposal mentions it at all.

When saying "Splice imports can't be re-exported, unless they are also imported normally", link back to that particular motivation, since the sole reason for that rule is that motivation.

I'm sad to lose the idea that everything in the proposal can be explained by the existing stating mechanism. It seems so simple and elegant! Perhaps you could list this possibility under "alternatives" and explain why you chose not to adopt it. (I suppose it is possible that the committee might take a different view.)

[mpickering]

I'm sad to lose the idea that everything in the proposal can be explained by the existing stating mechanism. It seems so simple and elegant! Perhaps you could list this possibility under "alternatives" and explain why you chose not to adopt it. (I suppose it is possible that the committee might take a different view.)

In fact, things can be explained in terms of levels still:

Level Specification of ``splice``                                               
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                
                                                                                 
 * Ordinary imports introduce variables at all non-negative levels (>= 0)        
 * Splice imports introduce variables at all negative levels. (< 0) 

[Ericson2314]

@simonpj The "cumulative stages" you proposed run afowl of cross compilation too: the build-time version of something may be quite different than it's run-time counterpart.

[goldfirere]

I'm in favor of the proposal's intent as written, and against @simonpj's strongly urged change.

The key observation is that cross-stage persistence has costs. One particularly big cost is if the stages are actually executed on different architectures, where cross-stage persistence requires two separate binaries. But the idea of avoiding linking some dependencies in the final executable is a nice example of a benefit of avoiding cross-stage persistence. Note that, even between levels 0 and 1, there is a cost to persistence in the form of Lift instances.

[simonpj]

The key observation is that cross-stage persistence has costs. One particularly big cost is if the stages are actually executed on different architectures, where cross-stage persistence requires two separate binaries. But the idea of avoiding linking some dependencies in the final executable is a nice example of a benefit of avoiding cross-stage persistence. Note that, even between levels 0 and 1, there is a cost to persistence in the form of Lift instances.

I am open to being persuaded. But the proposal does not present these arguments. Could it, please?

[simonpj]

I'm generally in support.

• "The splice modifier indicates to the compiler that module B is only used at compile time and hence the imports can only be used inside top-level splices (1). When the extension is enabled, imports without the splice modifier are only available at runtime and therefore not available to be used in top-level splices (2)" . I found this hard to grok. Try this:

  • Entities imported via import splice M can be used only in top-level splices (i.e. at negative levels)
  • Entities imported via import M can be used only outside top-level splices (i.e. as levels >= 0)

• I'm worried about type-class instances. If you use an instance, you need to link to the module from whence it came. So to deliver on your goals:

  • Instances imported (perhaps transitively) by import splice M can be used only in top-level splices
  • Instances imported (perhaps transitively) by import M can be used only outside top-level splices.

  Do you agree? This sounds as if it might be pretty hard to implement. At the moment we do not record how an instance is imported. For the rest of the proposal, implementation seems fairly easy. But I'm a bit concerned that instances may not be.

  We could wonder about type-family instances too. And (soon, maybe) defaults...

I can see the motivation, but I have lost a clear picture of how much implementation complexity this will impose.

[Ericson2314]

@simonpj I am hoping we can make each level it's own virtual module, as far as the implementation is concered. So as opposed to trying hard to keep things separate, we are separate by default.

The challenge then because the interactions between names, e.g. quotes in stage n - 1 need to be name-resolved in stage n. I think we can pull that off, however.

[mpickering]

I started implementing the proposal and it's not clear to me now how splice imports should work.

The problem is that if I splice import an identifier then, the result of running the splice can also contain identifiers which come from splice imported modules. Therefore we would need to link against the module where the splice import came from.

For example

module A where

data X = X

x = [| X |]

then

module B where

import splice A

-- y = X, so need to link against A, which defines X.
y = $(x)

Two ways to solve this:

1. Allow splice imported identifiers to be used anywhere
2. Also check the generated code for whether all the identifiers are non-splice imported.

I think I prefer option 1.

[Ericson2314]

This is why we need quote imports. If you can only use splice imports and splices and quote imports and in quotes, everything cancels out nicely, the phase separation is proper, and the driver has all the information it needs.

[Ericson2314]

A splice import means we must compile the import all the way to the end just to compile the current module half way (Tc). A quote import means we need only compile the import half way even if we are compiling the current module all the way to the end. But it crates a transitive dependency so quote import + splice import = (transitive) regular import, and then quote import gets forced the rest of the way.

[simonpj]

Allow splice imported identifiers to be used anywhere

That does seem inevitable, but is quite a disappointing conclusion, given the goal of this proposal to cleanly separate what is needed at compile time from runtime.

Suppose we have

module CompileTime( foo ) where
   import RunTime( f ) -- Call this a quote-import
   import OtherStuff
   foo n = .....lots of code... [| f |] ....

If a module M splice-imports CompileTime, then it needs compile-time versions of CompileTime and OtherStuff; but not RunTime. On the other hand, at run-time we will need M and RunTime.

Let's call RunTime a "quote import". It imports things used only in quotes.

So a splice-import of CompileTime tells us that we have a run-time dependency on the quote-imports of CompileTime, rather than (as previously thought) no run-time dependencies.

This makes my head spin a bit, but maybe there is something here.

[aspiwack]

I don't really have an opinion on this matter myself. But I find myself wondering: what does Racket do about this? I imagine that the answer to this question is likely to inform the solution to @mpickering 's conundrum.

[Ericson2314]

@aspiwack the racket solution is the "quote import", see https://docs.racket-lang.org/reference/require.html?q=require and the for-template and for-syntax shorthands in particular.

[Ericson2314]

The phase-offsetting when working with transitive dependencies, behind being how Racket works, is also what we did with Nixpkgs. As far is I know, it's the one and only way to preserve the proper separation of phases across module boundaries.

[Ericson2314]

This makes my head spin a bit, but maybe there is something here.

No doubt further refining "compile-time" and "run-time" to specific compiler phases as outlined in #412 (comment) will make the head-spinning worse. But I do think it's a really good way to have the implementation and design police each other.

[goldfirere]

I don't see it. That is, I don't see why import splice M must imply import M for this all to work. Some splice-imports will contain splices that expand out to include other definitions from M, but not all will. If a particular splice requires a particular definition, then that definition must be available in a regular import. I say "available" -- not necessarily in scope. Here is an example:

module A where

data X = MkX

module B where
import A

q = [| MkX |]

module C where
import splice B
import A ()   -- this is the key line

mkx = $q

The "key line" above says that A must be available at runtime in C. It's almost as if defining

module Z where
z = 5

in some module does two things: it creates an entity with original name Z.z. Entities have only original names, actually. The definition also creates a name z, exported from Z that refers to the entity Z.z. Entities are exported and imported just like instances: they are always exported (and re-exported) from a module and always imported. We have tighter control of names, though. Under this interpretation, all entities in the result of a splice must be in scope in a normal import.

Is this too complicated? Maybe. It does seem, to me, somewhat simpler than quote-imports.

[Ericson2314]

@goldfirere If I understand you correctly, your design means that if B changes, then consumers like C will need to adjust their import A () and similar. Do we really want that?

I feel like we should just do the Racket thing -- TH's biggest problem is not any one technical issue, I would say, but rather the fact that it is only in partial dialogue other langauge's macro efforts, especially as macros are more important and more studied over there than over here. I am excited about this sort of thing as a way to repair that conversation as much as I am about it fixing any particular issue.

At the very least, if we don't want to do the Racket way, I would like very detailed reasoning on why.

[treeowl]

I do broadly think that "Just do what Racket does" is a very good default position on macro systems. That said, Racket has a serious module system and Haskell doesn't, so the situations are not entirely identical.

[int-index]

Seems like there are some design questions to be resolved. Do we want quote imports or not? Assigning back for revision until the proposal takes a concrete stance on this. Hopefully the implementation efforts will also yield further insights.

[mpickering]

How I understand "the racket way" is like this:

Quote and splice imports express relative offsets for the current module.

• A splice import says I need this import a offset -1 to the current module
• A quote import says I need the import at offset +1 to the current module

When the module is compiled, we ask to compile it at a specific level, usually 0. This then fixes the level all the imports are required it, which fixes the level all it's imports are required at and so on. The result is a build graph which dictates at which levels all the modules must be compiled at.

In a future where GHC is multi-target, then each level can be for a different target.

CSP

Cross-stage persistence adds the assumption that if a module is available at n then it is available at n + 1. Therefore an import which is implicitly persisted using top-level CSP adds a dependency on both n and n + 1 implicitly.

Current Restrictions

At the moment GHC is not multi-target in the sense that we can switch generating code for different architectures at runtime. There are however two different modes which can be configured at runtime.

• -fno-code, don't produce any object files
• Normal compilation, produce object files

It only makes sense to use -fno-code at the last stage, as you wouldn't be able to execute any of the prior stages if you don't generate any code.

So if you set the top-level target to be compiled with -fno-code then if you encounter an import splice that will require the import is at stage -1, which is a object code stage. If that module import quotes a module, that module will only be required at level 0, a -fno-code stage, so that module doesn't have to be compiled to object code, just typechecked.

Example

Compiling the top module with -fno-code,

module A where

import splice B

foo = $(b)

module B where

import quote C

b = [| c |]

module C where

c = 0

> ghc -fno-code A
[1 of 3] C (Nothing)
[2 of 3] B (B.o)
[3 of 3] A (Nothing)

[simonpj]

I started implementing the proposal and it's not clear to me now how splice imports should work.

In my head at least, this proposal is now back with the author for revision, in the light of this discovery. It's not easy to see a way through here. I agree that Racket is an excellent place to look.

[Ericson2314]

Cross-stage persistence

Is incompatible with cross compilation, so I rather have a design that doesn't assume/require it so we can add knobs for that later.

At the moment GHC is not multi-target in the sense that we can switch generating code for different architectures at runtime.

Just a nit, since your -fno-code shows we should care about such things anyways, but GHC is quite close to that. In particular, for a few release now, everything but the native word size has been not hard-coded and instead controlled by the settings file. That last remaining hurdle we are also edging towards.

[Ericson2314]

To add on to @mpickering's good example, with more fine-grained scheduling, I would hope to see something like:

> ghc -fno-code A
[1 of 4] C Tc
[2 of 4] B Tc
[2 of 4] B Cg (B.o)
[3 of 4] A Tc

> ghc A
[1 of 6] C Tc
[2 of 6] B Tc
[2 of 6] B Cg (B.o)
[3 of 6] A Tc
[2 of 6] C Cg (C.o)
[2 of 6] A Cg (A.o)

[int-index]

Yeah, I agree that:

• We should copy what Racket does until there’s a reason to do otherwise. Or at least we should pick a forwards-compatible subset of it.
• Cross-compilation is an important use case to support, and the design should facilitate it, not hinder it. (I’m happy to trust @Ericson2314’s judgement in this regard).

[sgraf812]

I'm all for starting with what Racket does, but we should at all times question whether we can make it simpler or need to do it differently due to our use case (that is: metaprogramming and not module systems).

So I'm all for sketching out how Racket-style quote imports and splice imports should work in Template Haskell.
Part of that is figuring out how cross-stage persistence as implemented by Lift should work, which Racket does not have or need, as far as I can tell. (Perhaps it is implicitly part of its module system, though.)

We should really figure out how cross-stage persistence interacts with import splice/import quote, and which role Lift plays for cross-compiling. For example, if I write

module Def where

n :: Int
n = 0

{-# LANGUAGE TemplateHaskell #-}
module Use where

import Def
import Language.Haskell.TH.Syntax

laterN :: Q Exp
laterN = [| n |]

nowN :: Q Exp
nowN = [| $(lift n) |]
  -- = lift n

The bindings laterN and nowN desugar to

nowN = lift $fLiftBoxedRepInt $fQuoteQ n

laterN
  = varE
      $fQuoteQ
      (mkNameG_v
         (unpackCString# "main"#)
         (unpackCString# "Def"#)
         (unpackCString# "n"#))

So nowN needs the n from stage 0 (it occurs inside a quote inside a bracket, after all!), whereas laterN needs it only in stage 1 (quote stage). If it were only for laterN, we could use import quote Def, but not so with nowN.

Thus, implicit use of lift (such as in f x = [| x + 1 |], which desugars to f x = [| $(lift x) + 1 |]) must be treated like $(lift x), requiring x at stage offset -1. (Do we implicitly lift top-level identifiers such as n as well? Probably not!)

That would also allow us to understand Lift for what it is: It lifts some value from host stage 0 into a datum of the target stage 1, without needing to run code on the target. It is the job of the Lift instance to do this in a cross-compatible way (taking care of integer overflows, etc.). Explicit lifting by the user is their way to signal that some logic should be executed on the host and then converted into some datum for the target in a way that their Lift instance specifies.

To conclude, I don't think we need to ban cross-stage persistance/Lift. I think we can and should allow the following program, which requires n only on the host (note the import splice):

{-# LANGUAGE TemplateHaskell #-}
module OtherUse where

import splice Def
import Language.Haskell.TH.Syntax

copy :: Int
copy = $(lift n)
-- expands to `copy = I# 0#`

[sgraf812]

I have a feeling that the import quote/import splice distinction could easily be inferred by the compiler in the renaming phase (it can be checked, after all!).

But the appeal of this extension is perhaps to make the information evident in the module header, so that a static build plan can be constructed that says "compile module A for the target, module B for the host and module C for both" à priori, just by looking at the header and knowing what architecture the ultimate executable should run on.

This is a bit simpler than starting to rename module C, which would then need to be interrupted upon seeing $(B.f ...) and wait on full compilation of B.f. Arguably this would require some kind of coroutine mechanism that might not be so simple to implement across process boundaries. (But perhaps this would not be too complicated to implement after all!)

Regardless, I would like to see an argument for explicit annotations vs. inference in the proposal!

I thought that another reason is assurance for the user that module A is only compiled for the host and module C is only compiled for the target. But alas, I don't think that is possible to state, because it depends on how modules A and C are imported (perhaps by quote or by splice themselves), not on whether the modules import by splice or by quote.

So yeah, perhaps simply inferring quote and splice imports and persisting this information in the interface file would be a viable alternative to this proposal, and would perhaps fix https://gitlab.haskell.org/ghc/ghc/-/issues/14095 as well. It would like to discuss this and clarify the issue in the Alternatives section.

---

Furthermore, I was recently made aware of the following staged metaprogramming system for OCaml: https://dl.acm.org/doi/pdf/10.1145/3607851 it would be good to verify that their notion of quote and splice imports does not deviate too much from ours without good reason.

[michaelpj]

An observation that came up in the ecosystem workshop. HLS currently has an awful hack around TH. Notably, it has to compile any module dependencies which are needed for splice evaluation to bytecode, but it really wants to compile only those modules. Today, it's not possible to know which modules are necessary for splice evaluation until you actually typecheck the module. So HLS uses hscCompileCoreExprHook to do just that: typecheck the module, and then when you hit a splice, jump back out and ensure the build system has built the bytecode for the modules that you use.

However, if we had explicit splice imports we could get this information trivially from looking at the module header.

[adamgundry]

I think we can consider this proposal superseded by #682, which is effectively a revised version. Thanks to @mpickering for your work on this and to everyone who commented,