Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backpack for Stack #2540

Open
ezyang opened this issue Aug 29, 2016 · 72 comments
Open

Backpack for Stack #2540

ezyang opened this issue Aug 29, 2016 · 72 comments

Comments

@ezyang
Copy link

@ezyang ezyang commented Aug 29, 2016

Hello all,

I want to open up the discussion about how Backpack can be integrated with Stack; in particular, I'd be willing to work on and submit a patchset adding support for it in Stack. But before I do that, I want to hash out some general architectural concerns. The specification for the GHC and Cabal support for Backpack is here: https://github.com/ezyang/ghc-proposals/blob/backpack/proposals/0000-backpack.rst ; please read it for more motivation on why you might like to support it.

The design process over two years lead us to a design where Backpack requires the package manager (Stack) to know how to instantiate packages that have required signatures in them (I spent the better part of six months trying to figure out how to do it all in GHC; if you are interested in the sordid details I can let you know). Things seem to work most smoothly if:

  1. The package manager has a component oriented build model; that is, the unit of work in the install plan is a component rather than a package. (I don't believe Stack is architected in this way, although issues like #1406 suggest it might be a good architecture for Stack anyway.) See also https://github.com/ezyang/ghc-proposals/blob/master/proposals/0000-componentized-cabal.rst for the proposal that added per-component configure to Cabal
  2. The package manager knows enough to be able to finalize the package description (i.e., resolve all of its conditionals, getting a PackageDescription); we can't do Backpack instantiation until this finalization has occurred. I don't know where in the Stack codebase one would have to engineer this.

I'll note that private use of Backpack (i.e., the public library does not have any signatures) should be supported out-of-the-box, as there is no change to the outward-facing API of Setup scripts in this case (Stack will need to understand how to register multiple libraries produced by a Setup script, however). In the meantime, public libraries which have signatures would not be supported.

One of the major applications of Backpack I would like to see in the near future is the Backpack-ification of GHC's boot libraries to solve the string problem. Backpack would make it easy to write modules which are parametric over a particular desired string representation; the original package names would continue to export String-based interfaces, but versions using ByteString etc could be made available, and the availability of a package abstracted over holes means that an end-user could specify their own. Absent full support for Backpack, Stack would at least have to be able to handle traditional-looking packages which internally depended on libraries that made use of Backpack. Maybe this would be easy.

Please let me know if you have any questions or concerns; I'd be happy to answer them.

CC @mgsloan, @snoyberg, @Blaisorblade , @borsboom , @harendra-kumar

@mckeankylej
Copy link

@mckeankylej mckeankylej commented Aug 29, 2016

This would be awesome.

@Blaisorblade
Copy link
Collaborator

@Blaisorblade Blaisorblade commented Aug 29, 2016

@ezyang First let me thank you for all your work on this.

TL;DR. Does stack need this shipped tomorrow, in a month or in a year?

Before I can look into details and ask more substantive questions: is there a timeline for Backpack being merged/released/used, and an estimate how much work was required on cabal-install?

Absent full support for Backpack, Stack would at least have to be able to handle traditional-looking packages which internally depended on libraries that made use of Backpack. Maybe this would be easy.

I agree this is probably essential. Having feature parity also matters—having to choose between stack and Backpack, or stack and sane strings, would be pretty bad.

@borsboom
Copy link
Contributor

@borsboom borsboom commented Aug 29, 2016

@mgsloan, I think you know the most about the parts of Stack that @ezyang is talking about. Once you're back from BM, can you weigh in?

@ezyang: how is cabal-install going to handle Backpack? Both cabal-install and Stack use the Cabal library to actually build packages, so there may be opportunity to share much of the code.

@ezyang
Copy link
Author

@ezyang ezyang commented Aug 29, 2016

Does stack need this shipped tomorrow, in a month or in a year?

Definitely not tomorrow or a month. If all goes well Backpack will get merged to GHC and Cabal, and people can start experimenting with it, but there were certainly be bugs and I won't claim that anyone should start using it for real immediately. Backpack'ing base is at least blocked on teaching GHC's build system how to deal, which will take some time and I am not planning for 8.2. On the other hand, base is probably the most plausible, widely used package that will get Backpack'ed, because (1) the potential upside is so great, and (2) it closely tracks GHC versions, so we don't have to worry about supporting old GHCs.

how is cabal-install going to handle Backpack? Both cabal-install and Stack use the Cabal library to actually build packages, so there may be opportunity to share much of the code.

There are some key library functions for configuring and instantiating packages, and cabal-install (new-build only; the legacy codepath doesn't support it) just calls out to them. At present, there are some things that are duplicated because I couldn't figure out an API that could be reused in both cases, but this is definitely a case where I can try harder.

The two key functions have these signatures:

type ConfiguredComponentMap =
        (Map PackageName (ComponentId, PackageId), -- libraries
         Map String ComponentId)                   -- executables
toConfiguredComponent
    :: PackageDescription
    -> ComponentId
    -> Map PackageName (ComponentId, PackageId) -- external
    -> ConfiguredComponentMap
    -> Component
    -> ConfiguredComponent

type LinkedComponentMap = Map ComponentId (UnitId, ModuleShape)
toLinkedComponent
    :: Verbosity
    -> PackageId
    -> LinkedComponentMap
    -> ConfiguredComponent
    -> LogProgress LinkedComponent

and a LinkedComponent has all the information to actually build with Backpack (the model is that every source package produces a configured component, which is mix-in linked into a linked component; then you go ahead and instantiate all the linked components getting the final graph of build products you need to build.)

@chrisdone
Copy link
Member

@chrisdone chrisdone commented Sep 5, 2016

The package manager knows enough to be able to finalize the package description (i.e., resolve all of its conditionals, getting a PackageDescription); we can't do Backpack instantiation until this finalization has occurred. I don't know where in the Stack codebase one would have to engineer this.

Stack does this. We use Cabal to parse the .cabal file but then resolve the GenericPackageDescription manually. Here is an example: https://github.com/commercialhaskell/stack/blob/master/src/Stack/Package.hs#L765 The whole of Stack.Package is pretty much dedicated to resolving package info. See also https://github.com/commercialhaskell/stack/blob/master/src/Stack/Package.hs#L198

@ezyang
Copy link
Author

@ezyang ezyang commented Mar 1, 2017

Great. Another question I have (since I am looking into this) is how difficult it would be to get Stack to support a per-component build model. I'm hoping that I can take anywhere we talk about 'Package' and actually it's talking about a 'Component', but I am not familiar enough with the code to say if this is the case or not.

@mgsloan
Copy link
Contributor

@mgsloan mgsloan commented Mar 3, 2017

@ezyang Could be tricky! The current code is more package centric than component centric. This is for a variety of reasons. Off the top of my head:

  1. Dependencies vary based on whether --enable-tests or --enable-benchmarks is passed to Cabal, so this needs to be figured out on a package level
  2. Avoiding package reconfiguration - best to configure all the components we want at once

And no doubt, more. So, no, when the stack code is talking about a 'Package' it's definitely not talking about a 'Component'.

@ezyang
Copy link
Author

@ezyang ezyang commented Mar 7, 2017

OK. So perhaps a reasonable approach to start with is to assume that any package which exposes a Backpack library, ONLY contains that library, so that if I need to instantiate a package multiple times, this would only involve creating duplicate copies of the package (another thing where I'm not sure how well Stack will deal with this change.)

@ezyang
Copy link
Author

@ezyang ezyang commented Mar 10, 2017

I've been fishing around the codebase, and here are some initial architectural thoughts on what Backpack needs.

  • Stack defines a simplified Package data type, which it reads out the contents of PackageDescription into. Backpack introduces some new fields, most notably mixins, which need to be preserved so that Backpack can figure out how packages are being instantiated. At the moment, Package doesn't really preserve any information like exposed modules which we would need to do this. It seems the most convenient way of keeping all the information Backpack needs is to keep PackageDescription verbatim in Package? This will also make it easier to call into Cabal to do mix-in linking.

  • Once we have all the packages, we need to do a "mix-in" linking step to figure out exactly how packages have been instantiated. For example, if I have build-depends: regex-indef, str-string, I need to figure out that regex-indef's signature named Str was instantiated with the Str from str-string.

    It is an interesting question on how to integrate this with Stack. Stack's model is that it goes ahead and reads in all of the local source packages, but then lazily reads in the external package information using the accessor function created by withLoadPackage. The idea is that if a package is already installed, then we can avoid traversing its dependency graph. This should integrate reasonably well with Backpack: if a package is installed, we can read off the module shape (that's the metadata you need to mix-in link) from the installed package info; if it is not, we can compute it when we installPackage.

  • Mix-in linking is not enough: after we figure out how everything is linked, we have to instantiate, creating a build task for every configuration by which a package was instantiated. In Stack, a build Plan consists of some number of Tasks to be done. I'm not sure how dependencies between tasks are computed, but each task is uniquely identified by the PackageIdentifier it provides (taskProvides). A Task roughly corresponds to a configure/build pair on Cabal. So, we need to modify taskProvides to be a bit more flexible: rather than only build a PackageIdentifier, it builds both a package identifier and an instantiation (to be defined shortly). Let's call such an identifier a UnitIdentifier, following the terminology I've used in Cabal and my thesis. Roughly speaking, these data structures should look like this:

    data UnitIdentifier = UnitIdentifier PackageIdentifier [(ModuleName, Module)]
    data Module = Module UnitIdentifier ModuleName
    

If Stack was modified in this way, would you take the patch? How would you like the design to be different? What is unclear?

@mgsloan
Copy link
Contributor

@mgsloan mgsloan commented Jul 27, 2017

Hey @ezyang , sorry for letting this languish, got caught up in other things. Passing this over to @snoyberg

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jul 27, 2017

I'm just coming up to speed on this now Edward. It seems relevant to work I'm already doing upgrading to Cabal 2.0. I'm also in favor of pushing Stack towards a component base build system. My only concern with this is making sure we retain support for building packages with older versions of Cabal, as all existing snapshots include such an older library version.

I realize this issue is a bit dated now, and perhaps you have different thoughts. Maybe we could jump on a text chat and synch up?

@ezyang
Copy link
Author

@ezyang ezyang commented Aug 2, 2017

@snoyberg Sure thing; ping me on IRC some time? As far as I am concerned, the thinking on this ticket hasn't changed much (although maybe Stack's internals have!)

My only concern with this is making sure we retain support for building packages with older versions of Cabal, as all existing snapshots include such an older library version.

Yeah, so in cabal new-build, this is done by treating these packages as non-per-component, single packages in the plan. It's a little bit of complexity but not too bad, and keeps BC.

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Aug 2, 2017

Just chatted with @ezyang on IRC about this. Here's my summary:

  1. We first need to get Cabal 2.0 support merged into Stack, which is on me (ETA: 2 weeks)
  2. Once that's done, we will switch Stack to a per-component build plan. This is something that I'm very excited about in fact, see haskell/cabal#2802
  3. The biggest wrinkle is continuing support for pre-2.0 Cabals, which will be necessary for any LTS or Nightly release before GHC 8.2. Basic plan of attack is:
    1. Every build plan is calculated on a per-component basis
    2. If we're using Cabal 2.0 or later, we will also build on a per-component basis (separate configure with a separate dist dir for each and every component)
    3. If we're using pre-2.0 and there are no cycles in dependencies (which can happen in test suites and benchmarks), then we convert the per-component plan to a simple per-package plan and build
    4. If we're using pre-2.0 and there are cycles, we will convert into a staged per-package plan: any packages that have cycles will be split into lib/exe builds and full builds (including the tests and benchmarks). This is essentially the same as we have today, but can hopefully be streamlined by relying on the per-component plan logic. (Example use case: running stack test inside the bytestring package.)

I'll ping this issue with updates when Cabal 2.0 is merged in

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Aug 17, 2017

Pinging appropriately: Cabal 2.0 support has just been merged to master :)

@mgsloan
Copy link
Contributor

@mgsloan mgsloan commented Aug 18, 2017

One thing for switching stack to a per-component build plan will also be adding support for internal libraries - #3361

Also support for foreign libraries, not sure if that's as relevant - #3364

snoyberg added a commit that referenced this issue Sep 13, 2017
This should resolve both #3364 and #3361. There is a test case included
that should address both of them as well.

This is still relatively hacky. We will eventually be overhauling the
component system more dramatically to support Backpack with #2540 (CC
@ezyang).

This may still have some problems due to the upstream issue
haskell/cabal#4763, but at least in theory we're matching the behavior
of upstream. We can consider workarounds after that issue comes to a
conclusion.
@centromere
Copy link

@centromere centromere commented Nov 17, 2017

What work needs to be done to finish adding Backpack support to stack?

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Nov 19, 2017

There's still quite a bit of work to be done, see my comment above #2540 (comment). All that we've done so far is get support for Cabal 2.0 into Stack.

@ezyang
Copy link
Author

@ezyang ezyang commented Dec 31, 2017

I'm working on per-component build for Stack in https://github.com/ezyang/stack/tree/per-component-build

It doesn't build yet, but I've finally gotten it to typecheck to ConstructPlan, which means the real work can begin :)

@mgsloan
Copy link
Contributor

@mgsloan mgsloan commented Jan 2, 2018

Great, thanks @ezyang ! I skimmed it, and it looks good so far. Feel free to open a PR before its ready, as soon as you are ready to start getting review feedback.

@mikeplus64
Copy link

@mikeplus64 mikeplus64 commented Apr 24, 2018

How's this come/coming along?

@ezyang
Copy link
Author

@ezyang ezyang commented Apr 25, 2018

Very, very slowly XD

@LeanderK
Copy link

@LeanderK LeanderK commented Jun 7, 2018

any progress? 🙂
I don't want to be rude, it's just that I would really like to delve deeper into backpack. But not being able to integrate it into stack-projects is pretty discouraging (all of my projects use stack and there are also a lot out there 😕). I think it's a bit of a chicken or the egg problem, what comes first? Support or usage?

@stites
Copy link

@stites stites commented Jul 17, 2018

I noticed that this ticket has a call for help via "A year into Backpack." Since this is needed for the hasktorch project to get into the LTSs, I've posted a bounty for this issue.

@matthew-piziak
Copy link

@matthew-piziak matthew-piziak commented Jul 21, 2018

From @ezyang's blog.

In Stack issue #2540 I volunteered to implement Backpack support for Stack. However, over the past year, it has become abundantly clear that I don't actually have enough spare time to implement this myself. Looking for brave souls to delve into this; and I am happy to advise about the Backpack aspects.

Hey @ezyang, it was nice of you to volunteer, but when you realize you don't have time for a ticket it's good etiquette to say so in the ticket thread. That way people know it's unassigned and up for grabs. Don't lick the cookie—pass the plate!

You've done great work in characterizing this issue so far and I don't want to diminish that. But as they say, the last responsibility of a maintainer is to find a replacement, and it's good to do that in a centralized way. Many developers are on tenterhooks by this feature, so it's important that its status—viz. is somebody working on it / does it need a maintainer—is clearly advertised.

@mboes
Copy link
Contributor

@mboes mboes commented Jul 21, 2018

@matthew-piziak Cookie licking sounds pretty incongruous to this particular situation. @ezyang did a tremendous amount of work to actually get Backpack merged into GHC, and also on this particular ticket. This task isn't assigned to anyone and @ezyang makes it clear in his blog post that he'd be happy to see someone else pick this up.

@matthew-piziak
Copy link

@matthew-piziak matthew-piziak commented Jul 21, 2018

@mboes You're right, I realized now that I had a poor understanding of the ticket's history. I apologize for the misleading language in my comment. In any case I am happy that this issue's status has been clarified.

@junjihashimoto
Copy link
Contributor

@junjihashimoto junjihashimoto commented May 20, 2019

So I think we need some rebuilding processes.
Is the code of cabal's rebuild here?

https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectBuilding.hs#L607-L618

@dbaynard
Copy link
Contributor

@dbaynard dbaynard commented May 21, 2019

@dbaynard your test of backpack passes on latest stack

These tests should be considered work in progress. Thanks for identifying an issue! I'm not able to answer your questions, but I will be able to take a look, soon.

@ezyang
Copy link
Author

@ezyang ezyang commented May 22, 2019

Let me know if you want me to take a look. There isn't anything obvious that pops out from the error message, besides, "Sounds like Stack isn't compiling enough instantiations of the signatures"

@junjihashimoto
Copy link
Contributor

@junjihashimoto junjihashimoto commented May 23, 2019

@ezyang Thx! If you take a look, it would be a great help.
If you think the looking is not worth to proceed with this issue, you do not need to look at it.
Maybe it is better to put some tests in stack's test.
The code outputing above erros is hasktorch/hasktorch#149.
If you want any codes, logs and images(docker), I can provide you.

@TomLippincott
Copy link

@TomLippincott TomLippincott commented May 27, 2019

@ezyang @junjihashimoto I'm getting the same errors: let me know if I can end-user-test any potential solutions.

@haitlahcen
Copy link
Contributor

@haitlahcen haitlahcen commented Sep 12, 2019

I don't know if its related but I faced the same issue (Invalid package ID) on a backpacked project and fixed it by adding + in the parser.

@cartazio
Copy link

@cartazio cartazio commented Feb 3, 2020

this seems like a pretty egregious bug. why isn't it fixed?

@jkachmar
Copy link

@jkachmar jkachmar commented Feb 3, 2020

@cartazio For what it's worth, backpack is currently not fully supported by any build tools outside of cabal-install. In light of this I wouldn't consider the lack of support for it within stack to be a particularly "egregious" bug, but rather a sign that there is limited interest in supporting this functionality beyond cabal-install.

For reference, please note the following tracking issues showing that support for backpack is not a particularly high priority elsewhere in the Haskell build tool ecosystem:

nixpkgs
rules_haskell
haskell.nix

@cartazio
Copy link

@cartazio cartazio commented Feb 3, 2020

while i can empathize, i dont sympathize.
This is a bug in those tools, and perhaps they just haven't been motivated enough to take care of them.

@cartazio
Copy link

@cartazio cartazio commented Feb 3, 2020

if they support ghc, this is a bug. If they just support haskell 2010, fine. but this means any post Haskell 2010 feature support is underspecified.

Do these tools support extensions/changes relative to Haskell 2010? What reference implementation is guiding them? I think you're saying "not ghc/cabal, but something more nebulous"

@jkachmar
Copy link

@jkachmar jkachmar commented Feb 3, 2020

if they support ghc, this is a bug

I suppose my point here is that we've gone several years with all of the alternative build tools to cabal-install not supporting this feature, which (to me) means that regardless of whether it's considered to be a bug upstream it's certainly not a pressing problem for consumers of these tools.

Do these tools support extensions/changes relative to Haskell 2010? What reference implementation is guiding them?

My point was more that there doesn't seem to be much pressure on any build tools to support backpack, and there doesn't seem to be much negative backlash from the community for not having solved them (that rules_haskell issue is nearly two years old at this point).

To be fair, this is one of those tricky things in the space of open source community management. If the compiler implements a feature, but not enough people care to use it (or push for its adoption), is it really a bug not to support that?

EDIT:
This hadn't occurred to me when I wrote this up initially, but I think it's also important to consider this from the perspective that others may not consider a missing feature to be synonymous with a bug.

In the nixpkgs issue, you can see that the maintainer has even marked this functionality as an "enhancement".

@flip111
Copy link
Contributor

@flip111 flip111 commented Feb 3, 2020

it's certainly not a pressing problem for consumers of these tools

Most of the projects can move forward without backpack. Just that it offers really nice features which people like to use. "Not a pressing problem" doesn't justify the attention this issue got and the nearly $500 bounty on it

@cartazio
Copy link

@cartazio cartazio commented Feb 3, 2020

@cartazio
Copy link

@cartazio cartazio commented Feb 3, 2020

to be clear: (and a stack contributor nicely pm'd me on IRC to help me clarify what i mean)

  1. i'm the maintainer for vector
  2. i'm exploring ways to make vector better for everyone (even users who just assume the api is gonna be the same always)
  3. one direction that seems the most humane (vs commit war crimes of CPP or TH) is evaluating if backpack can support evolving vector while reducing the current material code footprint of the current vector code base (the Boxed/Storable/Unboxed vector apis are mostly the same/ share the same code, but with type specializations for improved inference and friendly UX for newcomers).

many packages depend on vector directly or indirectly. If vector starts using backpack, this impacts users of tools that dont support it.

If backpack is an effective tool to support evolving vector aggressively while maintaing super stable user interfaces and much more friendly "write your own new vector instances " workflow for users who want, idk, memmory mapped or gpu or tape-drive fusioned vector data types, i want to be able to do this. Or I want YOU to be able to this.

Many people use stack as a personal choice, and I dont want stack users or vector users to suffer. Accordingly it seems pretty important for backpack to be supported should this design mix pan out. Otherwise any library using vector directly or indirectly will not be buildable with stack.

@cartazio
Copy link

@cartazio cartazio commented Feb 3, 2020

on a meta note: i was concerned about how certain folks would react to my input on that ticket, i'm disappointed to be exactly correct about the type of abuse certain people hurl on the twitters
JoeKachmar-is-a-dick-02-03_at_2 38 11_PM

@flip111
Copy link
Contributor

@flip111 flip111 commented Feb 3, 2020

Summarized and paraphrased information so far.


Using components (parts of a package): allows for an architecturally sound implementation of Backpack, as the process of instantiating requirements of a Backpack only makes sense for libraries, rather than an entire package (an executable and test suite cannot be instantiated in any meaningful sense).


Interaction with Backpack. Backpack needs to perform mix-in linking on components, and then a further expansion step to instantiate components. Mix-in linking occurs as we are expanding the package graph into a component graph, while instantiation happens as a separate step after expansion.

At the moment stack does not have a graph for components. "mix-in" relates to which types are used for a package from which other package. See: #2540 (comment)


There are some key library functions for configuring and instantiating packages, and new-build just calls out to them. The two key functions toConfiguredComponent and toLinkedComponent

and a LinkedComponent has all the information to actually build with Backpack (the model is that every source package produces a configured component, which is mix-in linked into a linked component; then you go ahead and instantiate all the linked components getting the final graph of build products you need to build.)


how difficult it would be to get Stack to support a per-component build model?

Could be tricky! The current code is more package centric than component centric. This is for a variety of reasons. Off the top of my head:

  1. Dependencies vary based on whether --enable-tests or --enable-benchmarks is passed to Cabal, so this needs to be figured out on a package level
  2. Avoiding package reconfiguration - best to configure all the components we want at once
    And no doubt, more. So, no, when the stack code is talking about a 'Package' it's definitely not talking about a 'Component'.

Open question: Does stack know how to register multiple libraries produced by a setup script? --> If it can then mechanisms there could be re-used to do components (as one library would be one component)


So perhaps a reasonable approach to start with is to assume that any package which exposes a Backpack library, ONLY contains that library, so that if I need to instantiate a package multiple times, this would only involve creating duplicate copies of the package

An approach to start getting a proof of concept? Maybe better to have real components to get "architecturally sound implementation".


Best plan for implementation:

  1. See point 3 in #2540 (comment)
  2. Re-use types from https://www.haskell.org/cabal/release/cabal-latest/doc/API/Cabal/Distribution-Backpack.html
  3. Re-use or take inspirations from tests https://github.com/haskell/cabal/tree/master/cabal-testsuite/PackageTests/Backpack and dbaynard/stack@b8d8859...dbaynard:fb61d68
  4. Take inspiration from latest attempt master...ezyang:per-component-build It (used to) typecheck ConstructPlan

Feel free to correct me :)

@ezyang what did you mean with "register" in your opening post?

@flip111
Copy link
Contributor

@flip111 flip111 commented Feb 3, 2020

@snoyberg can you explain how far this file has the type of component that is needed to support backpack ?? https://github.com/commercialhaskell/stack/commits/master/src/Stack/Types/NamedComponent.hs Is this in principle the same "component" but might need some enhancement to store backpack specific information ?

EDIT: also is it ok for stack to use types like https://github.com/haskell/cabal/blob/1a31242dfe1f0376051057370e25cc846e319ea8/Cabal/Distribution/Types/Component.hs

I'm not sure what parts of cabal can being pulled in or not

@Bodigrim
Copy link

@Bodigrim Bodigrim commented Feb 4, 2020

Well, for what it's worth, I've been recently discussing with colleagues switching back from stack to cabal as our main build tool because of Backpack support.

Of course, there is not much pressure at the moment, because there are few Backpacked packages, because package authors avoid Backpack, because it cannot be built by one of two main Haskell build tools. It is a pity that packages cannot explore features, merged into GHC four versions ago. Changing this status quo would be a massive shift IMO.

On the other hand, this is open source: people are free to pop in, fix an issue and get away with praise/glory/bounty. It would be unfair to blame maintainers of stack for not catering everyone's needs immediately. Backpack support is no small thing.

@evanrelf
Copy link

@evanrelf evanrelf commented Feb 4, 2020

I agree with @Bodigrim. Very few Haskell packages take advantage of Backpack because a major build tool (stack) doesn't support it. Which means there's little motivation for stack developers to add support. It's a negative feedback loop.

@sjakobi
Copy link
Member

@sjakobi sjakobi commented Feb 4, 2020

Note that there are several GHC bugs related to backpack, although I don't see anything too bad.

@flip111
Copy link
Contributor

@flip111 flip111 commented Feb 4, 2020

@ezyang are the OpenUnitId and DefUnitId from Distribution.Backpack serving the same function as the proposed types

data UnitIdentifier = UnitIdentifier PackageIdentifier [(ModuleName, Module)]
data Module = Module UnitIdentifier ModuleName

My guess would be that these "units" are not the same kind of units, because one includes PackageIdentifier and the other doesn't. Confirmation would be appreciated.

@flip111
Copy link
Contributor

@flip111 flip111 commented Feb 4, 2020

@ezyang

* rather than only build a `PackageIdentifier`, it builds both a package identifier and an instantiation (to be defined shortly). Let's call such an identifier a `UnitIdentifier`

Is it possible that such an instantiation requires a new package? Or is it fine to do these instantiations after package dependencies have been calculated ?

@flip111
Copy link
Contributor

@flip111 flip111 commented Feb 4, 2020

On how to blend backpack code with stack code ...

Searching in cabal's codebase for toLinkedComponent brings up the buildComponent function. Even if this function was on top level and being exported on first glance it seems there is too much cabal specific stuff in there.

On the one hand for stack's purposes it seems useful to do the expansion of tasks to build that is needed for backpack as late as possible. So after tasks are made expand them in yet even more tasks.

tasks = M.fromList $ mapMaybe toTask adrs
This prevents having to modify code earlier and thus having to carry changes through all code paths.

On the other hand toConfiguredComponent and toLinkedComponent need information that will only be available earlier than toTask so this information needs to be carried through anyway.

First step seems to be to figure out where the arguments from toConfiguredComponent come from and if they can be constructed (have data constructors exported) and where in the stack constructPlan this information is available. After that the same for toLinkedComponent.

After that see if the return value of toConfiguredComponent correspond to a type Stack has, and wether this return type (LogProgress ConfiguredComponent) can be embedded in a Stack type.

@bbarker
Copy link

@bbarker bbarker commented Feb 4, 2020

I agree with @Bodigrim. Very few Haskell packages take advantage of Backpack because a major build tool (stack) doesn't support it. Which means there's little motivation for stack developers to add support. It's a negative feedback loop.

As a very new Haskell user, I was exposed to backpack (while still getting my bearings straight with regard to Haskell build tools), only to realize it would be a no-go if I wanted to use it with Stack. I've since been tempted to use it again, but realized the issue was still open. Had I been a user of backpack all along (which has only been a little over a year), I'm sure I'd have thought of many more places I could have used it. Just my 2c - I understand all too well that development time is limited.

@ezyang
Copy link
Author

@ezyang ezyang commented Feb 14, 2020

Sorry about the delay, I saw this in my GH notifications but sat on it for a week.

@flip111

what did you mean with "register" in your opening post?

referring to:

Stack will need to understand how to register multiple libraries produced by a Setup script

My recollection from the last time I looked at the Stack code, was that everything in Stack revolves around installing packages (there's a dep graph of packages, you install them one by one). The comment on private Backpack use is simply pointing out that sometimes installing one package results in multiple libraries getting installed, and as long as Stack can deal with this situation, all is good.


are the OpenUnitId and DefUnitId from Distribution.Backpack serving the same function as the proposed types

Yeah, they're not the same, precisely because the proposed type here refers to packages rather than components.

Components are a pretty big deal for usability of Backpack, because a lot of things you want to do involve having to write many small libraries and then recombining them together. If you'd have to make (and distribute) a separate package for each of these libraries, it would be a serious source of friction.


Is it possible that such an instantiation requires a new package? Or is it fine to do these instantiations after package dependencies have been calculated ?

It's fine to do it after depsolving; that's how Cabal-install does it. Depsolve the package structure, and then elaborate it into a more fine-grained component structure.


As for the last few comments, the kit in the Cabal library is what stack can usefully use, the stuff in cabal-install is a useful reference for how to use the APIs but I expect none of it is salvageable. That's OK; I have a "mini" implementation of the Cabal-install logic for Setup.hs and it was mostly fine.

To me, the problem was always how to drag Stack into the component world. That required more time investment learning about Stack that, sadly, I never found time to do in the end.

@snoyberg snoyberg removed this from To do in Pantry Feb 18, 2020
@cartazio
Copy link

@cartazio cartazio commented Feb 28, 2020

@snoyberg is this on the todo queue of y’all ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.