Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Post-multilibs checklist #5660

Open
17 of 27 tasks
fgaz opened this issue Nov 4, 2018 · 27 comments
Open
17 of 27 tasks

Post-multilibs checklist #5660

fgaz opened this issue Nov 4, 2018 · 27 comments

Comments

@fgaz
Copy link
Collaborator

@fgaz fgaz commented Nov 4, 2018

#5526 (multiple public libraries) is in, but there are still a few things to add and polish

Current blocker: #7270

Visibility enforcement

  • Add a visibility field to sublibraries πŸ”₯ πŸ”₯ πŸ”₯ ❗ ❗ ❗
    • add the field
    • update ghc-pkg
    • populate the field
    • enforce the field: #5848

cabal-install integration

  • Make the solver multilibs-aware #6039
    • Enforce visibility at the solver level #6047 #6836
    • Extend the compatibility to GHC<8.8 when using cabal-install #6836
    • Handle installed packages
  • Update cabal init to take into account sublibraries when populating build-depends

Tests

  • Add tests
    • positive (ghc>=8.8) (isssue #6409)
    • negative (all ghcs) (done in #5848)

Documentation

  • Update changelog 4d2ca52
    • #6362 Also update the cabal spec changelog, which I forgot
  • Update the Cabal and cabal-install docs (can be adapted from my post)

Bugfixes

  • #6038 (cannot depend on public sublibrary when the package does not contain a main library too)
  • #5846 (boundless dependency fails to parse)
  • #6083 (edge case with the old syntax)
  • mylib:{ } ^>= 42 syntax is accepted, though docs say it is not (#6083 related, makes sense to fix at the same time?) - fixedin 9b3686f
  • #6281 (parse failure with mixins)
  • #6358 Reconfiguration avoidance not affected by library visibility change
  • πŸ”₯ #7270 πŸ”₯ Cabal's simple solver may not pick up sublibraries unless --dependency is used?

Ecosystem changes

  • Make hackage render sublibraries too
    • haddock
    • cabal #5949
    • hackage
@phadej
Copy link
Collaborator

@phadej phadej commented Nov 25, 2018

I mark this as high priority as 3.0 cannot be released before Add a visibility field to sublibraries is done.

ping @23Skidoo @hvr

Loading

@phadej
Copy link
Collaborator

@phadej phadej commented Nov 25, 2018

Should solver be able to flip flags based on visible field? I think it should, but I don't know if it will make implementation tricky.

I don't know a good motivation, maybe pkgA:foo is visible, only if pkgB:bar is available, and author of pkgA.

e.g. pkgA.cabal:

cabal-version: 3.0
name: pkgA
...

library
   -- available always
  build-depends: foo

flag foo
  description: Provide pkgA:foo
  manual: False
  default: True

library foo
  -- this is contrived example, but ...
  -- also it's somewhat fragile, as there isn't a complement constraint; but it should work
  if foo
    visible: True
    build-depends: pkgB:bar

 -- if not pkgB:bar, there's still some stuff to be used by pkgA

Loading

@phadej
Copy link
Collaborator

@phadej phadej commented Nov 25, 2018

we will need MIN_VERSION_pkgB:foo kind of thing. No idea how the name should be generated though. Maybe double underscore: MIN_VERSION_pkgB__foo? as there cannot be package pkgB--foo?

Loading

@phadej
Copy link
Collaborator

@phadej phadej commented Nov 25, 2018

Should visibility be recorded into InstalledInfo. Then ./Setup.hs will be able to fail if you try to "manually" use a internal library which was registered as non-visible.

Note: this change would need GHC support (as package database is ghc-pkg business), so this makes things tricky. (practically limiting multilibs for ghc-8.8+)

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Nov 26, 2018

re IPI: yeah, I came to the same conclusion but didn't know how to go at it :-/

Is it just a matter of updating the cabal submodule in the ghc repo?

Loading

@phadej
Copy link
Collaborator

@phadej phadej commented Nov 26, 2018

@fgaz essentially yes.

Loading

@23Skidoo 23Skidoo removed this from the 3.0 milestone Oct 16, 2019
@23Skidoo 23Skidoo added this to the 3.0.1.0 milestone Oct 16, 2019
@phadej
Copy link
Collaborator

@phadej phadej commented Nov 28, 2019

I'm chaging the milestone to 3.2.

@fgaz are you confident you can do this by the mid December (only two weeks!), or should we be waiting this only for 3.4?

Loading

@phadej phadej removed this from the 3.0.1.0 milestone Nov 28, 2019
@phadej phadej added this to the 3.2 milestone Nov 28, 2019
@phadej
Copy link
Collaborator

@phadej phadej commented Dec 10, 2019

Moving to 3.4

Loading

@phadej phadej removed this from the 3.2 milestone Dec 10, 2019
@phadej phadej added this to the 3.4 milestone Dec 10, 2019
m-renaud added a commit to m-renaud/cabal that referenced this issue Apr 6, 2020
The current output when pretty-printing a Dependency is to unconditionally
include :{<sublibs>}, even when sub-libs is the singleton LMainLibName. This to
begin with is an experimental syntax (haskell#5660) and issues a warning if
cabal-version is <3.0. Since 'cabal format' and soon 'cabal init' use
'Dependency -> PrettyField a' this produces contents in the .cabal file which
cannot be parsed for older config versions.

Testing:

- Verified that round-trip tests through parser/pretty pass
- Unit tests for sublibs={} and sublibs={LMainLibName}
@cartazio
Copy link
Contributor

@cartazio cartazio commented Apr 27, 2020

how can this be progressed? :)

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Apr 30, 2020

Hmm actually some of these may be done. Let me update it first

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Jun 18, 2020

@phadej thank you for all the work you've been doing on fixing those bugs, especially #6893!

Loading

@phadej
Copy link
Collaborator

@phadej phadej commented Jun 23, 2020

@fgaz is anyone working on Handle installed packages and Make the solver multilibs-aware #6039?

If not, I'll stop waiting for them for Cabal-3.4?


Also if you could check what's missing for Also update the cabal spec changelog, which I forgot, I can do that as well if there are missing points.

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Jun 29, 2020

@phadej

is anyone working on Handle installed packages and Make the solver multilibs-aware

not me, maybe @grayjay?
The solver is multilibs-aware now btw (I just checked the box to make it clearer).
The remaining subtask is not super important, because with v2- there's almost always a source package from which visibility can be determined. The only counterexample I can think of is ghc-bundled packages, but those don't have sublibraries.

what's missing for Also update the cabal spec changelog

Well, there isn't any mention of the multilibs feature, despite it being implemented Cabal-side. Would it be ok to add it to the 3.0 changelog or is that frozen and the 3.4 one should be used?

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Jun 29, 2020

The only counterexample I can think of is ghc-bundled packages

Duh, there's also distros' packages :-/
Does anyone use those in combination with cabal-install v2- though? I can only think of nix

Loading

@phadej
Copy link
Collaborator

@phadej phadej commented Jun 29, 2020

The remaining subtask is not super important, because with v2- there's almost always a source package from which visibility can be determined. The only counterexample I can think of is ghc-bundled packages, but those don't have sublibraries.

I don't agree. I have heard this argument before, and ended up fixing issues myself because someone run into them. I won't be spending any time arguing whether it's important or not. If it's not done and known issue, the feature is not ready.

The Nix(pkgs) folks how install packages into global package database will run into this issue.

Loading

@grayjay
Copy link
Collaborator

@grayjay grayjay commented Jul 6, 2020

I haven't had time to look into handling installed packages.

Loading

@phadej
Copy link
Collaborator

@phadej phadej commented Jul 6, 2020

@grayjay no worries, as long as it's still on your todo list, I'm happy :)

Loading

@phadej phadej removed this from the Considered for 3.4 milestone Jul 9, 2020
@phadej phadej added this to the Considered for 3.6 milestone Jul 9, 2020
@recursion-ninja
Copy link
Contributor

@recursion-ninja recursion-ninja commented Aug 2, 2020

I've been waiting for about 5 months to upload an updated package to Hackage which was reworked to use multiple libraries.

Is there an ETA for Hackage supporting the multiple libraries feature?

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Aug 2, 2020

Afaik the warning and lack of installed-public-sublibs support in cabal-install does not prevent hackage to implement/enable multilibs support, since hackage deals with source packages. Not sure if they have plans for this, I don't see any issue about it in their tracker and haskell/hackage-server#852 is closed

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Aug 8, 2020

#6470 is another one, Cabal-related this time :-/

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Feb 3, 2021

Handle installed packages

So I just tried to figure out exactly what problems arise from this, and my conclusion is... none? since v2- (and maybe with v1- too?), cabal-install only handles packages of which the package description is known (either from an index or local source), and there's no way to make it use a package database directly as far as I can see.
So if packages from an existing package-db are only picked up when they are known from an index => the .cabal is known => the visibility is known => no visibility errors.

The only component that can deal with sourceless/indexless package dbs, as far as I can see, is Cabal. But then the "handle installed packages" issue does not apply, because it's cabal-install-solver-specific.

Am I missing some way of using indexless package dbs with cabal-install? @grayjay @phadej

Loading

@phadej
Copy link
Collaborator

@phadej phadej commented Feb 3, 2021

So if packages from an existing package-db are only picked up when they are known from an index => the .cabal is known => the visibility is known => no visibility errors.

This is not true. Packages from existing package-db ARE picked even they aren't in any repository index. SImple example: no package repositories => you can still use base, transformers etc bundled with the GHC.

The last point is crucial for nixpkgs based usage, where nix setup populates the global package db (and it would be great if users disabled hackage.haskell.org repository so cabal-install won't try to reinstall stuff).

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Feb 3, 2021

Hmm, I must have done something wrong then. I tried to build a package with a public sublibrary with nix, then I tried to use it in another project and cabal would complain (unknown package: a (dependency of b)). When I added a file+noindex repo with a inside, it worked (without rebuilding a, so it used the one from nix).

Loading

@phadej
Copy link
Collaborator

@phadej phadej commented Feb 3, 2021

In short, public sublibraries should work with ./Setup interface only. Without any reliance on cabal-install infrastructure.

What is still unclear to me is when visibility is enforced on that level and when not.

One solution would be to not enforce visibility in ./Setup at all and make all sublibraries public by default; and then make cabal-install "the bad guy" which would impose extra constraints.

Note: ./Setup acts as a sanity check. But e.g .--exact-configuration makes it to not check compatibility.

You have to keep in mind how this would fit into stack model too. They do use --exact-configuration so ./Setup won't check bounds. Having ./Setup not check visibility would be easy solution. Though then checking of that should be implemented in stack (at some point it would be required, I guess, but there won't be hard requirement).

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Feb 3, 2021

For the last part there's already a --allow-depending-on-private-libs Cabal flag

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Feb 3, 2021

Here's a more extensive reply:

In short, public sublibraries should work with ./Setup interface only. Without any reliance on cabal-install infrastructure.

This already works since #5526 was merged, at least with --dependency / --exact-configuration.

I just tested if it works between nix packages and it doesn't, probably because they don't explicitly use --dependency, and Cabal's simple solver can't pick up sublibs on its own (I'll have to check if I have to fix this. edit: yes I do: #7270)


What is still unclear to me is when visibility is enforced on that level and when not.

With #5848, visibility is stored in the IPI and it's always enforced at the Cabal/Setup.hs level.

We have a --allow-depending-on-private-libs Cabal/Setup.hs flag to allow jailbreaking for various reasons, like extending multilibs support to old ghc versions.

So this covers Setup.hs-managed -> Setup.hs-managed dependencies.


cabal-install-managed -> cabal-install-managed dependencies work, since there we do have access to the package description


cabal-install-managed -> Cabal/Setup.hs-managed

I think this worked before #6836: all libraries were treated as public and failure happened later in the Cabal stage, which was not ideal.
After #6836: I'm not sure. It may work for ghc>8.8 or it may not work at all. I'm trying to check that, but I'm having trouble making cabal-install pick up the external package db. I already tried the --package-db flag, but it seems to have no effect:

> cabal build b --package-db ../a/dist-newstyle/packagedb/ghc-8.10.3 -v3
[...]
/nix/store/2lkbmrf9iw3v4dnf0yixhldvff5397xl-ghc-8.10.3/bin/ghc --print-global-package-db
Reading installed packages...
/nix/store/2lkbmrf9iw3v4dnf0yixhldvff5397xl-ghc-8.10.3/bin/ghc-pkg dump --global -v0
/nix/store/2lkbmrf9iw3v4dnf0yixhldvff5397xl-ghc-8.10.3/bin/ghc --print-libdir
[...]
[__4] trying: ghc-prim-0.6.1/installed-0.6.1 (dependency of base)
[__5] unknown package: a (dependency of b)

edit: --package-db seems to be one of those inactive flags inherited from Cabal... how do I reproduce this then? I can't install packages in the global package db in nixos, so I guess I need a virtual machine or container :-/

Loading

@fgaz
Copy link
Collaborator Author

@fgaz fgaz commented Feb 3, 2021

Well, it looks like installed packages are problematic in general: #7270 :-/

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants