Support optionally-exposed modules #112

Closed
wants to merge 1 commit into
from

Projects

None yet

7 participants

@acowley
acowley commented Jul 6, 2016

This is a mechanism to eliminate the appeal of defining orphan instances
in Haskell packages.

The problem: If I define a new type class for which instances may be
defined for types defined in other packages, I must either depend on
those packages or ask that those packages' authors depend on my
package.

The solution: Allow a library author to identify individual modules that
should only be built if a necessary dependency is included in a build in
which the library is involved.

Details:

A new optionally-exposed section of a library stanza supports the
specification of potentially exposed modules and their dependencies. The
inclusion of these modules and dependencies in the generated Cabal file
depends upon the availability of a build plan.

A build plan in this context is simply a list of package names and
versions as output by, for example, stack list-dependencies.

If no build plan is supplied, then all "optionally-exposed" modules are
exposed, and their dependencies must be available.

If a build plan is supplied, then only those "optionally-exposed"
modules whose dependencies are satisfied by the build plan are included
in the Cabal file generated for the package.

Usage by build tools:

A tool like stack can invoke hpack --buildplan min for each listed
dependency when computing an initial build plan. Then it can call
hpack for each package again, this time passing the name of a file
containing the computed build plan. This second round will not add any
new packages to the build plan, but may add dependencies to the
individual packages included in the build plan.

Usage by library authors:

If you wish to provide an instance for, say, Data.Text.Text, but do
not otherwise use the text package, place the instance definition in
its own module, say, src/Foo/Text.hs. Then add an
optionally-exposed section to the library definition so it looks
something like this,

library:
  source-dirs: src
  optionally-exposed:
    dependencies: text > 1.2
    modules: Foo.Text

Now, if somebody uses your library without themselves depending on
text, they may do so without incurring a dependency on text. If they
do depend on text, then they will have access to the instance you
defined.

@acowley
acowley commented Jul 6, 2016

Ping @mgsloan

This is something a bit out of left field, but I wanted to see if it actually works, and it seems like it does. There may be issues with the overall design, so it needs some thought.

For a quick overview, check out the spec

@Peaker
Peaker commented Jul 6, 2016

Awesome!

Maybe "conditionally-exposed" instead of optionally?

@acowley
acowley commented Jul 6, 2016

I'm open to that. I avoided the word "conditional" since it is a distinct hpack concept.

@Peaker
Peaker commented Jul 6, 2016

"expose-if", "expose-when" or "try-exposing"?

On Wed, Jul 6, 2016, 21:59 Anthony Cowley notifications@github.com wrote:

I'm open to that. I avoided the word "conditional" since it is a distinct
hpack concept.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#112 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAC_A-cpj0-bPEpr3yrgQI4t0Y8DuMLIks5qS_sAgaJpZM4JF2Vq
.

@bitemyapp

expose-with since it's about sets/pairs of packages?

@acowley acowley Support optionally-exposed modules
This is a mechanism to eliminate the appeal of defining orphan instances
in Haskell packages.

The problem: If I define a new type class for which instances may be
defined for types defined in other packages, I must either depend on
those packages or ask that those packages' authors depend on my
package.

The solution: Allow a library author to identify individual modules that
should only be built if a necessary dependency is included in a build in
which the library is involved.

Details:

A new `optionally-exposed` section of a library stanza supports the
specification of potentially exposed modules and their dependencies. The
inclusion of these modules and dependencies in the generated Cabal file
depends upon the availability of a build plan.

A build plan in this context is simply a list of package names and
versions as output by, for example, `stack list-dependencies`.

If no build plan is supplied, then all "optionally-exposed" modules are
exposed, and their dependencies must be available.

If a build plan is supplied, then only those "optionally-exposed"
modules whose dependencies are satisfied by the build plan are included
in the Cabal file generated for the package.

Usage by build tools:

A tool like stack can invoke `hpack --buildplan min` for each listed
dependency when computing an initial build plan. Then it can call
`hpack` for each package again, this time passing the name of a file
containing the computed build plan. This second round will not add any
new packages to the build plan, but may add dependencies to the
individual packages included in the build plan.

Usage by library authors:

If you wish to provide an instance for, say, `Data.Text.Text`, but do
not otherwise use the `text` package, place the instance definition in
its own module, say, `src/Foo/Text.hs`. Then add an
`optionally-exposed` section to the `library` definition so it looks
something like this,

```
library:
  source-dirs: src
  optionally-exposed:
    dependencies: text > 1.2
    modules: Foo.Text
```

Now, if somebody uses your library without themselves depending on
`text`, they may do so without incurring a dependency on `text`. If they
do depend on `text`, then they will have access to the instance you
defined.
63a2e6a
@Blaisorblade

@acowley This looks very interesting, but it does raise a few questions on the design.
Caveat: I've gotten here from Twitter, skimmed the TODO and looked at hpack's slides, but I probably still don't understand the full workflow, both currently and in your proposal, especially regarding hpack packages depending on other hpack packages.

I also now think the closest thing to a design document might be the discussion on Reddit, though I haven't yet found all the info I'd like: https://www.reddit.com/r/haskell/comments/4pf86b/cabal_build_variants/d4kqfgi

  1. I suspect some of the issues would be addressed (technically speaking) inside Cabal if Cabal had a flag variant with different solver behavior (in principle, the two-step process you describe). Changing Cabal would certainly take longer, but if you don't change cabal it seems you need to change Hackage for a complete solution or use hacks (see point 2).
  2. Nowadays hpack seems to produce one fixed cabal file that can be used for uploading to Hackage (sensei even commits the generated file to the source repo). Your proposal changes that, so I'm not sure how to upload a conditional package to Hackage — do we add the package.yaml and hack tools to ignore the cabal file and actually use package.yaml? Or do we extend the whole pipeline to support hpack without hacks? Cabal flags would address that.
  3. What happens, in your scenario, if I later add text and need it and the "glue code" (src/Foo/Text.hs) together with your library in the same repository/sandbox? I imagine you either cannot, or that package names must be mangled. Cabal flags would address that.
@acowley
acowley commented Jul 7, 2016

@Blaisorblade This is intended to be used by stack which knows how to generate cabal files using hpack; this addresses your first two points, I think. For the third, if you change the hpack file, a new cabal file is automatically generated, and it would be different from the one in whatever cache is being used so would trigger invalidation.

@Blaisorblade
Blaisorblade commented Jul 7, 2016 edited

Ah, didn't know about the hpack support in stack, and the plans (mentioned in commercialhaskell/stack#1773) to add hpack support to Hackage!(EDIT: @sjakobi pointed out my misunderstanding).

For the third, if you change the hpack file, a new cabal file is automatically generated, and it would be different from the one in whatever cache is being used so would trigger invalidation.

Oh I see — invalidation would then rebuild the affected package and the packages depending on it, guaranteeing correctness. I can imagine that implementing invalidation for this scenario is tricky in some cases (especially invalidating a global package used in local snapshots), but I'll just trust stack to get this right.

@sjakobi
Contributor
sjakobi commented Jul 7, 2016

Ah, didn't know about the hpack support in stack, and the plans (mentioned in commercialhaskell/stack#1773) to add hpack support to Hackage!

I think you're misreading the comments in commercialhaskell/stack#1773. I believe the comments regarding Hackage are only about publishing a new version of hpack to Hackage, not about changing Hackage.

@Blaisorblade

@sjakobi Thanks!
Having a default cabal file on Hackage seems a bit hacky and makes me uneasy. OTOH it guarantees support for other existing tools; and since it's generated without a build plan (I assume) and because of what this PR does in that scenario, it keeps the current behavior of including all dependencies and all code, which is fine for backward-compatibility.

I'm coming to like this design more and more.

@23Skidoo

Re: interaction with Cabal, historically changing a library's API based on which flags are enabled has been frowned upon. Quoting @dcoutts:

The reason for this is that if this were allowed (and it were used) then it would become impossible to translate cabal packages into binary packages for systems like debian / fedora etc. Many people felt, and still feel that being able to translate into binary package systems is an important feature, and thus this restriction remains.

OTOH, as @ezyang points out in that discussion, it may make sense to allow using flags to expose extra functionality. After all, a lot of software that comes with Debian can be configured in this way (example).

@Blaisorblade

OTOH, as @ezyang points out in that discussion, it may make sense to allow using flags to expose extra functionality. After all, a lot of software that comes with Debian can be configured in this way (example).

@23Skidoo That's correct. Critically, in our scenario we're only subsetting the API, not having incompatible variants, so a distributor can always just add all dependencies if he wants to be safe, he just gets extra options by just omitting packages you don't want. Apparently some distributions already disable some optional dependencies by abusing flags, and @kmett had to add such flags to avoid worse.

Also, in the current vision, adding a package would automatically add the needed instances —
#112 (comment). (Not sure how exactly it works and whether it would be plausible with cabal-install).

Turns out we're discussing pretty much what @ezyang mentions here, though he seems to propose a different approach:

I've recently come around to the idea that maybe we should use flags to expose extra functionality (like instances, optional bonuses when people have more dependencies). This would help package authors address a very big issue, which is "how many dependencies should I put in the core package, as opposed to split out into separate packages." The issue is even more keenly felt when orphans are involved.
[...]

@ezyang
ezyang commented Jul 11, 2016

I have a bias against this sort of compile-time Cabal file rewriting, but let me try to comment on this proposal on its merits.

The downside of using this feature is that it interferes with Stack's ability to cache packages across workspaces. Packages that have optional dependencies must be built "in place" in the same way you would build vendored packages with local modifications. This thwarts normal mechanisms for sharing, and for packages which are deep in the dependency tree, this could be very painful—like, cabal sandbox levels of painful!

Of course, the way you fix this is make it so that there is some way to uniquely identify each of these separate Cabal files; in my proposal, the flag selection would identify this (no need to rewrite the Cabal file)—alternately, if you don't want to wait for Cabal to take whatever changes necessary to handle this, you could ask Stack to hash the Cabal file and cache accordingly.

I have a slight suggestion for the Cabal file that would be uploaded to Hackage: instead of uploading a Cabal file which has all or none of the optional dependencies enabled, how about putting all of the optional dependencies within flag conditionals with a special field, e.g.,

flag text-instances
  if-building: text

(You can bikeshed if-building to whatever you want.) cabal-install's solver today will default the flags to true (which is what you want), but in the future the solver could be extended to understand these types of flags and turn them off when they are not needed by the build-plan.

P.S. There is one other minor difficulty with this proposal, which is that is that it can only ever support adding the optional section if ALL of the dependencies are satisfied. In particular, you can't say, "If p is in the build-plan, then I want to generate a Cabal file with an extra dependency q." Then you are going to have to ping-pong with Stack, where Stack says "here's my build plan" and hpack says "oh wait, here is an extra dependency I need now" and then Stack says "here's my build plan"... and so forth. But I can't think of a situation where you'd actually want this.

@Blaisorblade

The downside of using this feature is that it interferes with Stack's ability to cache packages across workspaces. Packages that have optional dependencies must be built "in place" in the same way you would build vendored packages with local modifications.

I also wondered. I think @acowley might have implied otherwise in #112 (comment), but I'm not 100% sure how that would work and I might be misunderstanding him; IIUC not the whole vision is implemented. Also, I'd guess even in the best case lots of pain is involved, even if packages are not built in place.

Which is why I had argued using flags could work better; your idea of generating cabal files with flags appears nice.

I wrote:

What happens, in your scenario, if I later add text and need it and the "glue code" (src/Foo/Text.hs) together with your library in the same repository/sandbox? I imagine you either cannot, or that package names must be mangled. Cabal flags would address that.

@acowley answered:

if you change the hpack file, a new cabal file is automatically generated, and it would be different from the one in whatever cache is being used so would trigger invalidation.

Let me refine my scenario, because I'm not sure I understood your answer, and if @ezyang imagines right I'm concerned.

Say my hpack-developed package depend on foo, which is also developed with hpack, and foo has optional instances for text. Now say my package does not depend on text, then it is built, then I add a dependency on text. It's clear that the cabal file for my package gets invalidated. But what happens with text? Do we need to add dependency tracking to regenerate all cabal files for all packages with optional dependencies on text? Do we invalidate foo even if it's deep in the dependency chain? Do we eagerly or lazily invalidate and rebuild all dependencies? Can foo be shared across workspaces? Do we eagerly invalidate packages in other workspaces? (Answer to the last one: I'd say certainly not, we can do that when using those other workspaces).

The same questions apply to using flags, but with flags I can imagine more easily foo[-text-glue-code] and foo[+text-glue-code] (that is, foo with or without the text-glue-code flag) coexisting. Still, at best we get a lazy invalidation of sorts.

I'm not using text-instances as name because glue code might be more than instances I'd expect. I don't want to bikeshed the name, but point out that "only instances" is nowhere actually written down or enforced, and not sure it's needed—this might warrant discussion.

@acowley
acowley commented Jul 12, 2016

The details of stack support is out of scope for this PR. It amounts to an easier problem than dealing with multiple flag assignments for the same package since the differences are apparent in concrete cabal files. No new "serialization of settings" format need be invented.

The ping-pong criticism is not possible with this proposal, but may fall into the pre-existing biases bucket. I don't know.

@Blaisorblade
Blaisorblade commented Jul 12, 2016 edited

The details of stack support is out of scope for this PR

I'm discussing those details because those details might affect the design of this PR.
But we can indeed defer that, because I realized the "worst-case redesign" is not a problem — if you're curious or @ezyang, see below for why. So 👍 from me on that front. I still guess generating flags by default as @ezyang proposes won't hurt (except maybe warnings), but anybody interested can always change that later somewhat transparently.

Worst-case redesign

Even if

  • this PR is merged as-is
  • people start using the new hpack features
  • later one decides to change design and generate Cabal flags to solve specially (whoever solves these flags)

there's no problem, since one can keep the same hpack syntax.

At worst, Cabal might gain the same feature on its own, making hpack's use arguably somewhat redundant (and not really, since it would still have to expose the feature somehow). But that will take quite a while, if it ever happens, so this shouldn't be an argument against this PR.

@acowley
acowley commented Jul 12, 2016

@Blaisorblade You're absolutely right, I'm sure there is room for improvement, but my intention is that if you can use this feature manually immediately, which you can, then we don't need to couple hpack to stack too tightly.

The simplest support would be for stack to simply install the package in a local overlay as with any ad hoc patching scenario. More complex support has a variety of prior art to lean on, but there are so many degrees of freedom there that I'd rather the stack developers pick a solution they like than have it nailed down a priori by the package description format.

@acowley
acowley commented Dec 4, 2016

Closing for lack of interest.

@acowley acowley closed this Dec 4, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment