Cannot install a profiled library with cabal install --lib -p pkgname

[goldfirere]

Use case
I would like to experiment with performance characteristics of a few library functions. I am not building software to ship. I am just trying to learn more about Haskell. So I'd like to be able, say, to build an executable that depends on a few packages without needing to edit a .cabal file every time I change the set of dependencies. (Remember: the goal is quick experimentation!) A key requirement is that I need profiled libraries. The first one I wish to use is random. Read on to see what happens:

Describe the bug
I say cabal install --lib -p random. Cabal responds with Resolving dependencies... Up to date. Sounds encouraging. I can confirm that ghc Experiment.hs (which has import System.Random) works. But I want to say ghc Experiment.hs -prof. That does not work, with GHC saying I need profiled libraries. Note that I asked for profiled libraries with -p, above. I think: maybe my environment file has a stale entry. So I edit my env file to delete the line with rndm. I then cabal install --lib -p random again. The env file re-grows its rndm line... but the library still isn't profiled. I'm now pretty stuck.

Being stuck, but still eager to make progress, I try cabal v1-install -p random, remembering that such a command used to work brilliantly. After doing some building, I now get

Error while Error while updating world-file. : dieVerbatim: user error (cabal: Could not parse world file.
)

So that didn't work. I'm now completely stuck. I decide to do something other than learn more about Haskell for now.

To Reproduce
Included in the narrative above.

Expected behavior
I expected the first cabal install --lib -p random to update my env file with the correct entry to be able to do a profiled build depending on System.Random.

System information

• Operating system: macOS 11.6
• Cabal-3.6.2.0, cabal-install-3.6.2.0 ,ghc-8.10.7

[gbaz]

looks like a dup of #5982 and #7297 (and related to #7236)

Ideally this will all be resolved as part of revamping cabal v2-install functionality entirely as per #6481

(also note that this describes in the discussion that you can just delete the world file from your cabal directory if it was somehow created and is corrupt)

[goldfirere]

I understood #5982, #7297, and #7236 as being about executables, not libraries -- so I did not look closely at them (though I did find them in my searches before posting). Glad to see that there appears to be action at #6481 -- I'm eager for that feature.

For the record, the world file seems just fine -- I don't see how that could be the issue here. It says random -any.

If this ticket really is a dup of others, please close. I suppose the workaround is to create a .cabal and cabal.project file.

[fgaz]

Just the cabal.project should be enough iirc

[goldfirere]

That surprises me. I can have a cabal.project without a .cabal file? How would that work?

[fgaz]

It'd be a project with no local packages

[gbaz]

They were started about executables but the underlying issue is the same in all cases as far as we know from our sleuthing. Vis a vis world files, I'm just suggesting that if v1-anything is failing due to a world file error, its safe to delete the file, its vestigal and the entire half-implemented-never-working feature is removed in the next cabal release.

[ulysses4ever]

@fgaz I tried your suggestion about cabal.project without a local package (i.e. w/o <pkgname>.cabal), and it didn't work.

I have cabal.project with just profiling: true and I'm doing calling cabal-install ( 3.6.2.0):

cabal install -p --only-dependencies --lib --package-env=.  <space-separated list of packages>

(-p and --only-dependencies can be omitted with no change in the result)

cabal-install complains:

cabal: There is no <pkgname>.cabal package file or cabal.project file. To
build packages locally you need at minimum a <pkgname>.cabal file. You can use
'cabal init' to create one.

For non-trivial projects you will also want a cabal.project file in the root
directory of your project. This file lists the packages in your project and
all other build configuration. See the Cabal user guide for full details.

This is a strange message: it's either a logic bug (because I definitely do have cabal.project) or the message means and where it says or, and it's a documentation bug then.

I also tried to rename cabal.project into cabal.project.local, in that case I get another error message:

Warning: both implicit and explicit configuration is present.When using configuration(s) from <path to current directory>/cabal.project.local, the following errors occurred:
The package location glob './*.cabal' does not match any files or directories.

Any thoughts on this or how to get profiling libs without <pkgname>.cabal in general?

[phadej]

How much time is really "wasted" if you'd rather write a proper .cabal file (with cabal init or manually). Over few past years improvements to the rough edges in non-package/non-project cabal usage haven't made experience much smoother.

I.e. what can be done better or differently so people would just make ad-hoc packages&projects, instead of trying to use cabal-install as a "more clever" ghc-pkg (which cabal-install is not). IMHO that's just a echo of the past, if having a package setup would been always required, people would just do that. (And that's is my exprerience from other "mainstream" language ecosystems: use a package skeleton, even you don't intend distributing&publishing it).

[fgaz]

To answer @ulysses4ever question, you can work around that error by putting those packages (or even just a random package, like base) in extra-packages. Kinda ugly but it works. Note that due to #7297 (comment) a global profiling: True probably won't work and you'll have to apply it to every package explicitly (see the rest of #7297):

package *
  profiling: True

I do somewhat agree with @phadej though

[fgaz]

This is a strange message: it's either a logic bug (because I definitely do have cabal.project) or the message means and where it says or, and it's a documentation bug then.

Hmm, you're right... I think it should work so imo it's a bug. I've used this capability plenty of times with source-repository-package, and in that case it does work.

[goldfirere]

I.e. what can be done better or differently so people would just make ad-hoc packages&projects, instead of trying to use cabal-install as a "more clever" ghc-pkg (which cabal-install is not). IMHO that's just a echo of the past, if having a package setup would been always required, people would just do that. (And that's is my exprerience from other "mainstream" language ecosystems: use a package skeleton, even you don't intend distributing&publishing it).

I think this is a great question. Perhaps you're right that some of us (myself definitely included) are simply pre-conditioned to not want to make a project file. Here are some things that make writing a .cabal file annoying for me:

• I often don't know the dependencies of a throwaway Haskell file beforehand. It's already annoying to have to add (say) import Data.Text at the top of my file, but I see no way of avoiding this. Yet to have to also add a text dependency to a cabal file, figure out the right version bounds, and then restart HLS (which, I believe, does not react by default when a .cabal file is updated) is even more annoying.
• The annoyance is compounded manifold when I am doing a public demonstration -- such as in a classroom setting (a very realistic scenario from my recent past).
• .cabal files expose the wrong sorts of details for throwaway Haskell files: there's no executable, there's no exposed modules, and there's no testsuite. So in practice I would have to claim there's a library with some set of exposed modules, even when that's not really what I want.
• .cabal files are in want of a license file (among a few other requirements, I believe), not appropriate for the sort of situation I'm imagining.
• I want to be able to be very deliberate in how I invoke GHC, passing just certain flags, etc. That is, saying cabal repl to load my file is not the right answer, just in case I want to pass e.g. -ddump-tc-trace -dsuppress-var-kinds -fprint-explicit-kinds or something. I do not want to have to edit the .cabal file to get those flags to GHC. And it's annoying to have to say --ghc-options="..." or similar. GHC has its own interface: I want to use it.

Actually, those bullets aren't an answer to your question of "what would have to be different?". I suppose, for me, one way to move forward would be to allow some ~/.cabal/default.cabal file (say) that specifies default settings I wish to use for building. This could include a list of installed libraries, a default language, and even default extensions. Running cabal configure (or something) outside of any project could make sure that those libraries are available. It could produce a configuration file ~/.ghc/default-config (or something). Then, we could add a flag to GHC --config=... that loads the configuration stored in the file provided; saying --config (without the =) would use ~/.ghc/default-config. Or, somewhat equivalently, add two new top-level commands to cabal: cabal ghc and cabal ghci. These work by invoking ghc (respectively ghci) with the remainder of the arguments on the command-line, exactly as they are. The only difference between cabal ghc and ghc would be that the former would load the default config file (or, perhaps, the local project configuration if you're within a project directory).

In short: I, personally, am willing to maintain some kind of global .cabal-like file that specifies what dependencies I want generally available, and then contriving for a way to get GHC to see those settings. That would seem to fit in with the current cabal mindset, while solving some of the problems I've seen personally (and I've seen others echo).

[phadej]

, figure out the right version bounds,

.cabal files are in want of a license file (among a few other requirements, I believe), not appropriate for the sort of situation I'm imagining.

That is incorrect. Minimal .cabal file to get stuff going is no longer than:

cabal-version: 3.4
name: foo
version: 0

library
  build-depends: base, text, containers, deepseq, another-dependency
  exposed-modules: Foo

(Thing about exposed-modules as an argument which you would need to pass to GHC either way).

You should write the rest of the details if you want to publish the package, as you probably should to clean up the Haskell code as well.

I want to be able to be very deliberate in how I invoke GHC,

Add

{-# OPTIONS_GHC -ddump-tc-trace -dsuppress-var-kinds -fprint-explicit-kinds #-}

to the top your module.

[goldfirere]

The workflows you suggest still require me, for every directory I'm working in, to maintain separate .cabal files. When I add a dependency to each one, I'm sure some time will be spent building the libraries. This would not work in a teaching setting. And, in my GHC work, I frequently call GHC with different flags serially, in order to get different outputs, so editing the source file would be annoying. Maybe that's just me, and I'm too small a use case.

You asked what I would need in order to be more satisfied in my use of cabal. I answered, and you offered workarounds. I've been aware of workarounds to my problems for years, but they've never been satisfying. They still aren't.

[phadej]

The workflows you suggest still require me, for every directory ...'

What you mean by "maintain"? I had impression that you want to try something out, and then "forget" about that code. But if you expect to come back to these experiments days, weeks or months later, then writing .cabal file is a good time investment, isn't it? Dependencies listed which you used (even GHC flags in haskell or .cabal file), not all the libs you accumulated in your global environment (and maybe removed some if you had to start over).

I'm sure some time will be spent building the libraries.

Maybe, but somewhat unlikely: that's the "magic" of v2-build, the dependencies are in the global cache most of the time (especially when unrestricted and you don't cabal update often.

[ulysses4ever]

Two points here. I have been a happy user of (although mouthful) cabal install --lib --package-env=. for a while now, I really like it. It's very sad if I have to create two extra files when I want to get a profiling build (I am not quite sure if I can avoid cabal.project with profiling: true and use a command-line flag instead: I seem to remember that it's not possible but I may be wrong).

Second, to your question @phadej, I'd ask another question: do you see it reasonable to ask cabal-install to make the version and cabal-version into optional fields with some sensible defaults like 0 (for users like me or Richard, I suppose, any default would be sensible probably). Maybe name too? (Could be defaulted to the .cabal-file name, if it has to be non-empty). That would cut the bloat in half and make me much more susceptible to creating a package. Also, it would be nice to have a non-intrusive way to mute warnings about missing fields that cabal thinks are "really important" (if there are such warnings nowadays).

Some other random issues I have with cabal-install. My use cases are similar to Richard's but more often than not I need an executable that I can run and sometimes benchmark (e.g. with hyperfine). Using cabal run is not compatible with benchmarking: even when it doesn't suddenly decide to rebuild (and sometimes it does for no good reason I think) it still slowdowns things and pollutes output (unless I pass -v0? but I don't want to bother). Also, the executable is buried somewhere deep-deep under dist-newstyle -- this is very annoying. I think there's an issue to make it easier to get to the executable, but so far we have what we have. For example, Nix's feature with result in the current directly is rather ugly but super handy for my use cases.

[tomjaguarpaw]

I often don't know the dependencies of a throwaway Haskell file beforehand

My suggestion for this has always been, let there be a cabal configuration option to expose some maximal set of compatible packages. It probably doesn't particularly matter which one; good heuristics would be the biggest such set, or the set containing the most recently installed package.

This hypothetical option would strike a great balance between the convenience of cabal v1 and the composability of cabal v2.

[gbaz]

Calculating a maximal set of compiled packages is worst case exponential I think

(I mean, solving alone is np-complete, but this is arguably exp-worst or so on top of that)

[gbaz]

by the way, cabal list-bin lets you get the path to generated executables.

[ulysses4ever]

@gbaz thanks a lot! It's nowhere to find in the docs, so I opened #7945. It's a pity it has no documentation (#7774).

[tomjaguarpaw]

Calculating a maximal set of compiled packages is worst case exponential I think

(I mean, solving alone is np-complete, but this is arguably exp-worst or so on top of that)

Could be, but I can't see what's wrong with this algorithm sketch which seems to be polynomial:

1. Initialise the set of packages S = {}
2. Pick an arbitrary package, P, not in S: polynomial
3. Form the set of P and all P's transitive dependencies. Call it PD: polynomial
4. Check whether anything in PD conflicts with anything in S: polynomial
   • If there is a conflict, discard PD
   • If there is no conflict, add PD to S
5. Go to 2

It looks polynomial as I have presented it. Perhaps I have missed an important ingredient.

Practically speaking, I think that choosing P to be the most recently installed package not in S would be a good heuristic.

[tomjaguarpaw]

Perhaps "maximal" is ambiguous. I don't mean a consistent set of packages of largest size. I mean a consistent set of packages to which no other package can be consistently added. I agree the former has an exponential smell to it.

[ulysses4ever]

@tomjaguarpaw while it seems polynomial to me too, it does not guarantee that you end up with a substantial portion of Hackage (or even Stackage) because of 4.a ("if ... discard PD").

Maybe with a cleverly picked seed value of S ("roots" of the search), instead of {}, this may yield reasonable results (or maybe not: one'd have to try).

I'm sympathetic to this idea. In fact, several month ago @emilypi mentioned somewhere that cabal.freeze files could effectively model stack resolvers and the only thing that is missed is infrastructure to easily get these files on the user side: someone have to make effort 1) converting stack snapshots into cabal.freeze files; 2) publish them somewhere; 3) (perhaps hardest) figure out a seamless way to deliver them to the user (ideally, a cabal-command, I think). That's predicated on willingness to rely on stack of course. Which sounds very reasonable to me: someone is already working hard to deliver something close to what you want, so why reinvent the wheel. In fact, both ways can co-exist, lts-based freezes and solver-based ones, which would use whatever algorithm. Just development and testing of the algorithm could happen in parallel to the more boring lts-based approach. My worry is that the other steps to achieve the ideal state also require bits of volunteer work.

[fgaz]

1. converting stack snapshots into cabal.freeze files; 2) publish them somewhere;

Those are already public, though hidden: commercialhaskell/stackage-server#232

[tomjaguarpaw]

it does not guarantee that you end up with a substantial portion of Hackage

My guess is that the algorithm, combined with the heuristic of always choosing as P the most recently installed package not in S, ends up with a superset of the packages that would be available if the user had instead been using the v1 commands (perhaps exactly the same set of packages). Since @rae seems implicitly to be asking for v1-style behaviour this algorithm seems like a good way of satisfying him whilst staying within the more principled v2 world.

[gbaz]

I worry that as a heuristic this may be usually ok, but it will at times be utterly confusing. Suppose I have a setup I'm fairly happy with, then I go off and try out some old piece of code that still compiles, but has a lot of strict upper bounds, which in turn pulls in other older stuff, etc. Now, all of a sudden the "most recent package" has a generally far less desirable universe, and the workflow that was nice for me yesterday when just playing around all of a sudden has broken, because the heuristic now is picking out a very different set of packages.

As this discussion has gone pretty astray, I should underline, we do have on our roadmap fixing the actual issue highlighted here, and more generally reworking the ergonomics and UI of cabal install to make managing environments actually straightforward for users. Further, we do think the use case Richard has outlined of a "nice universe for a playground" is an important one.

However, experience has repeatedly shown that "do what I mean" algos in package management lead to frustrated and confused users, and while things should be as easy as possible, they should also be no easier than that, and finding that balance is not going to be simple.

[goldfirere]

Further, we do think the use case Richard has outlined of a "nice universe for a playground" is an important one.

Great -- glad to hear it. Thank you.

However, experience has repeatedly shown that "do what I mean" algos in package management lead to frustrated and confused users, and while things should be as easy as possible, they should also be no easier than that, and finding that balance is not going to be simple.

I agree here, and have not thought about the specific of the algorithms above. But, there is an algorithm that is known to (mostly) work: what v1 did. That is, the universe expands only by user request, when that user asks to add a new package to their universe. At that point, cabal selects the most recent release of that package (or whatever version the user asked for). Then, cabal uses the same algorithm it usually uses to decide its dependencies. This worked for the lifetime of v1. But it only mostly worked, because of cabal-hell: that's why v2 exists! So there would be a potential pitfall in that users try to build an inconsistent universe, or they build a consistent universe that doesn't serve their actual needs. I'm not sure those pitfalls are avoidable... but maybe they could be mitigated with cabal destroy-universe which blows away the current universe, allowing the user to re-expand it from scratch. (In my own practice, a universe went stale at about the same rate as GHC released, and so I just moved to a new GHC when things got rough in the universe.)

[jneira]

Well there had been really interesting discussions and thoughts here, but the original report is essentially #7297, so maybe we should rename it or close it as duplicate

[goldfirere]

This doesn't strike me as exactly #7297: this bug is about cabal install --lib -p, while #7297 is about cabal run. Maybe the two tickets share a solution, but I do think they're asking for different things (and should have e.g. different test cases in the testsuite). That said, this isn't my project, and so if closing this in favor of the other ticket makes sense (and will get us marginally closer to actually allowing me to profile my experiment), then go for it.

[GeorgeCo]

@gbaz , you wrote

we do have on our roadmap fixing the actual issue highlighted here, and more generally reworking the ergonomics and UI of cabal install to make managing environments actually straightforward for users. Further, we do think the use case Richard has outlined of a "nice universe for a playground" is an important one.

That is great! Do you have a time or an eta for the preceding? Thanks!

[gbaz]

The underlying issue is discussed in #7297 which has been in the list of desired foxes for a few releases, but it needs someone to want to work on it. Essentially cabal install doesn't turn global flags into project-local flags in the course of installing a remote tarball.

[alt-romes]

This should also be fixed by #9697.

[alt-romes]

Fixed by #9697