-
Notifications
You must be signed in to change notification settings - Fork 697
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The Grinch stole all the vowels: Odd naming in .cabal/store (macOS) #7209
Comments
About a third?
No it doesn't, but things are less likely to hit the limit
Yes. See e.g. #5220
No. People work on it though, e.g. #7094
It's a necessary hack, for now. |
Cabal removes lower-case vowels from package names on macOS, see haskell/cabal#7209. Thus Agda might be just Agd in error messages reporting source locations.
* [ fix #5302 ] make 'cabal test' work for part of the test-suite 'cabal test' will run only the tests defined by 'agda-tests' * [ #5302 ] CI: work around haskell/cabal#7577 An attempt to get `cabal test` to run after *installing* Agda rather than just building it. * [ #5302 ] disable latex-tests for `cabal test` Mainly for CI, so that we do not have to install LaTeX. * [ #5302 ] disable compiler-stdlib tests for `cabal test` Do not work out of the box, need installation of the stdlib (for `Everything.agda`). * [ #5302 ] CI: `cabal test` also on macOS and Windows * [ #5302 ] `cabal test` macOS: work around haskell/cabal#7209 Cabal removes lower-case vowels from package names on macOS, see haskell/cabal#7209. Thus Agda might be just Agd in error messages reporting source locations. * [ #5302 ] giving up on `cabal test` under Windows The testsuite fails but does not display diffs. tasty-silver might not yet be portable, see phile314/tasty-silver#16 * [ #5302 ] fix the alert printed by `cabal test` Correct reference is `v1-install` in `cabal v1-install --run-tests`. * PR #5536: final polishing
@gbaz: I protest, this isn't a just a "user-question". I strongly hope this crutch will be dropped as soon as possible! It is already requiring workarounds, see agda/agda@b44a0bd. |
Sorry, I tried to read through the big diff, but I can't figure out what the workaround is there? It seems to me that the naming in cabal's store should be an internal detail that users don't have to concern themselves with. It seems like Big Sur has improved linker limitations that existed with prior mac os versions, but this is entirely an ad-hoc thing we noticed at work, not something I have any clear documentation of, and even if this is true, we can't remove the vowel stripping anytime soon. If we can confirm that it is true, and to what degree it is true (i.e. did they just bump the limit, remove it, raise it drastically?), then maybe we could start a clock at least. |
@gbaz wrote:
Sorry for the long diff (commits got squashed), the workaround is here: To normalize an error message, we replace I agree that "the naming in cabal's store should be an internal detail that users don't have to concern themselves with", but unfortunately, is leaks e.g. through error locations given by Also, as long as there is no automatic garbage collection, the user might have to go into |
easy, More seriously: if you just go and selectively delete stuff in
It's great that there are these "receipt" files! |
Congrats, @phadej! (I suppose |
@andreasabel is there some actionable here? do you think we should keep this open, tracking and blocked on a upstream os clear and coumented fix? |
Are we tracking the upstream issue somewhere? Maybe it would be good to a add a pointer here. |
This won't be fixes in macOS ever; they've set the hard limit and the only known program to me that has an issue with this is GHC, and it's almost exclusively due to GHCs own making. More recent GHCs are better at not polluting the load command space as much though, by dead-stripping unused dependencies, or those that can be recursively loaded. See https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3947 |
I'm not so sure of that. I think they may have even already bumped the linker cmd limit on the very latest "OS X" -- based on the fact that nix derivations which got to large were failing to build before, and this was solved (at our company) by upgrading to Big Sur. They don't really document this stuff though, so its all slightly guesswork. Even if this is the case, we would need to A) verify, and B) leave the hack in for at least a few more releases until dropping support for anything before Big Sur was already compelled by other considerations. |
* [ fix #5302 ] make 'cabal test' work for part of the test-suite 'cabal test' will run only the tests defined by 'agda-tests' * [ #5302 ] CI: work around haskell/cabal#7577 An attempt to get `cabal test` to run after *installing* Agda rather than just building it. * [ #5302 ] disable latex-tests for `cabal test` Mainly for CI, so that we do not have to install LaTeX. * [ #5302 ] disable compiler-stdlib tests for `cabal test` Do not work out of the box, need installation of the stdlib (for `Everything.agda`). * [ #5302 ] CI: `cabal test` also on macOS and Windows * [ #5302 ] `cabal test` macOS: work around haskell/cabal#7209 Cabal removes lower-case vowels from package names on macOS, see haskell/cabal#7209. Thus Agda might be just Agd in error messages reporting source locations. * [ #5302 ] giving up on `cabal test` under Windows The testsuite fails but does not display diffs. tasty-silver might not yet be portable, see phile314/tasty-silver#16 * [ #5302 ] fix the alert printed by `cabal test` Correct reference is `v1-install` in `cabal v1-install --run-tests`. * PR #5536: final polishing
I think I ran into this problem. I had a build fail with a typical unused dependency warning: https://github.com/tfausak/monadoc/runs/5713573255?check_suite_focus=true#step:13:17
Except of course I eventually figured it out, but this was very puzzling. https://twitter.com/taylorfausak/status/1508265860917600257 |
If you look at the dyld code, it looks like this size limitation was present with dyld2, defined as: #define MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE (32*1024) (I made that search based on the error message in this Nix issue.) However, if you look at the latest official release of dyld, that macro is not present, and that constant doesn't seem to show up either. (search query) According to the docs, there used to be some fallback to dyld2 mode previously:
but that fallback is no longer present.
|
at very least this hack:
so it should not leaked to the user while we don't remove it. Reading the hash file or whatever. Hack on top of the hack, but hacks work that way, that is the main reason we have to avoid them hard |
This would be optional. Just remind of the scope of this issue: the unique package identifiers (with hashes) also end up in the Let's keep this hack and rather organize package identifier deobfuscation schools (PIDS) and yearly tournaments where Haskell programmers compete over reconstructing the real names: (Apologies for the sarcasm. This is just too easy a target. Can't resist.) |
I would say ghc displaying user face errors with the shortened names is worse than leak them to the env file, as it is a implementation detail from the cabal-install perspective imo.
Not sure, windows package names in the store are being also shortened to workaround its max path limit ( |
checked:
and in the cabal ghc call we can see the package id with the name shortened and the hash:
but ghc somehow is able to get the full clean package name without the hash, i guess it is getting it from the package db:
so the package name in the store package.db is the shortened one in macos? i would run the above command to check it |
On macOS:
|
So the unique hypothesis i can think of it is a ghc bug and should be reported if it is not already. //cc @tfausak |
I added an unused dependency on ieee just to see what would happen. Here's the relevant output from
# ghc-pkg --package-db ~/.cabal/store/ghc-9.2.2/package.db describe ieee | head -n 5
name: ieee
version: 0.7
visibility: public
id: -0.7-c9bfa6bf
key: -0.7-c9bfa6bf |
|
Oh yes, absolutely. |
Brilliant, the best counterexample to the belief that dropping vowels would be a reasonable heuristics. |
There was an exisiting pr in ghc to improve the error message, showing the package name, version and id: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7883 🎉
For ghc-9.2.1 the output in windows (it changed from the ghc-8.10.7 one) also was the package id (with the package name shortened):
|
What’s left on this adventure of a ticket ? On the ghc side I see something has been merged, is it in any releases ? |
If I read the GHC ticket correctly, the fix is going to be in 9.4.1 and not backported. |
good news! as each ghc version has a dedicated folder in dist-newstyle and store dir we could fix it conditionally in our code |
This affects only macOS, I think.
If I look into
.cabal/store
, I see:Seriously! The Grinch stole all the vowels.
I cannot help feeling trolled by cabal v2. (Which I am still trying to learn.)
#5399 (comment)
Apparently, removing the vowels should help the Mach-O linker with its space limitations.
https://github.com/typedrat/cabal/blob/638c78e790721566780dbf75a2e6f884ceafa970/cabal-install/Distribution/Client/PackageHash.hs#L137-L145
I feels I am on the C64 again. Sure this isn't an Easter egg?
Questions:
Odd.
The text was updated successfully, but these errors were encountered: