cabal run fails on windows due to the long path issue #8841

ulysses4ever · 2023-03-12T16:51:21Z

Describe the bug
Copied from IRC:

maerwald:
windows cabal test consistently fails: https://github.com/haskell/ghcup-metadata/actions/runs/4396989936/jobs/7699817403#step:3:14665

the same test works with 3.6.2.0: https://github.com/haskell/ghcup-metadata/actions/runs/4397003656/jobs/7699838528

D:\a_temp\foobarbaz\ghcup\ghc\9.4.4\lib../mingw/bin/llvm-ar.exe: error: C:\cabal\script-builds\ff17c4bce8853274ef8c5beb95175084fb96425f714475e54f50bdf49f3b33c6\dist-newstyle\build\x86_64-windows\ghc-9.4.4\fake-package-0\x\cabal-script-main.hs\build\cabal-script-main.hs\cabal-script-main.hs-tmp\Main.p_o.tmp: No such file or directory

yes, there's a regression with long paths on windows

Artem:
any way to check if it works with 3.8.1.0? or how to repro it in general? 'Cause it seems like it cabal runs print $ 1+1, which 3.10 is definitely capable of (even on Windows!). Do I just put the script in a folder with a long path (like above)?

maerwald:
just make sure you don't have CABAL_DIR set and it should reproduce

To Reproduce

No idea.

System information

Operating system: Windows
cabal, ghc versions: 3.10.1.0, 9.4.4

The text was updated successfully, but these errors were encountered:

gbaz · 2023-03-12T17:36:17Z

Note that @hasufell considers this a blocker to making cabal-install 3.10.1.0 a recommended release

jneira · 2023-03-12T20:51:24Z

Thanks for the report, to delimite the scope of the bug:

it seems to affect only cabal run, at least not cabal build in my tests
as commented setting CABAL_DIR to a short path workarounds the issue. Another one could be the usual one: enable long paths on windows (i've not tested it though)
it likely was introduced by the cabal run/script improvements in cabal-3.8 so it is not a regression only for 3.10:

PS D:\dev\ws\haskell\cabal-test> cabal-3.8.1.0.exe .\script.hs
realgcc.exe: error: C:\Users\atrey\AppData\Roaming\cabal\script-builds\8451f75c9bb9fd520bd934aea714d2e240d258f1e884c03cde6784c44c03d9ce\dist-newstyle\build\x86_64-windows\ghc-8.10.7\fake-package-0\x\cabal-script-script.hs\build\cabal-script-script.hs\cabal-script-script.hs-tmp\Main.o: No such file or directory
`gcc.exe' failed in phase `Linker'. (Exit code: 1)
PS D:\dev\ws\haskell\cabal-test>

ulysses4ever · 2023-03-12T21:59:17Z

Very interesting and useful, thank you, @jneira !

For the future: am I understanding correctly that it's impossible to test in our test suite because we have CABAL_DIR set.

jneira · 2023-03-12T22:21:11Z

Very interesting and useful, thank you, @jneira !

For the future: am I understanding correctly that it's impossible to test in our test suite because we have CABAL_DIR set.

To be precise the fact which causes the bug is to set CABAL_DIR to a enough long path. It turns out that the default one is long enough to do it.

So setting CABAL_DIR to a long enough path should reproduce the bug. It could be done globally or specifically for a dedicated test

ulysses4ever · 2023-03-13T12:23:49Z

it likely was introduced by the cabal run/script improvements in cabal-3.8

Maybe page in @bacchanalia in that case

RyanGlScott · 2023-03-13T13:50:15Z

Just to check, is this reproducible with all GHC releases, or just GHC 9.4? I ask since LLVM 14 (the version of LLVM that ships with GHC 9.4 on Windows) suffers from a long-path–related issue on Windows. I believe older GHCs (which ship with GCC rather than LLVM) are unaffected, and GHC 9.6.1 ships with LLVM 15, which has fixed the aforementioned issue.

Mikolaj · 2023-03-17T10:27:44Z

Could somebody please verify if this affects only GHC 9.4.4 or any other GHC as well?

@bgamari: I wonder if there's a possibility that GHC 9.4.5 fixes the long path issue on Windows (see the previous comment)?.

@bacchanalia: given your recollection of the code of this feature, is there any way to work around the problem? Make a temporary short link just before executing the script? Shorten or eliminate the hash? Remove all the "cabal-script" prefixes to fake package names?

Mikolaj · 2023-03-17T13:55:53Z

Good news: the GHC fix is on the horizon: https://gitlab.haskell.org/ghc/ghc/-/issues/21293 [Edit: actually, that's the same ticket @RyanGlScott has linked to, but now it has one more user group advocating for it's backport to 9.4.5 and Ben has kindly written this down in the ticket and made sure it's marked for backport.]

Still, long paths have been plaguing the ecosystem for long, for various reasons, so if we can make cabal less fragile with respect to this issue, that would be worthwhile IMHO.

Mikolaj · 2023-03-22T07:32:41Z

@bacchanalia: I'm not sure if the notifications are reaching you, so I may try the email. I'd prefer to avoid mucking around with somebody's design and code without as much as informing the author.

Mikolaj · 2023-04-04T19:05:45Z

@bacchanalia: I think you started working on this recently, but I can't find the PR. Could you point me to it? Thank you.

bacchanalia · 2023-04-05T03:04:07Z

I'm sorry, my life has been really hectic the past week. I didn't realize this was a blocker, sorry. Looking into this now.

This saves 21 chars in the build path, which should help avoid long path issues on Windows. Bug haskell#8841

bacchanalia · 2023-04-05T03:49:19Z

@Mikolaj I put a quick fix up which saves 21 chars by using a more compact encoding for the hash. If this isn't enough, I can investigate other methods, like you were discussing in your previous comment. I don't have a Windows development environment set up to test it right now, but should be able to set one up in the next few days.

Mikolaj · 2023-04-05T07:23:06Z

Well done! Apologies for badgering --- this is not a clear blocker, but still every saved character is one less potential fresh user struggling, before GHC fixes this upstream.

Could a kind Windows user test if this suffices in any standard Windows install? Alternatively, could somebody tweak our CI not to shorten the path by setting CABAL_DIR to something unnaturally short?

Using a Base64 hash and truncating it to 26 characters, saves 38 chars, which helps avoid long paths issues on Windows, while still providing 130 bits of hash in order to avoid collisions. Bug haskell#8841

* Use shorter hash for script-builds directories Using a Base64 hash and truncating it to 26 characters, saves 38 chars, which helps avoid long paths issues on Windows, while still providing 130 bits of hash in order to avoid collisions. Bug #8841 * Use the script cache dir as the dist dir Putting script build products under dist-newstyle within the cache directory is unnecessary because we already control the cache directory and can ensure there are no conflicts. * Use the actual script name in the executable name Previously, the script name was sanitized in final executable name, because the executable name had to match the component name, which only allowed for a limited character set. Now we can use the actual script name in the executable name. This only lets us shorten the component name without losing clarity. * Add changelog entry * Reenable script tests for Windows/ghc-9.4.*

* Use shorter hash for script-builds directories Using a Base64 hash and truncating it to 26 characters, saves 38 chars, which helps avoid long paths issues on Windows, while still providing 130 bits of hash in order to avoid collisions. Bug #8841 * Use the script cache dir as the dist dir Putting script build products under dist-newstyle within the cache directory is unnecessary because we already control the cache directory and can ensure there are no conflicts. * Use the actual script name in the executable name Previously, the script name was sanitized in final executable name, because the executable name had to match the component name, which only allowed for a limited character set. Now we can use the actual script name in the executable name. This only lets us shorten the component name without losing clarity. * Add changelog entry * Reenable script tests for Windows/ghc-9.4.* (cherry picked from commit a482a63) # Conflicts: # cabal-install/src/Distribution/Client/CmdRun.hs # cabal-install/src/Distribution/Client/ProjectConfig.hs

ulysses4ever · 2023-05-27T04:41:33Z

This is fixed on master now (#8898) but automatic backport to 3.10 didn't land because of merge conflicts, unfortunately (#8970).

jneira · 2023-05-28T06:56:26Z

hi, only want to note that to be precise the issue is not totally fixed, still is possible to hit the limit with enough long script file names as it is noted in the QA notes of the original pr

hasufell · 2023-05-28T07:04:39Z

hi, only want to note that to be precise the issue is not totally fixed, still is possible to hit the limit with enough long script file names as it is noted in the QA notes of the original pr

Right, but that's not a regression to 3.6, so to speak. Right?

ulysses4ever · 2023-05-28T09:07:14Z

@jneira cabal can't fix this issue for real, can it? It's limitation of Windows, and we can only try to not hit it too often.

hasufell · 2023-05-28T09:14:17Z

@jneira cabal can't fix this issue for real, can it? It's limitation of Windows, and we can only try to not hit it too often.

It could, by using \\?\ style paths everywhere, but that's kinda difficult: https://learn.microsoft.com/en-us/archive/blogs/bclteam/long-paths-in-net-part-1-of-3-kim-hamilton

jneira · 2023-05-28T11:48:39Z

or fixing the script file name (losing the complete file name in exchange)

ulysses4ever · 2023-05-30T01:43:04Z

I backported (#8970) the fix to 3.10 and as the original reporter I feel confident that we can close the ticket because it was specifically describing the regression in 3.8 w.r.t 3.6 that is fixed now (on master and on 3.10 branch).

ulysses4ever · 2023-05-30T01:44:57Z

Ideas on how to improve our UX on Windows are welcome as separate tickets.

jneira · 2023-05-30T18:45:55Z

well the maximum long script file name (it does not depend on full path only the name) is around 30 chars so I I think it will be not reached frequently

ulysses4ever added type: bug platform: windows regression in 3.10 labels Mar 12, 2023

gbaz added the priority: high 🔥 label Mar 12, 2023

gbaz added this to To Do in cabal-install 3.10.2.0 via automation Mar 12, 2023

jneira added cabal-install: cmd/run can-workaround There is a (maybe partial) workaround for the issue or missing feature regression in 3.8 labels Mar 12, 2023

ulysses4ever added the re: scripts label Mar 12, 2023

jneira changed the title ~~GHCup CI fails for cabal-3.10.1.0 on Windows~~ cabal run fails on windows due to the long path issue Mar 12, 2023

Mikolaj mentioned this issue Mar 13, 2023

Choco install of cabal 3.2 fails in connection with ghc 9.6.1 haskell/actions#202

Closed

bacchanalia added a commit to bacchanalia/cabal that referenced this issue Apr 5, 2023

Use Base64 hash for script-builds directories

de690ea

This saves 21 chars in the build path, which should help avoid long path issues on Windows. Bug haskell#8841

bacchanalia mentioned this issue Apr 5, 2023

Shorten script-builds paths #8898

Merged

bacchanalia self-assigned this Apr 9, 2023

Kleidukos mentioned this issue Apr 24, 2023

Fix 575, regression tests haskell/bytestring#582

Merged

ulysses4ever mentioned this issue May 27, 2023

How are "recommended" versions chosen? haskell/ghcup-hs#833

Closed

ulysses4ever closed this as completed May 30, 2023

cabal-install 3.10.2.0 automation moved this from To Do to Done May 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cabal run fails on windows due to the long path issue #8841

cabal run fails on windows due to the long path issue #8841

ulysses4ever commented Mar 12, 2023 •

edited

Loading

gbaz commented Mar 12, 2023

jneira commented Mar 12, 2023 •

edited

Loading

ulysses4ever commented Mar 12, 2023

jneira commented Mar 12, 2023 •

edited

Loading

ulysses4ever commented Mar 13, 2023

RyanGlScott commented Mar 13, 2023

Mikolaj commented Mar 17, 2023 •

edited

Loading

Mikolaj commented Mar 17, 2023 •

edited

Loading

Mikolaj commented Mar 22, 2023

Mikolaj commented Apr 4, 2023

bacchanalia commented Apr 5, 2023

bacchanalia commented Apr 5, 2023 •

edited

Loading

Mikolaj commented Apr 5, 2023

ulysses4ever commented May 27, 2023

jneira commented May 28, 2023

hasufell commented May 28, 2023

ulysses4ever commented May 28, 2023

hasufell commented May 28, 2023

jneira commented May 28, 2023

ulysses4ever commented May 30, 2023 •

edited

Loading

ulysses4ever commented May 30, 2023

jneira commented May 30, 2023

cabal run fails on windows due to the long path issue #8841

cabal run fails on windows due to the long path issue #8841

Comments

ulysses4ever commented Mar 12, 2023 • edited Loading

gbaz commented Mar 12, 2023

jneira commented Mar 12, 2023 • edited Loading

ulysses4ever commented Mar 12, 2023

jneira commented Mar 12, 2023 • edited Loading

ulysses4ever commented Mar 13, 2023

RyanGlScott commented Mar 13, 2023

Mikolaj commented Mar 17, 2023 • edited Loading

Mikolaj commented Mar 17, 2023 • edited Loading

Mikolaj commented Mar 22, 2023

Mikolaj commented Apr 4, 2023

bacchanalia commented Apr 5, 2023

bacchanalia commented Apr 5, 2023 • edited Loading

Mikolaj commented Apr 5, 2023

ulysses4ever commented May 27, 2023

jneira commented May 28, 2023

hasufell commented May 28, 2023

ulysses4ever commented May 28, 2023

hasufell commented May 28, 2023

jneira commented May 28, 2023

ulysses4ever commented May 30, 2023 • edited Loading

ulysses4ever commented May 30, 2023

jneira commented May 30, 2023

ulysses4ever commented Mar 12, 2023 •

edited

Loading

jneira commented Mar 12, 2023 •

edited

Loading

jneira commented Mar 12, 2023 •

edited

Loading

Mikolaj commented Mar 17, 2023 •

edited

Loading

Mikolaj commented Mar 17, 2023 •

edited

Loading

bacchanalia commented Apr 5, 2023 •

edited

Loading

ulysses4ever commented May 30, 2023 •

edited

Loading