New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows: fopen cannot create file if its full path is longer than 254 chars #1514

Open
volth opened this Issue Jan 9, 2019 · 11 comments

Comments

Projects
None yet
3 participants
@volth
Copy link

volth commented Jan 9, 2019

Chromium build started from a directory which is ~30 chars long fails with

[3572/20749] STAMP obj/services/network/public/mojom/mojom_ip_address_shared_cpp_sources__jumbo_merge.stamp
[3573/20749] STAMP obj/services/metrics/public/mojom/mojom_blink__jumbo_merge.stamp
ninja: error: WriteFile(__services_network_public_mojom_mutable_network_traffic_annotation_interface_shared_cpp_sources__jumbo_merge___build_toolchain_win_win_clang_x64__rule.rsp): Unable to create file. No such file or directory

The filename "__services_network_public_mojom_mutable_network_traffic_annotation_interface_shared_cpp_sources__jumbo_merge___build_toolchain_win_win_clang_x64__rule.rsp" is 155 chars long + paths inside Chromium source tree + prefix caused by superviser package manager (nixpkgs) result in full path being about 260 chars long

Alhough fopen()'s argument has length of only 155 chars, it hits the limit set for the full path.

I propose to use raw WinAPI instead of fopen() and add \\?\ prefix to the argument to allow paths up to 32767 chars long.

@jhasse

This comment has been minimized.

Copy link
Collaborator

jhasse commented Jan 9, 2019

It's possible to lift the MAX_PATH limitation via a manifest setting, see https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#paths. I'm not sure if this will apply to fopen though. Can you test if setting the registry setting already fixes your issue?

@jhasse jhasse added windows bug labels Jan 9, 2019

@atetubou

This comment has been minimized.

Copy link
Contributor

atetubou commented Jan 9, 2019

I think it is better to file a bug to crbug too so that jumbo build generates shorter file name.

@volth

This comment has been minimized.

Copy link

volth commented Jan 9, 2019

It's possible to lift the MAX_PATH limitation via a manifest setting, see https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#paths. I'm not sure if this will apply to fopen though

That seems to require Windows 10 and affects only Unicode API (CreateFileW but not CreateFileA)

@jhasse

This comment has been minimized.

Copy link
Collaborator

jhasse commented Jan 9, 2019

We're using neither CreateFileW nor CreateFileA but fopen. I wonder if the setting will affect it. If not, it might affect _wfopen which is needed for #1195 anyway.

@volth

This comment has been minimized.

Copy link

volth commented Jan 9, 2019

We're using neither CreateFileW nor CreateFileA but fopen. I wonder if the setting will affect it. If not, it might affect _wfopen

Yes, that what I meant. fopen is not WinAPI, it should be a wrapper around some of CreateFile* functions, likely CreateFileA (and _wfopen for CreateFileW).

Even if the manifest would help, if will work only on Windows newer than "Windows 10, version 1607" (as stated on https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file) which would be a weird requirement for ninja.

@jhasse

This comment has been minimized.

Copy link
Collaborator

jhasse commented Jan 9, 2019

That isn't a requirement for running ninja at all, only for the fix.

@volth

This comment has been minimized.

Copy link

volth commented Jan 9, 2019

If we will fix it by adding the manifest, what about the next person who step into this bug?
Will he be advised to upgrade Windows to Windows 10?

@jhasse

This comment has been minimized.

Copy link
Collaborator

jhasse commented Jan 9, 2019

I doubt that this would be an option for him, otherwise he would have done it already.

@volth

This comment has been minimized.

Copy link

volth commented Jan 10, 2019

Well, even if ninja managed to create the file with long path (I fixed it with master...volth:patch-1), Chromium jumbo build is still broken: python fails to open that file :(

Traceback (most recent call last):
  File "../../build/config/merge_for_jumbo.py", line 144, in <module>
    main()
  File "../../build/config/merge_for_jumbo.py", line 113, in main
    with open(args.file_list) as file_list_file:
IOError: [Errno 2] No such file or directory: '__services_network_public_mojom_mutable_network_traffic_annotation_interface_shared_cpp_sources__jumbo_merge___build_toolchain_win_win_clang_x64__rule.rsp'

So fixing it in Chromium would be a right solution.
Although, as nixpkgs adds long prefixes, I anticipate more bugs like this one so I will be happy to see ninja and python and other tools be able to work with long paths (on Windows 10 as well on Windows 7 / Windows Server 2008)

@volth

This comment has been minimized.

Copy link

volth commented Jan 10, 2019

setting HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem@LongPathsEnabled to 1 did not help (on Windows 10, I did reboot after changing the value)

volth pushed a commit to volth/nix-windows that referenced this issue Jan 10, 2019

Your Name
use root of subst'ituted drive as $NIX_BUILD_ROOT. it helps to work a…
…round the limits on path length (example: ninja-build/ninja#1514) but drive letters are very limited resources which could leak on nix-build abnormal termination

volth pushed a commit to volth/nix-windows that referenced this issue Jan 10, 2019

Your Name
use root of subst'ituted drive as $NIX_BUILD_ROOT.
It helps to work around the limits on path length (example: ninja-build/ninja#1514).

But.
Drive letters is a very limited resource which could leak on nix-build abnormal termination.
@jhasse

This comment has been minimized.

Copy link
Collaborator

jhasse commented Jan 11, 2019

So it probably only applies to _wfopen.

volth added a commit to volth/nix-windows that referenced this issue Jan 16, 2019

use root of subst'ituted drive as $NIX_BUILD_ROOT.
It helps to work around the limits on path length (example: ninja-build/ninja#1514).

But.
Drive letters is a very limited resource which could leak on nix-build abnormal termination.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment