# bazel fails if workspace is not on C: #1463

Closed
opened this Issue Jul 1, 2016 · 47 comments

Projects
None yet

### divydal commented Aug 25, 2016

 Hi, I have the same problem when trying to build 7b5e1af

### sunsided commented Sep 6, 2016

 Same here, trying to build Bazel master (180d1b5) or 0.3.1 on a German Windows 10 Version 1607, using msys2 x64 and jdk1.8.0_74: $./compile.sh INFO: You can skip this first step by providing a path to the bazel binary as second argument: INFO: ./compile.sh compile /path/to/bazel 🍃 Building Bazel from scratch....... D:\dev\FremdeSources\bazel>call "C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/VCVARSALL.BAT" amd64 Microsoft (R) C/C++-Optimierungscompiler Version 19.00.24210 für x64 Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten. windows_error_handling.cc C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xlocale(341): warning C4530: C++-Handler verwendet, aber Entladesemantik ist nicht aktiviert. Geben Sie /EHsc an. C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\exception(359): warning C4577: "noexcept" wird ohne angegebenen Ausnahmebehandlungsmodus verwendet. Die Beendigung bei einer Ausnahme ist nicht sichergestellt. Geben Sie "/EHsc" an. windows_file_operations.cc C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xlocale(341): warning C4530: C++-Handler verwendet, aber Entladesemantik ist nicht aktiviert. Geben Sie /EHsc an. C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\exception(359): warning C4577: "noexcept" wird ohne angegebenen Ausnahmebehandlungsmodus verwendet. Die Beendigung bei einer Ausnahme ist nicht sichergestellt. Geben Sie "/EHsc" an. windows_processes.cc C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xlocale(341): warning C4530: C++-Handler verwendet, aber Entladesemantik ist nicht aktiviert. Geben Sie /EHsc an. C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\iosfwd(343): warning C4577: "noexcept" wird ohne angegebenen Ausnahmebehandlungsmodus verwendet. Die Beendigung bei einer Ausnahme ist nicht sichergestellt. Geben Sie "/EHsc" an. Code wird generiert... Microsoft (R) Incremental Linker Version 14.00.24210.0 Copyright (C) Microsoft Corporation. All rights reserved. /dll /implib:D:\dev\msys64\tmp\bazel_N9GyDiSO\jni\windows_jni.lib /out:D:\dev\msys64\tmp\bazel_N9GyDiSO\jni\windows_jni.dll windows_error_handling.obj windows_file_operations.obj windows_processes.obj Bibliothek "D:\dev\msys64\tmp\bazel_N9GyDiSO\jni\windows_jni.lib" und Objekt "D:\dev\msys64\tmp\bazel_N9GyDiSO\jni\windows_jni.exp" werden erstellt. . 🍃 Building Bazel with Bazel. .WARNING: D:/dev/msys64/tmp/bazel_vSsqtL7b/out/external/bazel_tools/WORKSPACE:1: Workspace name in D:/dev/msys64/tmp/bazel_vSsqtL7b/out/external/bazel_tools/WORKSPACE (@io_bazel) does not match the name given in the repository's definition (@bazel_tools); this will cause a build error in future versions. ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_74 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). INFO: Elapsed time: 0,832s  and on subsequent calls to compile.sh: $ ./compile.sh INFO: You can skip this first step by providing a path to the bazel binary as second argument: INFO: ./compile.sh compile /path/to/bazel 🍃 Building Bazel from scratch...... 🍃 Building Bazel with Bazel. .WARNING: D:/dev/msys64/tmp/bazel_cJf7e10a/out/external/bazel_tools/WORKSPACE:1: Workspace name in D:/dev/msys64/tmp/bazel_cJf7e10a/out/external/bazel_tools/WORKSPACE (@io_bazel) does not match the name given in the repository's definition (@bazel_tools); this will cause a build error in future versions. ERROR: no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_74 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). ERROR: no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_74 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). INFO: Elapsed time: 0,798s 
Contributor

### petemounce commented Sep 17, 2016 • edited Edited 1 time petemounce edited Sep 18, 2016 (most recent)

 This is still a problem for me in my own source tree (but not bazel's) at current HEAD, c484f19. I have Windows 10 Anniversary Update, Bash-on-Windows is installed. I have jdk1.8.0_102 I get the same error with powershell (since #1453 allows its use) and msys2 x64. Edit: I get the same error regardless of whether b-o-w bash.exe appears first or later than msys2's in PATH I think. @sunsided @divydal do you have b-o-w too...?

### sunsided commented Sep 18, 2016

 @petemounce Yes, installed. Didn't try it though because I need Windows binaries.

Contributor

### petemounce commented Oct 6, 2016 • edited Edited 1 time petemounce edited Oct 6, 2016 (most recent)

 c484f19: When I run bazel commands on my c: drive, they work; when I run them from my d: drive then I get this. I tested this by running bazel build and bazel info in the root of this repo while on c: and d:, same system as I reported from. I get this error when on d:. I get it inside msys2 shell and powershell. Unfortunately I can't easily test this against my in-house project because that would currently mean moving it to an unencrypted drive, which I won't do. Still, pretty indicative. (Also, same when I run the bazel.exe that lives on c: from my d: drive, fwiw, by fully qualifying the path to the exe)
Contributor

### laszlocsomor commented Oct 6, 2016

 @petemounce : Thanks for reporting it. I'm seeing the same. Looking for the root cause and fixing soon.
Contributor

### laszlocsomor commented Oct 6, 2016 • edited Edited 1 time laszlocsomor edited Oct 6, 2016 (most recent)

 The problem is rooted in a design limitation of Bazel, namely that a file system is assumed to have a single root. That's not true on Windows, and so the hack we've been using is treating "C:" as a top-level directory, not a root. WindowsFileSystem.getFilesystemRoot() reports / as the root. This is breaking down if the workspace is on another drive, and we have to fix this design flaw properly. This hack also means that all RootedPath objects are incorrect in the sense that their root is / and not the drive letter, so the RootedPath for C:\ itself is [/]/[C:]. No PathFragment objects are absolute on Windows because C: is just a directory, / is the root. These factors lead to more problems down the road, for example in FileFunction.resolveFromAncestors where: rootedPath = "/C:" // root="/", relative="C:" relativePath = "C:" baseName = "C:" parentRealRootedPath = "/" // root="/", relative="" parentRealRootedPath.getRelativePath() = "" parentRealRootedPath.getRelativePath().getRelative(baseName) = "" ---> thus realRootedPath = "/" // root="/", relative=""  Which means that we attempt to evaluate C:-relative paths relative to the current drive (D:), and this manifests as incorrect-looking missing file errors.
Contributor

### laszlocsomor commented Oct 6, 2016

 One more factor: when a PathFragment is just C:, then its segments are empty, though isAbsolute = false, and driveLetter = 'C'. When it's a path like C:/Program Files, then the driveLetter = 'C' (fine), isAbsolute = false (still not fine I guess) but segments = ["C:", "Program Files"]. This discrepancy is bad, this is what's causing parentRealRootedPath.getRelativePath().getRelative(baseName) in my previous comment to be "/", because parentRealRootedPath.getRelativePath() returns the empty PathFragment, and baseName is PathFragment("C:"), and the .getRelative call joins the segments which neither of these PathFragments have any of.
Contributor

### laszlocsomor commented Oct 6, 2016

 @dslomov : Your workaround idea works! Workaround: create a junction somewhere on "C:" pointing to the workspace. Then bazel works. To fix this bug properly we have to introduce the concept of multiple filesystem roots.
Contributor

### laszlocsomor commented Oct 6, 2016

 Dropping priority because there's a workaround.

Contributor

### petemounce commented Oct 6, 2016

 @laszlocsomor So, the workaround is to (from an admin cmd shell) mklink /d c:\ , right...? I wasn't able to get it to work: # I have my bazel clone at d:\bazel for testing this mklink /d c:\bazel d:\bazel c: dir Volume in drive C is foobar Volume Serial Number is blahblah Directory of c:\ ....redacted... 06/10/2016 13:50 bazel [d:\bazel] ....redacted... cd c:\bazel powershell #which has a profile that adds msys DLL to my PATH bazel info ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build).
Contributor

### laszlocsomor commented Oct 6, 2016

 @petemounce : I did this from a non-Admin command prompt: mklink /j c:\work\bazel_ws d:\path\to\actual\bazel_ws  Then in msys, I cd'd into /c/work/bazel_ws and build there.
Contributor

### laszlocsomor commented Oct 6, 2016 • edited Edited 1 time laszlocsomor edited Oct 6, 2016 (most recent)

 FYI, mklink /d creates a directory symlink, mklink /j creates a junction. Here's the difference explained: http://superuser.com/a/343079
Contributor

### petemounce commented Oct 6, 2016 • edited Edited 1 time petemounce edited Oct 6, 2016 (most recent)

 That didn't work for me (Windows 10 anniversary update) In non-admin cmd shell: rmdir c:\bazel # remove previous experiment mklink /j c:\bazel d:\bazel Junction created for c:\bazel <<===>> d:\bazel set PATH=c:\tools\msys64\usr\bin;%PATH% dir Volume in drive C is foobar Volume Serial Number is blahblah Directory of C:\ ..... 06/10/2016 14:12 bazel [d:\bazel] ...... cd c:\bazel bazel info ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build).  With msys2: Pete@foobar MSYS ~ $cd /c/bazel Pete@foobar MSYS /c/bazel$ /c/Users/Pete/bazel/output/bazel.exe info # bazel is not on my PATH in msys2 ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). 

### tomzx commented Oct 6, 2016 • edited Edited 1 time tomzx edited Oct 6, 2016 (most recent)

 Same result as @petemounce for me on Windows 7 SP1. However in my case, I'm still stuck at the ./compile.sh step.
Contributor

### laszlocsomor commented Oct 6, 2016

 @petemounce , @tomzx : It works if the junction is not the last path element. I don't know why and I haven't investigated (and probably won't unless I have to) but here it is. My D: drive is a mapped folder from C: (using subst, see #1463 (comment)), and my E: is a USB mass storage. I have the following bazel workspaces: d:\bazel_ws d:\bazel\ws e:\bazel_ws  In cmd.exe: C:\>mklink /j bazel_d1 d: Local volumes are required to complete the operation. C:\>mklink /j bazel_d1 d:\ Local volumes are required to complete the operation. C:\>mklink /j bazel_d1 d:\bazel_ws Junction created for bazel_d1 <<===>> d:\bazel_ws C:\>mklink /j bazel_d2 d:\bazel\ Junction created for bazel_d2 <<===>> d:\bazel\ C:\>mklink /j bazel_e1 e:\bazel_ws Junction created for bazel_e1 <<===>> e:\bazel_ws C:\>mklink /j bazel_e2 e: Junction created for bazel_e2 <<===>> e: C:\>dir bazel_d1 ... 2016-10-05 11:11 AM foo 2016-10-05 11:11 AM 0 WORKSPACE ... C:\>dir bazel_d2 ... 2016-10-06 03:37 PM ws ... C:\>dir bazel_e1 ... 2016-10-06 01:29 PM 13 hello.txt 2016-10-06 01:30 PM 0 WORKSPACE ... C:\>dir bazel_e2 ... 2016-10-06 01:29 PM bazel_ws ...  (Note that I couldn't create a junction just to D:, only to a subdirectory in it. I don't know why this is, probably a limitation of NTFS.) In msys: $cd /c/bazel_d1 ; bazel --batch info >&/dev/null ; echo$? 2 $cd /c/bazel_d2/ws ; bazel --batch info >&/dev/null ; echo$? 0 $cd /c/bazel_e1 ; bazel --batch info >&/dev/null ; echo$? 2 $cd /c/bazel_e2/bazel_ws ; bazel --batch info >&/dev/null ; echo$? 0 

Contributor

### laszlocsomor commented Oct 7, 2016

 Bumping priority again because this is a serious blow on usability.
Contributor

### laszlocsomor commented Oct 10, 2016

 I have a fix for this, stay tuned.
Contributor

### laszlocsomor commented Oct 11, 2016

 Out for review. If all goes well, we can push the fix to GH today or tomorrow.
Contributor

### petemounce commented Oct 13, 2016

 Hi @laszlocsomor - I didn't spot this on [gerrit|https://bazel.googlesource.com/bazel/]. Is it somewhere else? Happy to try it out, since it's the next thing to a blocker for us.

Closed

Contributor

### laszlocsomor commented Oct 14, 2016

 Hey @petemounce , it's not yet submitted, sorry about that. It should be in today or Monday, if not I'll ping this thread.

### youkpan commented Oct 15, 2016

 seems java.exe path problem set JAVA_HOME=C:/Program\ Files/Java/jdk1.8.0_101&c:\bazel.exe Couldn't find java at 'C:/Program\ Files/Java/jdk1.8.0_101/bin/java'.
Contributor

### dslomov commented Oct 15, 2016

 @youkpan this is not really related to this bug, is it? In the future, please file a separate issue if the problem is unrelated to the problem discussed in the thread. Anyhow, in your case the content of JAVA_HOME variable after your command is literally C:/Program\ Files/Java/jdk1.8.0_101/bin/java, including a \ before the space. The following should work: C:> set JAVA_HOME=C:/Program Files/Java/jdk1.8.0_101 C:> bazel.exe 

Closed

Contributor

### petemounce commented Oct 16, 2016

 For later reference, I'm able to work around this by having my source code inside a sub directory on my separate drive (s:\whatever\here) create a junction to that from inside a directory on c: mklink /j c:\src\s-drive s:\whatever\here
Contributor

### laszlocsomor commented Oct 17, 2016

 FYI I'm still working on this change, still not submitted.

Closed

Contributor

### laszlocsomor commented Oct 17, 2016

 Submitted internally, will push it to GitHub in the next 24 hours.

Contributor

### laszlocsomor commented Oct 17, 2016

 Looks like I made a mistake in WindowsFileSystem.java retrieving UNIX_ROOT (if you don't see that then the change is not on GitHub yet), but I sent out a fix for it. If I'm lucky we'll push both changes to GH in the same run.
Contributor

### laszlocsomor commented Oct 17, 2016

 If not, then in the meantime the workaround will be to pass --host_jvm_args=-Dbazel.windows_unix_root="C:/tools/msys64/" to bazel.

### bazel-io pushed a commit that referenced this issue Oct 18, 2016

 VFS: implement a Windows-specific Path subclass 
The new subclass WindowsFileSystem.WindowsPath is
aware of Windows drives.

This change:
- introduces a new factory for Path objects so
FileSystems can return a custom implementation
that instantiates filesystem-specific Paths
- implements the WindowsPath subclass of Path that
is aware of Windows drives
- introduces the bazel.windows_unix_root JVM
argument that defines the MSYS root, which
defines the absolute Windows path that is the
root of all Unix paths that Bazel creates (e.g.
"/usr/lib" -> "C:/tools/msys64/usr/lib") except
if the path is of the form "/c/foo" which is
treated as "C:/foo"
- removes all Windows-specific logic from Path

PathFragment is still aware of drive letters and
it has to remain so because it is unaware of file
systems.

WindowsPath restricts the allowed path strings to
absolute Unix paths where the first segment, if
any, is a volume specifier. From now on if Bazel
attempts to create a WindowsPath from an absolute
Unix path, Bazel will make it relative to
WindowsPath.UNIX_ROOT, unless the first component
is a single-letter name (e.g. "/c/foo" which is
"C:/foo").

Subclassing Path is necessary because a Unix-style
absolute path doesn't sufficiently define a full
Windows path, as it may be relative to any drive.

Fixes #1463

--
MOS_MIGRATED_REVID=136350304
 e0d7a54 

### bazel-io pushed a commit that referenced this issue Oct 18, 2016

 VFS, WindowsFileSystem: fix UNIX_ROOT retrieval 
Executing bash.exe directly instead of through
cmd.exe doesn't seem to work. This change fixes
that problem.

Fixes #1463 (again)

--
MOS_MIGRATED_REVID=136364606
 8bb4299 

### bazel-io pushed a commit that referenced this issue Oct 19, 2016

 Rollback of commit 8bb4299. 
*** Reason for rollback ***

Suspected root cause for Windows bootstrap on Bazel CI breakage:

java.lang.NullPointerException

Closed

Contributor

### petemounce commented Oct 31, 2016 • edited Edited 1 time petemounce edited Oct 31, 2016 (most recent)

 @laszlocsomor in the docs, BAZEL_SH wants path separators to be / not \. That's what my own var is set with. Also, I set those via $env:BAZEL_SH if I do them manually within a powershell, rather than $global:BAZEL_SH`.

Closed