bazel fails if workspace is not on C: #1463

Closed
opened this Issue Jul 1, 2016 · 47 comments

None yet

11 participants

commented Jul 1, 2016 edited
 Hi, I'm trying to build master (a18add1) following the instruction provided and I'm getting the error below. The path to my java jdk is correct and the folder does indeed exist. Furthermore, I can see that the first part of the compile script calls javac from that directory without any issue. I'm willing to try and debug this out if someone is willing to provide me with some guidance. Thanks! $./compile.sh INFO: You can skip this first step by providing a path to the bazel binary as second argument: INFO: ./compile.sh compile /path/to/bazel 🍃 Building Bazel from scratch...... 🍃 Building Bazel with Bazel. .WARNING: C:/tools/msys64/tmp/bazel.jR1zd19q/out/external/bazel_tools/WORKSPACE:1: Workspace name in C:/tools/msys64/tmp/bazel.jR1zd19q/out/external/bazel_tools/WORKSPACE (@io_bazel) does not match the name given in the repository's definition (@bazel_tools); this will cause a build error in future versions. ERROR: no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_73 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). ERROR: no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_73 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). INFO: Elapsed time: 1.048s Building output/bazel  Member commented Jul 4, 2016  Assigning to @dslomov who has more knowledge about Windows. was assigned by aehlig Jul 4, 2016  I'm seeing the same error here when trying to build 89484da . Contributor commented Aug 17, 2016 edited  I am able to compile bazel tags 0.3.0 and 0.3.1, and current master (60af9db) on Windows 10, but I get the same error when trying to compile (with either version) my in-house project. Sample output: $ bazel build --verbose_failures --cpu='x64_windows_msvc' --host_cpu='x64_windows_msvc' cpp/protocgenfabric/... . ERROR: no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). ERROR: no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). INFO: Elapsed time: 1.948s  (There's also a reference to running blaze clean which I guess is a typo) I should mention - that JDK directory definitely exists. To try to eliminate the space in the Program Files being an issue, I've used an admin prompt to do mklink /j prfi "c:\Program Files". That hasn't helped: Pete@petemounce-win-imp MSYS /s/e $echo$JAVA_HOME C:/Program Files/Java/jdk1.8.0_102 Pete@petemounce-win-imp MSYS /s/e $export JAVA_HOME=c:/prfi/Java/jdk1.8.0_102 Pete@petemounce-win-imp MSYS /s/e$ $JAVA_HOME/bin/java -version java version "1.8.0_102" Java(TM) SE Runtime Environment (build 1.8.0_102-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode) Pete@petemounce-win-imp MSYS /s/e$ $JAVA_HOME/bin/javac -version javac 1.8.0_102 Pete@petemounce-win-imp MSYS /s/e$ bazel build --verbose_failures --cpu='x64_windows_msvc' --host_cpu='x64_windows_msvc' cpp/protocgenfabric/... ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). INFO: Elapsed time: 0.280s  Where is it that bazel is deciding to use that JDK if not the JAVA_HOME variable?
commented Aug 25, 2016
 Hi, I have the same problem when trying to build 7b5e1af
commented Sep 6, 2016
 Same here, trying to build Bazel master (180d1b5) or 0.3.1 on a German Windows 10 Version 1607, using msys2 x64 and jdk1.8.0_74: $./compile.sh INFO: You can skip this first step by providing a path to the bazel binary as second argument: INFO: ./compile.sh compile /path/to/bazel 🍃 Building Bazel from scratch....... D:\dev\FremdeSources\bazel>call "C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/VCVARSALL.BAT" amd64 Microsoft (R) C/C++-Optimierungscompiler Version 19.00.24210 für x64 Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten. windows_error_handling.cc C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xlocale(341): warning C4530: C++-Handler verwendet, aber Entladesemantik ist nicht aktiviert. Geben Sie /EHsc an. C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\exception(359): warning C4577: "noexcept" wird ohne angegebenen Ausnahmebehandlungsmodus verwendet. Die Beendigung bei einer Ausnahme ist nicht sichergestellt. Geben Sie "/EHsc" an. windows_file_operations.cc C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xlocale(341): warning C4530: C++-Handler verwendet, aber Entladesemantik ist nicht aktiviert. Geben Sie /EHsc an. C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\exception(359): warning C4577: "noexcept" wird ohne angegebenen Ausnahmebehandlungsmodus verwendet. Die Beendigung bei einer Ausnahme ist nicht sichergestellt. Geben Sie "/EHsc" an. windows_processes.cc C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xlocale(341): warning C4530: C++-Handler verwendet, aber Entladesemantik ist nicht aktiviert. Geben Sie /EHsc an. C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\iosfwd(343): warning C4577: "noexcept" wird ohne angegebenen Ausnahmebehandlungsmodus verwendet. Die Beendigung bei einer Ausnahme ist nicht sichergestellt. Geben Sie "/EHsc" an. Code wird generiert... Microsoft (R) Incremental Linker Version 14.00.24210.0 Copyright (C) Microsoft Corporation. All rights reserved. /dll /implib:D:\dev\msys64\tmp\bazel_N9GyDiSO\jni\windows_jni.lib /out:D:\dev\msys64\tmp\bazel_N9GyDiSO\jni\windows_jni.dll windows_error_handling.obj windows_file_operations.obj windows_processes.obj Bibliothek "D:\dev\msys64\tmp\bazel_N9GyDiSO\jni\windows_jni.lib" und Objekt "D:\dev\msys64\tmp\bazel_N9GyDiSO\jni\windows_jni.exp" werden erstellt. . 🍃 Building Bazel with Bazel. .WARNING: D:/dev/msys64/tmp/bazel_vSsqtL7b/out/external/bazel_tools/WORKSPACE:1: Workspace name in D:/dev/msys64/tmp/bazel_vSsqtL7b/out/external/bazel_tools/WORKSPACE (@io_bazel) does not match the name given in the repository's definition (@bazel_tools); this will cause a build error in future versions. ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_74 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). INFO: Elapsed time: 0,832s  and on subsequent calls to compile.sh: $ ./compile.sh INFO: You can skip this first step by providing a path to the bazel binary as second argument: INFO: ./compile.sh compile /path/to/bazel 🍃 Building Bazel from scratch...... 🍃 Building Bazel with Bazel. .WARNING: D:/dev/msys64/tmp/bazel_cJf7e10a/out/external/bazel_tools/WORKSPACE:1: Workspace name in D:/dev/msys64/tmp/bazel_cJf7e10a/out/external/bazel_tools/WORKSPACE (@io_bazel) does not match the name given in the repository's definition (@bazel_tools); this will cause a build error in future versions. ERROR: no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_74 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). ERROR: no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_74 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). INFO: Elapsed time: 0,798s 
Contributor
commented Sep 17, 2016 edited
 This is still a problem for me in my own source tree (but not bazel's) at current HEAD, c484f19. I have Windows 10 Anniversary Update, Bash-on-Windows is installed. I have jdk1.8.0_102 I get the same error with powershell (since #1453 allows its use) and msys2 x64. Edit: I get the same error regardless of whether b-o-w bash.exe appears first or later than msys2's in PATH I think. @sunsided @divydal do you have b-o-w too...?
 @petemounce Yes, installed. Didn't try it though because I need Windows binaries.
 This is also an issue for me on Windows 10, using the nightly built here $bazel-nightly.exe info WARNING: C:/Users/jared/AppData/Local/Temp/Bazel/9ncInwMl/external/io_bazel_rules_scala/scala/scala.bzl:473:12: Argument cfg = "host" or cfg = "data" is required if executable = True is provided for a label. WARNING: C:/Users/jared/AppData/Local/Temp/Bazel/9ncInwMl/external/io_bazel_rules_scala/scala/scala.bzl:474:13: Argument cfg = "host" or cfg = "data" is required if executable = True is provided for a label. WARNING: C:/Users/jared/AppData/Local/Temp/Bazel/9ncInwMl/external/io_bazel_rules_scala/scala/scala.bzl:475:14: Argument cfg = "host" or cfg = "data" is required if executable = True is provided for a label. WARNING: C:/Users/jared/AppData/Local/Temp/Bazel/9ncInwMl/external/io_bazel_rules_scala/scala/scala.bzl:480:12: Argument cfg = "host" or cfg = "data" is required if executable = True is provided for a label. WARNING: C:/Users/jared/AppData/Local/Temp/Bazel/9ncInwMl/external/io_bazel_rules_scala/scala/scala.bzl:481:13: Argument cfg = "host" or cfg = "data" is required if executable = True is provided for a label. WARNING: C:/Users/jared/AppData/Local/Temp/Bazel/9ncInwMl/external/io_bazel_rules_scala/scala/scala.bzl:482:11: Argument cfg = "host" or cfg = "data" is required if executable = True is provided for a label. WARNING: C:/Users/jared/AppData/Local/Temp/Bazel/9ncInwMl/external/io_bazel_rules_scala/scala/scala.bzl:493:11: Variables HOST_CFG and DATA_CFG are deprecated in favor of strings "host" and "data" correspondingly. WARNING: C:/Users/jared/AppData/Local/Temp/Bazel/9ncInwMl/external/io_bazel_rules_scala/scala/scala.bzl:545:21: Argument cfg = "host" or cfg = "data" is required if executable = True is provided for a label. ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). C:/Program Files/Java/jdk1.8.0_102 does exist.  Contributor commented Oct 4, 2016  We were able to repro this when a workspace is not on C: drive. Fix for that will hopefully be coming soon, but is it the case for everyone reporting this? Contributor  Yes, my source is on my s: drive. Good thought! commented Oct 4, 2016  Yes , my workspace is on d:. commented Oct 4, 2016  Same, sources on D: commented Oct 4, 2016 edited  @dslomov Yes mine isn't stored on the C: drive. However I did try to create a symlink on the C:, assuming it might related to that, but that didn't work either. I'm assuming symlinks are resolved? Contributor  @tomzx how did you create the symlink (ie, exact command)? There are a couple of different types on Windows, with differing characteristics. commented Oct 4, 2016 edited  @petemounce IIRC, mklink /D C:\bazel E:\bazel. Contributor  Ah ok. That's what I would have tried too. Do any of the methods listed in http://superuser.com/questions/752538/mklink-vs-junction-exe allow workaround? assigned laszlocsomor and unassigned dslomov Oct 5, 2016 added this to the 0.4 milestone Oct 5, 2016 Contributor  FTR i just did subst d: c:\d_drive and that works too (bonus: good old #1744 still around): $ bazel info ERROR: CreateFile(C:\tools\msys64\var\tmp\Bazel\WLSiXh7n\install): The system cannot find the file specified. (2) ........... ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build)  I'll dive into that part of the code and see what I find.
Contributor
commented Oct 6, 2016 edited
 c484f19: When I run bazel commands on my c: drive, they work; when I run them from my d: drive then I get this. I tested this by running bazel build and bazel info in the root of this repo while on c: and d:, same system as I reported from. I get this error when on d:. I get it inside msys2 shell and powershell. Unfortunately I can't easily test this against my in-house project because that would currently mean moving it to an unencrypted drive, which I won't do. Still, pretty indicative. (Also, same when I run the bazel.exe that lives on c: from my d: drive, fwiw, by fully qualifying the path to the exe)
Contributor
 @petemounce : Thanks for reporting it. I'm seeing the same. Looking for the root cause and fixing soon.
Contributor
commented Oct 6, 2016 edited
 The problem is rooted in a design limitation of Bazel, namely that a file system is assumed to have a single root. That's not true on Windows, and so the hack we've been using is treating "C:" as a top-level directory, not a root. WindowsFileSystem.getFilesystemRoot() reports / as the root. This is breaking down if the workspace is on another drive, and we have to fix this design flaw properly. This hack also means that all RootedPath objects are incorrect in the sense that their root is / and not the drive letter, so the RootedPath for C:\ itself is [/]/[C:]. No PathFragment objects are absolute on Windows because C: is just a directory, / is the root. These factors lead to more problems down the road, for example in FileFunction.resolveFromAncestors where: rootedPath = "/C:" // root="/", relative="C:" relativePath = "C:" baseName = "C:" parentRealRootedPath = "/" // root="/", relative="" parentRealRootedPath.getRelativePath() = "" parentRealRootedPath.getRelativePath().getRelative(baseName) = "" ---> thus realRootedPath = "/" // root="/", relative=""  Which means that we attempt to evaluate C:-relative paths relative to the current drive (D:), and this manifests as incorrect-looking missing file errors.
Contributor
 One more factor: when a PathFragment is just C:, then its segments are empty, though isAbsolute = false, and driveLetter = 'C'. When it's a path like C:/Program Files, then the driveLetter = 'C' (fine), isAbsolute = false (still not fine I guess) but segments = ["C:", "Program Files"]. This discrepancy is bad, this is what's causing parentRealRootedPath.getRelativePath().getRelative(baseName) in my previous comment to be "/", because parentRealRootedPath.getRelativePath() returns the empty PathFragment, and baseName is PathFragment("C:"), and the .getRelative call joins the segments which neither of these PathFragments have any of.
Contributor
 @dslomov : Your workaround idea works! Workaround: create a junction somewhere on "C:" pointing to the workspace. Then bazel works. To fix this bug properly we have to introduce the concept of multiple filesystem roots.
Contributor
 Dropping priority because there's a workaround.
added P2 and removed P1 labels Oct 6, 2016
Contributor
 @laszlocsomor So, the workaround is to (from an admin cmd shell) mklink /d c:\ , right...? I wasn't able to get it to work: # I have my bazel clone at d:\bazel for testing this mklink /d c:\bazel d:\bazel c: dir Volume in drive C is foobar Volume Serial Number is blahblah Directory of c:\ ....redacted... 06/10/2016 13:50 bazel [d:\bazel] ....redacted... cd c:\bazel powershell #which has a profile that adds msys DLL to my PATH bazel info ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build).
Contributor
 @petemounce : I did this from a non-Admin command prompt: mklink /j c:\work\bazel_ws d:\path\to\actual\bazel_ws  Then in msys, I cd'd into /c/work/bazel_ws and build there.
Contributor
commented Oct 6, 2016 edited
 FYI, mklink /d creates a directory symlink, mklink /j creates a junction. Here's the difference explained: http://superuser.com/a/343079
Contributor
commented Oct 6, 2016 edited
 That didn't work for me (Windows 10 anniversary update) In non-admin cmd shell: rmdir c:\bazel # remove previous experiment mklink /j c:\bazel d:\bazel Junction created for c:\bazel <<===>> d:\bazel set PATH=c:\tools\msys64\usr\bin;%PATH% dir Volume in drive C is foobar Volume Serial Number is blahblah Directory of C:\ ..... 06/10/2016 14:12 bazel [d:\bazel] ...... cd c:\bazel bazel info ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build).  With msys2: Pete@foobar MSYS ~ $cd /c/bazel Pete@foobar MSYS /c/bazel$ /c/Users/Pete/bazel/output/bazel.exe info # bazel is not on my PATH in msys2 ERROR: in target '//external:jdk-default': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). 
commented Oct 6, 2016 edited
 Same result as @petemounce for me on Windows 7 SP1. However in my case, I'm still stuck at the ./compile.sh step.
Contributor
 @petemounce , @tomzx : It works if the junction is not the last path element. I don't know why and I haven't investigated (and probably won't unless I have to) but here it is. My D: drive is a mapped folder from C: (using subst, see #1463 (comment)), and my E: is a USB mass storage. I have the following bazel workspaces: d:\bazel_ws d:\bazel\ws e:\bazel_ws  In cmd.exe: C:\>mklink /j bazel_d1 d: Local volumes are required to complete the operation. C:\>mklink /j bazel_d1 d:\ Local volumes are required to complete the operation. C:\>mklink /j bazel_d1 d:\bazel_ws Junction created for bazel_d1 <<===>> d:\bazel_ws C:\>mklink /j bazel_d2 d:\bazel\ Junction created for bazel_d2 <<===>> d:\bazel\ C:\>mklink /j bazel_e1 e:\bazel_ws Junction created for bazel_e1 <<===>> e:\bazel_ws C:\>mklink /j bazel_e2 e: Junction created for bazel_e2 <<===>> e: C:\>dir bazel_d1 ... 2016-10-05 11:11 AM foo 2016-10-05 11:11 AM 0 WORKSPACE ... C:\>dir bazel_d2 ... 2016-10-06 03:37 PM ws ... C:\>dir bazel_e1 ... 2016-10-06 01:29 PM 13 hello.txt 2016-10-06 01:30 PM 0 WORKSPACE ... C:\>dir bazel_e2 ... 2016-10-06 01:29 PM bazel_ws ...  (Note that I couldn't create a junction just to D:, only to a subdirectory in it. I don't know why this is, probably a limitation of NTFS.) In msys: $cd /c/bazel_d1 ; bazel --batch info >&/dev/null ; echo$? 2 $cd /c/bazel_d2/ws ; bazel --batch info >&/dev/null ; echo$? 0 $cd /c/bazel_e1 ; bazel --batch info >&/dev/null ; echo$? 2 $cd /c/bazel_e2/bazel_ws ; bazel --batch info >&/dev/null ; echo$? 0 
changed the title from "Inconsistent filesystem operations" on Windows to bazel fails if workspace is not on C: Oct 7, 2016
added P1 and removed P2 labels Oct 7, 2016
Contributor
 Bumping priority again because this is a serious blow on usability.
Contributor
 I have a fix for this, stay tuned.
Contributor
 Out for review. If all goes well, we can push the fix to GH today or tomorrow.
Contributor
 Hi @laszlocsomor - I didn't spot this on [gerrit|https://bazel.googlesource.com/bazel/]. Is it somewhere else? Happy to try it out, since it's the next thing to a blocker for us.
referenced this issue Oct 13, 2016
Closed

On Windows BAZEL_PYTHON is ignored while finding Python #1940

Contributor
 Hey @petemounce , it's not yet submitted, sorry about that. It should be in today or Monday, if not I'll ping this thread.
commented Oct 15, 2016
 seems java.exe path problem set JAVA_HOME=C:/Program\ Files/Java/jdk1.8.0_101&c:\bazel.exe Couldn't find java at 'C:/Program\ Files/Java/jdk1.8.0_101/bin/java'.
Contributor
commented Oct 15, 2016
 @youkpan this is not really related to this bug, is it? In the future, please file a separate issue if the problem is unrelated to the problem discussed in the thread. Anyhow, in your case the content of JAVA_HOME variable after your command is literally C:/Program\ Files/Java/jdk1.8.0_101/bin/java, including a \ before the space. The following should work: C:> set JAVA_HOME=C:/Program Files/Java/jdk1.8.0_101 C:> bazel.exe 
referenced this issue in tensorflow/tensorflow Oct 15, 2016
Closed

Windows Support and Documentation #17

Contributor
 For later reference, I'm able to work around this by having my source code inside a sub directory on my separate drive (s:\whatever\here) create a junction to that from inside a directory on c: mklink /j c:\src\s-drive s:\whatever\here
Contributor
 FYI I'm still working on this change, still not submitted.
referenced this issue Oct 17, 2016
Closed

Windows: Path.relativeTo is unaware of Windows paths #1827

Contributor
 Submitted internally, will push it to GitHub in the next 24 hours.
Contributor
 Looks like I made a mistake in WindowsFileSystem.java retrieving UNIX_ROOT (if you don't see that then the change is not on GitHub yet), but I sent out a fix for it. If I'm lucky we'll push both changes to GH in the same run.
Contributor
 If not, then in the meantime the workaround will be to pass --host_jvm_args=-Dbazel.windows_unix_root="C:/tools/msys64/" to bazel.
pushed a commit that referenced this issue Oct 18, 2016
 laszlocsomor + philwo VFS: implement a Windows-specific Path subclass The new subclass WindowsFileSystem.WindowsPath is aware of Windows drives. This change: - introduces a new factory for Path objects so FileSystems can return a custom implementation that instantiates filesystem-specific Paths - implements the WindowsPath subclass of Path that is aware of Windows drives - introduces the bazel.windows_unix_root JVM argument that defines the MSYS root, which defines the absolute Windows path that is the root of all Unix paths that Bazel creates (e.g. "/usr/lib" -> "C:/tools/msys64/usr/lib") except if the path is of the form "/c/foo" which is treated as "C:/foo" - removes all Windows-specific logic from Path PathFragment is still aware of drive letters and it has to remain so because it is unaware of file systems. WindowsPath restricts the allowed path strings to absolute Unix paths where the first segment, if any, is a volume specifier. From now on if Bazel attempts to create a WindowsPath from an absolute Unix path, Bazel will make it relative to WindowsPath.UNIX_ROOT, unless the first component is a single-letter name (e.g. "/c/foo" which is "C:/foo"). Subclassing Path is necessary because a Unix-style absolute path doesn't sufficiently define a full Windows path, as it may be relative to any drive. Fixes #1463 -- MOS_MIGRATED_REVID=136350304 e0d7a54
pushed a commit that referenced this issue Oct 18, 2016
 laszlocsomor + philwo VFS, WindowsFileSystem: fix UNIX_ROOT retrieval Executing bash.exe directly instead of through cmd.exe doesn't seem to work. This change fixes that problem. Fixes #1463 (again) -- MOS_MIGRATED_REVID=136364606 8bb4299
pushed a commit that referenced this issue Oct 19, 2016
 philwo Rollback of commit 8bb4299. *** Reason for rollback *** Suspected root cause for Windows bootstrap on Bazel CI breakage: java.lang.NullPointerException at com.google.devtools.build.lib.vfs.Path$1.run(Path.java:123) http://ci.bazel.io/view/Bazel%20bootstrap%20and%20maintenance/job/Bazel/922/JAVA_VERSION=1.8,PLATFORM_NAME=windows-x86_64/console *** Original change description *** VFS, WindowsFileSystem: fix UNIX_ROOT retrieval Executing bash.exe directly instead of through cmd.exe doesn't seem to work. This change fixes that problem. Fixes #1463 (again) -- MOS_MIGRATED_REVID=136581532 4b4b9d3 reopened this Oct 20, 2016 Contributor  I figured out why the bootstrapping job was broken and why we had to roll this change back. The WindowsPath.translateParent call returned a different parent for top-level directories (UNIX_ROOT instead of /), but we registered the child with the old parent, not the new one. Contributor  Fix is out for review, stay tuned. pushed a commit that closed this issue Oct 25, 2016  laszlocsomor + katre VFS: implement a Windows-specific Path subclass This change rolls forward commit e0d7a54 and commit 8bb4299, and adds a bugfix: - FileSystem.PathFactory got a new translatePath method that WindowsFileSystem.PathFactory overrides to translate absolute Unix paths to MSYS-relative paths - Path.getCachedChildPath calls this translatePath method so the child path is registered with the correct (translated) parent and under the correct name (e.g. "C:" instead of say "c") Below is the rest of the original change description: The new subclass WindowsFileSystem.WindowsPath is aware of Windows drives. This change: - introduces a new factory for Path objects so FileSystems can return a custom implementation that instantiates filesystem-specific Paths - implements the WindowsPath subclass of Path that is aware of Windows drives - introduces the bazel.windows_unix_root JVM argument that defines the MSYS root, which defines the absolute Windows path that is the root of all Unix paths that Bazel creates (e.g. "/usr/lib" -> "C:/tools/msys64/usr/lib") except if the path is of the form "/c/foo" which is treated as "C:/foo" - removes all Windows-specific logic from Path PathFragment is still aware of drive letters and it has to remain so because it is unaware of file systems. WindowsPath restricts the allowed path strings to absolute Unix paths where the first segment, if any, is a volume specifier. From now on if Bazel attempts to create a WindowsPath from an absolute Unix path, Bazel will make it relative to WindowsPath.UNIX_ROOT, unless the first component is a single-letter name (e.g. "/c/foo" which is "C:/foo"). Subclassing Path is necessary because a Unix-style absolute path doesn't sufficiently define a full Windows path, as it may be relative to any drive. Fixes #1463 -- MOS_MIGRATED_REVID=137149483 ca99bb7 closed this in ca99bb7 Oct 25, 2016 Contributor  @laszlocsomor Thanks! Just tried this out with 0.4.0-rc3; works great. Contributor  @laszlocsomor Actually, weirdly, this just failed from within powershell and msys2. I'm not sure how I tested, above, coming back to this. λ bazel build //Cpp/WorkerSdk:WorkerSdk ..... ERROR: in target '//external:jdk': no such package '@local_jdk//': com.google.devtools.build.lib.skyframe.InconsistentFilesystemException: Inconsistent filesystem operations. C:/Program Files/Java/jdk1.8.0_102 is no longer an existing directory. Did you delete it during the build? The results of the build are not guaranteed to be correct. You should probably run 'blaze clean' and investigate the filesystem inconsistency (likely due to filesytem updates concurrent with the build). INFO: Elapsed time: 0.234s s:\everything [feature/WIT-761-bazel-for-cpp-on-windows ≡ +2 ~2 -0 !] λ bazel version Build label: 0.4.0rc3 Build target: bazel-out/local-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar Build time: Wed Oct 26 13:40:38 2016 (1477489238) Build timestamp: 1477489238 Build timestamp as int: 1477489238  Contributor  @petemounce : Indeed, the 0.4.0-rc3 doesn't contain this change. I assume it was rolled back at the time we cut the canary; we had to roll it back and forward a couple of times. It works for me at 9b61b0f, both under msys and powershell. For powershell I have to set some envvars and bazel prints some errors -- i don't know if it's normal, i never used bazel under PS: PS C:\work\bazel2>$global:PATH = $env:Path PS C:\work\bazel2>$global:BAZEL_SH = "c:\tools\msys64\usr\bin\bash.exe" PS C:\work\bazel2> cd d:\bazel_ws PS D:\bazel_ws> C:/tools/msys64/var/tmp/Bazel/MGAgSgPx/execroot/bazel2/bazel-out/local-fastbuild/bin/src/bazel.exe --batch info --color=no --curses=no ERROR: 'null' is not recognized as an internal or external command, operable program or batch file. (exit code: 1) WARNING: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/external/bazel_tools/tools/cpp/cc_configure.bzl:57:3: Auto-Configuration Warning: 'BAZEL_PYTHON' is not set, start looking for python in PATH. WARNING: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/external/bazel_tools/tools/cpp/cc_configure.bzl:57:3: Auto-Configuration Warning: Python found at C:/Python27/python.exe WARNING: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/external/bazel_tools/tools/cpp/cc_configure.bzl:57:3: Auto-Configuration Warning: 'BAZEL_VS' is not set, start looking for the latest Visual Studio installed. WARNING: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/external/bazel_tools/tools/cpp/cc_configure.bzl:57:3: Auto-Configuration Warning: 'BAZEL_SH' is not set, start looking for bash in PATH. WARNING: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/external/bazel_tools/tools/cpp/cc_configure.bzl:57:3: Auto-Configuration Warning: Bash binary found at C:/tools/msys64/usr/bin/bash.exe WARNING: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/external/bazel_tools/tools/cpp/cc_configure.bzl:57:3: Auto-Configuration Warning: Visual Studio found at C:\Program Files (x86)/Microsoft Visual Studio 14.0 bazel-bin: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/execroot/bazel_ws/bazel-out/local-fastbuild/bin bazel-genfiles: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/execroot/bazel_ws/bazel-out/local-fastbuild/genfiles bazel-testlogs: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/execroot/bazel_ws/bazel-out/local-fastbuild/testlogs command_log: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/command.log committed-heap-size: 301MB execution_root: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/execroot/bazel_ws gc-count: 4 gc-time: 59ms install_base: C:/tools/msys64/var/tmp/_bazel_laszlocsomor/install/e456d4f6cf0d1a673f1ea38bb8525ecf max-heap-size: 7621MB message_log: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/message.log output_base: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n output_path: C:/tools/msys64/var/tmp/Bazel/WLSiXh7n/execroot/bazel_ws/bazel-out package_path: %workspace% release: development version server_pid: 1644 used-heap-size: 73MB workspace: D:/bazel_ws 
referenced this issue Oct 31, 2016
Closed

October release #1902

Contributor
commented Oct 31, 2016 edited
 @laszlocsomor in the docs, BAZEL_SH wants path separators to be / not \. That's what my own var is set with. Also, I set those via $env:BAZEL_SH if I do them manually within a powershell, rather than $global:BAZEL_SH`.
referenced this issue in tensorflow/tensorflow Nov 22, 2016
Open