XLA/AOT Windows support

[rongjiecomputer]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 x64
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): master
• Python version: 3.6
• Bazel version (if compiling from source): 0.8.0
• GCC/Compiler version (if compiling from source): VS 2017 15.5
• CUDA/cuDNN version: N/A
• GPU model and memory: N/A
• Exact command to reproduce: N/A

Similar to #8310, but specifically about running tfCompile on Windows rather than Linux to produce x86_64-windows-msvc binaries.

XLA/AOT depends on LLVM which has excellent Windows support via CMake, but Bazel cannot interop with CMake. llvm.BUILD is auto-generated and the script to generate it is not open-sourced, this make it difficult for external contributor to make improvement. tensorflow/compiler might not need too much changes as #9908 already addressed some of them.

One possible path is to let user to run CMake in host machine when invoking configure.py, then feed CMake generated files into custom script to generate LLVM.BUILD.

Note:

Rumour has it that there is a Google-internal tool called tfNative to generate .h/.cpp files instead of .lib binaries, though I suspect that even if the tool is open-sourced, it might not be immediately available for Windows developers.

[reedwm]

/CC @tatatodd

[argman]

@rongjiecomputer seems you can compile tensorflow with bazel on windows, can you share how you get that?

[rongjiecomputer]

@argman Building Tensorflow with Bazel on Windows should work out of the box now.

1. Download bazel-0.8.1-windows-x86_64.exe from https://github.com/bazelbuild/bazel/releases, rename it to bazel.exe and put it in PATH.
2. set BAZEL_VS or BAZEL_VC to point to your VS 2017 installation directory (see https://docs.bazel.build/versions/master/windows.html)
3. Clone Tensorflow.
4. Run configure.py (see https://www.tensorflow.org/install/install_sources).
5. Open .tf_configure.bazelrc and remove -march=native (otherwise the console output will be very noisy).
6. Set TMP, TEMP and TMPDIR to shorter name like C:\tmp due to Resolve Windows long path issue during C++ compilation bazelbuild/bazel#4149.
7. Run bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package.

If the build seems to stuck at compiling conv_grad_ops_3d.cc and conv_ops_3d.cc, don't worry about it. These two files are known to take a very long time to compile in MSVC due to __force_inline. The compilation will eventually finish. See #10521.

[rongjiecomputer]

@tatatodd I managed to build tfcompile with Bazel in Windows and build simple XLA/AOT examples.

First two PRs are at #15310 and #15335, but no reviews received. Can you help me to assign them to suitable reviewers?

I can only send the third PR when #15229, #14893 and #14531 get merged (not by me).

Once these PRs are merged, I will post a guide on how to write new BUILD file to import pre-built LLVM binaries built with CMake into Bazel to build tfcompile with Bazel. I don't intend to touch llvm.BUILD since it is marked as "Do not edit".

Caveat:

tf_library macro from tfcompile.bz can't be used as labels with prefix of @org_tensorflow will cause Bazel to generate path name that exceeds Windows path length limit (See bazelbuild/bazel#4149). Workaround is to write another macro without @org_tensorflow prefix and build targets in-tree.

[tatatodd]

@rongjiecomputer Thanks very much for all the work on Windows! I've responded to the outstanding PRs that I know of; feel free to ping if I've missed any.

[tensorflowbutler]

It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

[tatatodd]

Marking as community support, since @rongjiecomputer has already been sending many great PRs.

[rongjiecomputer]

Status update:

Most of the changes needed to make tfcompile built and run on Windows have been merged. As promised, I made a repo with all the instructions to build and use XLA/AOT on Windows. (Contributions are welcomed to improve my silly MNIST C++ demo).

https://github.com/rongjiecomputer/tensorflow-xla-aot-windows

There are some obstacles that prevent XLA/AOT to work out of the box for Windows users:

• Add a wrapper for cc_library: tf_cc_library #15466 : Tensorflow team and Bazel team are still discussing about how to set global compile flags, a solved problem in build system like CMake.
• Resolve Windows long path issue during C++ compilation bazelbuild/bazel#4149 : Bazel tends to generate long path name, sometime it exceed the max path length on Windows. Changes to Bazel's design are needed to fix this.
• Bazel runfiles on Windows (used by freeze_graph target). Waiting Resolve Windows long path issue during C++ compilation bazelbuild/bazel#4149 and [WIP] Add bazel runfiles manifest support for Windows #15475.

It should be possible to build components of XLA/AOT with CMake, but unfortunately Tensorflow's CMake build scripts are written specifically for MSBuild. It just won't work with Ninja which I prefer. Any CMake expert is welcomed to bring XLA/AOT to CMake so that both LLVM and Tensorflow can be built together in CMake directly.

@tatatodd @hawkinsp

llvm.BUILD is auto-generated and the script to generate it is not open-sourced.

Any thoughts on this? My handcrafted BUILD.bazel is quite similar to the auto-generated llvm.BUILD. Since its purpose is to import prebuilt libraries, headers and some defines to Tensorflow, I think it is possible to make it works on Linux and Mac OS as well (the only problem is I don't have machines to test).

Please do not close this issue yet. It is far from being "fixed".

[tellarin]

@rongjiecomputer @tatatodd @hawkinsp Any thoughts on possible next steps?

[rongjiecomputer]

@tellarin There was an issue in highwayhash in rongjiecomputer/tensorflow-xla-aot-windows#1 (comment) but I have not sent out the PR yet.

I have not build XLA/AOT on Windows for awhile, so not sure if current master can be built with MSVC or not. (internal team occasionally write some template code that behaves differently with MSVC)

[cg31]

It seems the function AllOf(...) in pattern_matcher.h doesn't compile with MSVC (in VS2017 or VS2019) or LLVM 8.

[rongjiecomputer]

I use clang-cl with MSVC and -fno-delayed-template-parsing flag to compile Tensorflow. If you use /permissive- flag with pure MSVC from latest update of VS2017/VS2019, it might work too.

[cg31]

/permissive- did the trick, the code can be compiled, thanks.

[ListsOfArrays]

In VS2019 /permissive does not work as of this post & master branch commit bd67e65 .
"tensorflow/compiler/xla/service/gpu/hlo_algorithm_blacklist.cc(28): error C2131: expression did not evaluate to a constant". This was with /O2 and /arch:AVX2 options.

[tilakrayal]

@rongjiecomputer,
We could see that multiple PR's were raised and also merged. Also there are similar issues raised by the other community members for the similar feature and those were resolved. #35983

tf_library(
    name = "test_graph_tfmatmul",
    testonly = 1,
    config = "test_graph_tfmatmul.config.pbtxt",
    cpp_class = "foo::bar::MatMulComp",
    graph = "test_graph_tfmatmul.pb",
    tags = [
        "manual",
    ],
    tfcompile_flags = "--target_triple=x86_64-pc-windows"
)

Please have a look and move this issue to closed status, if the issue was resolved. Thank you!

[github-actions]

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further.