python configure fails on Windows if "bash on Ubuntu on Windows" installed

[snnn]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Windows 10, with "bash on Ubuntu on Windows" installed
• TensorFlow installed from (source or binary):
  source
• TensorFlow version (use command below):
  ec808c9
• Python version:
  3.5.3
• Bazel version (if compiling from source):
  9e62187df84ae425a5d7226b6baf1bef576f0a10
• CUDA/cuDNN version:
  None
• GPU model and memory:
  None
• Exact command to reproduce:
  bazel --output_base C:\t clean --expunge
  bazel --output_base C:\t build --features generate_pdb_file --action_env=USE_DYNAMIC_CRT=1 --action_env=NO_MSVC_WRAPPER=1 --color=no --compilation_mode fastbuild --verbose_failures --experimental_ui --copt=/Z7 --linkopt=/DEBUG:FASTLINK --copt=/DNDEBUG --host_copt=/DNDEBUG //tensorflow/tools/pip_package:build_pip_package

Describe the problem

Build tensorflow on windows with bazel may fail because "bash on Ubuntu on Windows" is installed
It may invoke the wrong "bash" for executing these commands.

Related: bazelbuild/bazel#3445

Source code / logs

Analyzing: target //tensorflow/tools/pip_package:build_pip_package
ERROR: C:/os/tensorflow/third_party/py/numpy/BUILD:11:1: no such package '@local_config_python//': Traceback (most recent call last):
        File "C:/os/tensorflow/third_party/py/python_configure.bzl", line 310
                _create_local_python_repository(repository_ctx)
        File "C:/os/tensorflow/third_party/py/python_configure.bzl", line 269, in _create_local_python_repository
                _check_python_bin(repository_ctx, python_bin)
        File "C:/os/tensorflow/third_party/py/python_configure.bzl", line 225, in _check_python_bin
                _python_configure_fail("PYTHON_BIN_PATH is not executab...")
        File "C:/os/tensorflow/third_party/py/python_configure.bzl", line 37, in _python_configure_fail
                fail(("%sPython Configuration Error:%...)))
Python Configuration Error: PYTHON_BIN_PATH is not executable.  Is it the python binary?

[gunan]

This is due to bazel. Please see this page in bazel website:
https://docs.bazel.build/versions/master/windows.html

Similarly, if you have bash on Ubuntu on Windows installed, you should make sure c:\tools\msys64\usr\bin appears in PATH before c:\windows\system32, because otherwise Windows' bash.exe is used before msys2's.

[snnn]

Hi @gunan ,

I set that.
This is how did I set the env vars before running bazel command

set PATH=C:\Tools\msys64\usr\bin;C:\jdk\bin;C:\python35;C:\python35\scripts;C:\bazel;%PATH%
set BAZEL_SH=C:/Tools/msys64/usr/bin/bash.exe
set JAVA_HOME=C:/jdk
set BAZEL_PYTHON=C:/python35/python.exe
set PYTHON_BIN_PATH=C:\python35\python.exe
set PYTHON_LIB_PATH=C:\python35\lib\site-packages
set BAZEL_VS=C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise

[gunan]

@meteorcloudy could you take a look?

…

On Jul 25, 2017 11:57 AM, "Changming Sun" ***@***.***> wrote: Hi @gunan <https://github.com/gunan> , I set that. This is how did I set the env vars before running bazel command set PATH=C:\Tools\msys64\usr\bin;C:\jdk\bin;C:\python35;C:\python35\scripts;C:\bazel;%PATH% set BAZEL_SH=C:/Tools/msys64/usr/bin/bash.exe set JAVA_HOME=C:/jdk set BAZEL_PYTHON=C:/python35/python.exe set PYTHON_BIN_PATH=C:\python35\python.exe set PYTHON_LIB_PATH=C:\python35\lib\site-packages set BAZEL_VS=C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11735 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHlCOZviDdzGL-1cL1y36XiSCAWBO-lLks5sRa37gaJpZM4OiDt3> .

[meteorcloudy]

@gunan No problem, I'll reply in the corresponding bazel issue.

[snnn]

How about this:
add a function

def get_shell():
    if os.name == 'Windows'
         return env['BAZEL_SH']
    else:
         return 'bash'

[meteorcloudy]

I think the best solution would be avoiding invoking bash explicitly.
We can write the python code in a file first instead of using bash -c "python - <<END <content> END" syntax, then we can just do repository_ctx.execute(['python', "./find_python_lib.py"])
example:

def test_bash(repository_ctx):
  """Gets the python lib path."""
  print_lib = (
      "from __future__ import print_function\n" +
      "import site\n" +
      "import os\n" +
      "\n" +
      "try:\n" +
      "  input = raw_input\n" +
      "except NameError:\n" +
      "  pass\n" +
      "\n" +
      "python_paths = []\n" +
      "if os.getenv('PYTHONPATH') is not None:\n" +
      "  python_paths = os.getenv('PYTHONPATH').split(':')\n" +
      "try:\n" +
      "  library_paths = site.getsitepackages()\n" +
      "except AttributeError:\n" +
      " from distutils.sysconfig import get_python_lib\n" +
      " library_paths = [get_python_lib()]\n" +
      "all_paths = set(python_paths + library_paths)\n" +
      "paths = []\n" +
      "for path in all_paths:\n" +
      "  if os.path.isdir(path):\n" +
      "    paths.append(path)\n" +
      "if len(paths) >=1:\n" +
      "  print(paths[0])\n")
  repository_ctx.file("find_python_lib.py", print_lib)
  result = repository_ctx.execute(['python', "./find_python_lib.py"], quiet=False)
  print("STDOUT: " + result.stdout)
  print("STDERR: " + result.stderr)
  if result.return_code == 1:
    print('error')
  print('end')
  
python_configure = repository_rule(
    implementation = test_bash,
    environ = ["PATH"]
)

[snnn]

Hi @meteorcloudy , Do you have any future plans of this? Could you please also fix bazelbuild/bazel#2675 , which breaks http://ci.tensorflow.org/job/tf-master-win-bzl/ for a long time.

[meteorcloudy]

Hi @snnn
I'll send a patch to fix this problem in python_configure.bzl using the solution I mentioned above.

As for bazelbuild/bazel#2675, we really want to fix it, but it's not reproducible at all, so we haven't had any clue to proceed.. And which failure exactly do you think is caused by it?

[snnn]

Hi @meteorcloudy
http://ci.tensorflow.org/job/tf-master-win-bzl/1425/console
14:54:19 external/protobuf_archive/python/google/protobuf/internal/api_implementation.cc(31): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

http://ci.tensorflow.org/job/tf-master-win-bzl/1422/console
18:50:31 c:\tmp_bazel_system\424zmya1\execroot\org_tensorflow\bazel-out\msvc_x64-py3-opt\genfiles\external\local_config_python\numpy_include\numpy\npy_common.h(5): fatal error C1083: Cannot open include file: 'numpyconfig.h': No such file or directory

http://ci.tensorflow.org/job/tf-master-win-bzl/1421/console
17:18:10 bazel-out/msvc_x64-py3-opt/genfiles/external/local_config_python/python_include\Python.h(70): fatal error C1083: Cannot open include file: 'typeslots.h': No such file or directory

They failed because the _read_dir function in third_party/py/python_configure.bzl returned an incomplete file list. Currently I've no idea why it happens so often in CI systems, but , it's not reproducible on my local computer!

[meteorcloudy]

@snnn , I see, thanks for pointing it out.

[gunan]

Any updates here?
Were we able to come to a resolution?

[meteorcloudy]

There are actually two problems mentioned here.

1. As the title says python configure fails when "bash on Ubuntu on Windows" installed. For this, I had a solution but forgot to implement it, sorry about that, will fix it soon.

2. C1083: Cannot open include file: 'Python.h': No such file or directory
   This is a flaky error happens a lot on tf-master-win-bzl, fortunately the root cause has been identified and fixed, will be rollout in Bazel 0.8.0.

[tensorflowbutler]

It has been 14 days with no activity and this issue has an assignee.Please update the label and/or status accordingly.

[tensorflowbutler]

It has been 14 days with no activity and this issue has an assignee.Please update the label and/or status accordingly.

[gunan]

We are now upgraded to bazel 0.8, which should fix the 2nd issue.
@meteorcloudy any updates on (1)?

[tensorflowbutler]

Nagging Assignee: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[tensorflowbutler]

Nagging Assignee: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[tensorflowbutler]

Nagging Assignee: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[tensorflowbutler]

Nagging Assignee @meteorcloudy: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[tensorflowbutler]

Nagging Assignee @meteorcloudy: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[tensorflowbutler]

Nagging Assignee @meteorcloudy: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[tensorflowbutler]

Nagging Assignee @meteorcloudy: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[tensorflowbutler]

Nagging Assignee @meteorcloudy: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[tensorflowbutler]

Nagging Assignee @meteorcloudy: It has been 141 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[gunan]

Just checking again. Any updates?

[meteorcloudy]

So sorry for the long delay here, sent a fix for this.