ndk-build unable to build shaderc on Windows due to outdated Python version

[pumexx]

It looks like ndk-build ignores python from PATH / PYTHONPATH / PYTHONHOME environment variables and uses python 2.7.5 from ndk-bundle\prebuilt\windows-x86_64\bin directory
( that's a really outdated python version, to be honest )

As a result - shaderc compilation fails with message "ImportError: No module named _struct".

When I remove python*.exe from ndk-bundle\prebuilt\windows-x86_64\bin directory and use my own python 2.7.15 - shaderc is able to compile.

[ggfan]

Got the machine and tried with the repo and it seems to be ok -- the python coming with NDK is sufficient to get the samples built and run on device. Here is what I did from this fresh machine ( nothing installed, no python at all. ):

• install Android Studio, SDK, NDK, CMake, LLDB ( all within IDE )
• build shaderc with

C:\sdk\ndk-bundle\sources\third_party\shaderc>..\..\..\ndk-build.cmd NDK_PROJECT_PATH=. APP_BUILD_SCRIPT=Android.mk APP_STL:=c++_static APP_ABI=all libshaderc_combined -j16

it built through without problem.

• Import samples from Android Studio ( within IDE ) File > New > Import Samples > Ndk > Vulkan API samples
• wait for it to finish ( or complain and want to update to the latest Gradle plugin version and install necessary platform tools, just let AS do whatever it needs to )
• Build > App Bundle / APK > Build APKs ( wait for it to complete )
• Select project 15-draw_cube in IDE, and click the "Play" button ( of course I would have a device connected via USB already now )
  I see the cube is rendered on Pixel3 phone running Pie GSI image ( GSI is an accelerated Emulator )

Also tried to get the latest shaderc code to this windows machine and build is fine too:

• get source code from https://github.com/google/shaderc.git
• follow the instruction on the github page to update dependencies

c:\sdk\ndk-bundle\prebuilt\windows-x86_64\bin\python.exe  git-sync-deps
then build it with:
c:\sdk\ndk-bundle\ndk-build.cmd NDK_PROJECT_PATH=. APP_BUILD_SCRIPT=Android.mk APP_STL:=c++_static APP_ABI=arm64-v8a libshaderc_combined -j16

it built fine too.

Thank you for raising the issue

• the python version is old ( although it is sufficient to build shaderc inside NDK ), I will try to communicate to NDK owner or you could help to file an issue at https://github.com/android-ndk/ndk/issues ( let me know you do it -- then I will not do it anymore )
• The doc on DAC needs to be re-visited ( should take a stance whether we recommend to use IDE or get the repo manually; and no need to update the gslang anymore), I will give it a shot and see how process goes.

Question for you: may you clarify your error coming from which point?

thank you!

[ggfan]

@pumexx
For doc, internal change is at: cl/228002567; for python update request, b/122417465.
I think I am almost done with this one, please have a look if you could access the internal links.

One thing, seems ndk 19 beta has some problem building shaderc, so use ndk-r18 instead for now ( internal ticket b/122414666)

[pumexx]

Thank you for performing the test.
You mentioned making the test on a "fresh machine" while my own computer is far from being fresh, so I started digging.

Earlier I wrote that I was able to build shaderc when I removed python*.exe from NDK ( it means that I was able to build shaderc with my own Python 2.7.15 )

Now I was also able to build shaderc with Python shipped with Android NDK ( Python 2.7.5 ) simply by setting these environment variables :

set PATH=c:\Windows\System32

set PYTHONPATH=

set PYTHONHOME=

So the problem is that NDK mixed different Python versions due to existing environment variables.
Maybe these variables ( PYTHONHOME, PYTHONPATH ) should be supressed while ndk-build is running ?

Otherwise documentation should mention something like "Ensure that no other Python version is able to run while building shaderc".

[ggfan]

would like to see it happen before make a note to doc team, what is your installed python version, besides NDK? The detailed steps to see it would be good.

[ggfan]

Oh, NDK-r19 formal release will fix the issue noted above, please refer to
android/ndk#884

Thank you for raising this issue: it helped making a better NDK :-)

[pumexx]

Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)] on win32
I did not save the the error message, but as far as I remember it happened when Python JSON parser was included. I may try to reproduce once more if you like.

[ggfan]

Thank you so much! may you kindly confirm win32 or win64 for your OS? thank you!

[pumexx]

OK, so I ran a command and got shaderc_err.txt file ( file is attached ):

......\ndk-build.cmd NDK_PROJECT_PATH=. APP_BUILD_SCRIPT=Android.mk APP_STL:=c++_static APP_ABI=all libshaderc_combined -j16 2> shaderc_err.txt

shaderc_err.txt

System : Windows 7 SP1, 64-bit

Environment variables:
PYTHONHOME=c:\Python27
PYTHONPATH=c:\Python27\DLLs;c:\Python27\Lib;c:\Python27\Lib\site-packages
Path=c:\Python27;c:\Python27\Scripts;( and here goes the rest of the PATH variable )

[ggfan]

thank you again for the update! Unfortunately I could not see the failure after installing 2.7.15, and manually setting up the environment variables ( including PATH ). Mine is Windows 10 64 bit version.

I am running out of the ways to see it, hope you could do further to debug it on your system, please kindly let us know what you findings. Thank you!

[ggfan]

The latest Vulkan API samples switched to pre-compiled shaders, the compiling of the shaderc inside NDK is not that necessary anymore: check out https://github.com/LunarG/VulkanSamples/blob/master/BUILD.md#building-on-android