Build is failing with latest RIOT on windows

[bluephoton]

I've been building on windows 10 with no issue until I decided to get latest RIOT. I'm getting the error:

/usr/bin/env: python3: No such file or directory

I have both python and python3 on my PATH and can issue these commands from build cmd prompt with no issue but still build fails.

I noticed that this is due something called lazysponge.py which is invoked early in the build process.

Any Ideas on how to deal with this error in windows?

[cladmi]

Hi @bluephoton at least it means there is not that much missing for building in windows :)

I am not used to windows and the way shebangs are handled, so I will try some debug things I am thinking about.
Can you give me the output of these commands:

which env
which python3
env env
env which python3
echo foo | dist/tools/lazysponge/lazysponge.py  -v out; cat out

[jcarrano]

Is env able to find the tool?, i.e. if you type:

env python3

in the mingw terminal.

Also, can you post the result of running ./dist/tools/ci/print_toolchain_versions.sh

I don't have a windows machine now, but I'll try to procure a VM around these days.

[cladmi]

@vac did you by any chance had the same issue and solved it ? Thank you in advance.

[vac]

I had problems with the very same file. I'm using python27. I've changed this file to be python2 compatible. It is working fine since then. It is not the best solution and I haven't merged into the latest RIOT since 2018-04 so I will paste just git diff instead of PR, sorry about that:

diff --git a/dist/tools/lazysponge/lazysponge.py b/dist/tools/lazysponge/lazysponge.py
index 6bfefef79..4d088c4a1 100755
--- a/dist/tools/lazysponge/lazysponge.py
+++ b/dist/tools/lazysponge/lazysponge.py
@@ -1,5 +1,7 @@
-#! /usr/bin/env python3
+#! /usr/bin/env python
+# coding: utf-8
 
+from __future__ import print_function
 #
 # Copyright (C) 2018 Gaëtan Harter <gaetan.harter@fu-berlin.de>
 #
@@ -84,10 +86,10 @@ def main():
     try:
         with open(opts.outfile, 'rb') as outfd:
             oldbytes = outfd.read()
-    except FileNotFoundError:
+    except:
         oldbytes = None
 
-    stdinbytes = sys.stdin.buffer.read()
+    stdinbytes = sys.stdin.read()
     if opts.verbose:
         _print_hash_debug_info(opts.outfile, oldbytes, stdinbytes)
 

Hope it will help you.

[bluephoton]

Thanks guys for helping with this! Here is the info you requested:

Command	Result
which env	C:\MinGW\msys\1.0\bin\env.exe
which python3	python3 is an alias for C:\WinPython-64bit-3.5.4.1Qt5\python-3.5.4.amd64\python.exe
env env	too much to share here, any specific setting?
env which python3	which: python3: unknown command
which python3	no such file or directory
echo foo | .\dist\tools\lazysponge\lazysponge.py -v out.txt & cat out.txt	Replacing out.txt ( != de3191a6a40ae2c4810504f2af6f4625) foo
./dist/tools/ci/print_toolchain_versions.sh	launches another window and not able to get its output but seems to work

Notes:

• I use normal windows cmd prompt (not mingw, but tried it and got same result)
• I don't have 'which', so I used 'where' instead.
• replaced '/' with '' in the above to work
• I tried to add python3.cmd (just launched python.exe) to my environment and didn't help
• I can run python and python3 from my cmd prompt but build script can't see it seems.

Question:

Is there harm if I commented this lazy spongy thingy for now? 😄 what does it do?

Let me know if you have more questions...

Edit:
Thanks @vac for sharing the diff (just noticed your response!) The file works fine for me if I ran it from cmd prompt, I'm just having hard time getting python3 to be recognized! I don't have 2.7.

[cladmi]

What's is somehow strange, is that it worked outside of make as this one worked

echo foo | .\dist\tools\lazysponge\lazysponge.py -v out.txt & cat out.txt

But not from within make.

Somehow the fact that python3 is an alias for C:\WinPython-64bit-3.5.4.1Qt5\python-3.5.4.amd64\python.exe which is not named python3 could be part of the issue but not too sure how to debug this.

[vac]

On Windows please use where python to locate executables.
If which is available in your Windows command line it probably comes from Cygwin or MinGW. Not sure about that but environment from within Cygwin or MinGW executables can differ from system one.

Additionally, check make --version.

I'm also wondering if your MinGW doesn't contain its own python?
So maybe when you are executing MinGW make your MinGW python overrides the system one.

[bluephoton]

Ok, it turns out I must have the python.exe in my global environment not user environment.
Also had to follow @vac approach and change #! /usr/bin/env python3 to #! /usr/bin/env python
since python.exe for me is actually python3, I didn't need more changes.

Tried to use alias and python3.cmd to avoid changing RIOT code but none worked. will be nice if we change it in RIOT to just python if linux can tolerate this change.

I can compile now - minding few breaking changes I need to deal with!

Thanks a lot guys for your help!!

[vac]

Indeed, there is no python3, nor python2 on Windows. Python 2.7, Python 3.6, and the latest Python 3.7 include python.exe only. So sadly, there is no way to make python3 command work on default Python3 windows installation without hacks...

@bluephoton : what about making a copy of your Python3\python.exe named python3.exe ?

[bluephoton]

@vac smart! 😄 this works!!

[miri64]

Instead of copying you could do what most Linux distributions do: create a symbolic link. This has the advantage that the binary will still be the same even after an update ;-).

[bluephoton]

@miri64 is that possible on windows? will that link continue to exist even after closing and opening cmd prompt?

[miri64]

Yepp. At least the last time I used Windows NTFS supported symbolic links ;-) https://www.howtogeek.com/howto/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/

[bluephoton]

Cool!! will give it a try, thanks @miri64 for sharing the link!!

[bluephoton]

Symbolic link worked! good stuff! thanks @miri64

mklink python3.exe python.exe

[miri64]

Though this seems to be more a patchy fix, I'd say we close this. Should we document this somewhere?

[jcarrano]

Should we document this somewhere?

On the windows setup guide?
BTW, I will try to follow the guide with a clean windows install in a VM, see if it is still accurate.

[jcarrano]

@vac , @bluephoton

There is a way of fixing it that does not require creating a link (since creating symlinks in windows requires elevated privileges), by making a wrapper in msys.

In the msys prompt type:

cat > /mingw/bin/python3 <<SCRIPT
#!/bin/sh
exec python "\$@"
SCRIPT

[bluephoton]

@jcarrano , for windows 10 Pro I have, I didn't need elevated command prompt to create symbolic link. It may be needed for server SKUs, not sure.