CMake on Windows: Unescaped backslashes

[jensboe]

Hi,
I am using Windows and CMake and just start using modm. If I like modm I will start a PR to fix this issue by myself but for now I just reporting.

The generated repo.cmake contains unescaped backslashes.
${CMAKE_CURRENT_LIST_DIR}/src\modm\architecture\driver\atomic\flag.cpp

CMake on Windows accept slashes or escaped backslashes. I don't know the common preferences but i preferer normal slashes.

[jensboe]

The set of CCFLAGS makes also trouble.
After removing the, after removing file-prefix-map a can create hex files.

-ffile-prefix-map=${MODM_GCC_PATH}=. \
# -ffile-prefix-map=${CMAKE_CURRENT_SOURCE_DIR}=. \

[salkinium]

Hm, Windows support is difficult to get right…

There is a new CMake generator, see #568. Perhaps it solves some of these issues.

[jensboe]

It only generates other issues.
I hate CMake. I only use it because the working interaction between VSCode Intellisense and the compiler switches.
I think it's easier to rewrite some code(year, python) for sync between VSCode and SCons(which works) or switching to WSL2.

[rleh]

VSCode Intellisense

For me VSCode works great (with the C++/Intellisense plugin) using SCons (or even without a build system) if I open the project folder (with the project.xml file) in VSCode. I then use the build-in terminal for compiling, programming and debugging...

[salkinium]

I'm sorry for your troubles, I also dislike CMake a lot…

Perhaps you can try to import this project, which I've just tested to work on VSCode with CMake (but on macOS):
https://github.com/modm-ext/modm_starter_project

[jensboe]

#568 was already merged into main develop so no need to switch to the pr.

I use following setitngs:

    <extends>modm:nucleo-f446ze</extends>
    <options>
        <option name="modm:build:build.path">build</option>
        <option name="modm:build:cmake:include_cmakelists">true</option>
        <!-- <option name="modm:build:cmake:include_makefile">false</option> -->
    </options>
    <modules>
        <module>modm:build:cmake</module>
        <module>modm:platform:itm</module>
        <module>modm:docs</module>
    </modules>

CMake wasn't able to find python3 find_program(PYTHON3_EXECUTABLE python3 REQUIRED) (from .\modm\cmake\ModmConfiguration.cmake) trying around with python, python.exe, the real path, real path in qoutes etc doesn't work out. Finally i comment it out.

Than again unescaped backslashes in .\modm\CMakeLists.txt I simply convert them into normal slashes.

Building and linking works.
elf and other build artefacts are made.

After that a "open file with" dialog opens up and and after selecting a program, the cmd says failed.

I think the call of size.py isn't correct.

I will investigate further and will report and/or create a PR.

Anonymous log:

[build] [2/2 100% :: 1.442] Linking CXX executable projectname.elf
[build] FAILED: projectname.elf 
[build] cmd.exe /C "cd . && C:\PROGRA~2\GNUARM~1\102020~1\bin\AR10B2~1.EXE -g --specs=nano.specs --specs=nosys.specs
-LC:/abs/path/to/project/root -nostartfiles
-Tmodm/link/linkerscript.ld -Wl,--build-id=sha1 -Wl,--fatal-warnings -Wl,--gc-sections -Wl,--no-wchar-size-warning -Wl,--relax -Wl,-Map,projectname.map,--cref -mcpu=cortex-m4
-mfloat-abi=hard -mfpu=fpv4-sp-d16 -mthumb modm/CMakeFiles/modm.dir/ext/gcc/cabi.c.obj modm/CMakeFiles/modm.dir/ext/gcc/cxxabi.cpp.obj
modm/CMakeFiles/modm.dir/ext/gcc/new_delete.cpp.obj
modm/CMakeFiles/modm.dir/src/modm/architecture/driver/atomic/flag.cpp.obj modm/CMakeFiles/modm.dir/src/modm/board/board.cpp.obj
modm/CMakeFiles/modm.dir/src/modm/io/iostream.cpp.obj modm/CMakeFiles/modm.dir/src/modm/io/iostream_printf.cpp.obj
modm/CMakeFiles/modm.dir/src/modm/math/utils/bit_operation.cpp.obj modm/CMakeFiles/modm.dir/src/modm/math/utils/pc/operator.cpp.obj
modm/CMakeFiles/modm.dir/src/modm/platform/clock/rcc.cpp.obj modm/CMakeFiles/modm.dir/src/modm/platform/clock/systick_timer.cpp.obj
modm/CMakeFiles/modm.dir/src/modm/platform/core/assert.cpp.obj modm/CMakeFiles/modm.dir/src/modm/platform/core/delay.cpp.obj
modm/CMakeFiles/modm.dir/src/modm/platform/core/no_heap.c.obj modm/CMakeFiles/modm.dir/src/modm/platform/core/reset_handler.sx.obj
modm/CMakeFiles/modm.dir/src/modm/platform/core/startup.c.obj modm/CMakeFiles/modm.dir/src/modm/platform/core/startup_platform.c.obj
modm/CMakeFiles/modm.dir/src/modm/platform/core/vectors.c.obj modm/CMakeFiles/modm.dir/src/modm/platform/gpio/enable.cpp.obj
modm/CMakeFiles/modm.dir/src/modm/platform/itm/itm.cpp.obj modm/CMakeFiles/modm.dir/src/modm/platform/uart/uart_3.cpp.obj
modm/CMakeFiles/modm.dir/src/modm/utils/dummy.cpp.obj CMakeFiles/projectdir.dir/main.cpp.obj -o projectname.elf
 && cmd.exe /C "cd /D C:\abs\path\to\build\dir
 && arm-none-eabi-objcopy -Obinary projectname.elf projectname.bin
 && arm-none-eabi-objcopy -Oihex projectname.elf projectname.hex
 && cd /D C:\abs\path\to\build\dir
&& C:\abs\path\toproject\root\modm\modm_tools\size.py projectname.elf "[{'name': 'flash', 'access': 'rx', 'start': 134217728, 'size': 524288}, {'name': 'sram1', 'access': 'rwx', 'start': 536870912, 'size': 131072}]"""
[build] ninja: build stopped: subcommand failed.
[build] Build finished with exit code 1

Edit: Missed the size parameter.

[rleh]

After that a "open file with" dialog opens up and and after selecting a program, the cmd says failed.

I think the call of size.py isn't correct.

Probably both because "CMake wasn't able to find python3".
(I assume you have python3 installed on you windows system, otherwise I wonder how lbuild works...)

[rleh]

Maybe we should use FindPython3 instead of assuming a python3 executable is in $PATH?

[salkinium]

I think the call of size.py isn't correct.

There's an extra src/ in the path, because it's designed for the sources to be in a src/ folder unlike our examples. This should be switched based on whether the include_cmakelists option is set to true.

[jensboe]

After that a "open file with" dialog opens up and and after selecting a program, the cmd says failed.
I think the call of size.py isn't correct.

Probably both because "CMake wasn't able to find python3".
(I assume you have python3 installed on you windows system, otherwise I wonder how lbuild works...)

I assume that too. Python 3 is installed automatically after the first call with Windows store. (I don't know who and why somebody at MS think that that is a good idea)

[jensboe]

Replacing all occurrences of ${PYTHON3_EXECUTABLE} with python leads to a clean exitcode 0 build. :)

I think CMake itself is the problem now.

[jensboe]

find_package(Python3 COMPONENTS Interpreter REQUIRED)
leads to

[cmake] CMake Error at C:/Program Files/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
[cmake]   Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter)

[salkinium]

Could Windows perhaps not install python3, only python?

find_package(Python COMPONENTS Interpreter REQUIRED)

[jensboe]

I don't know who and why somebody at MS think that that is a good idea

I's a bad idea. After install Python without the windows store app function and cleaning the path:

[cmake] -- Found Python3: C:/Python/Python39/python.exe (found version "3.9.5") found components: Interpreter 

On a fresh python installation pyelftools is needed but not mention (only for old ubuntu versions)

[jensboe]

Could Windows perhaps not install python3, only python?

find_package(Python COMPONENTS Interpreter REQUIRED)

That didn't set PYTHON3_EXECUTABLE.

find_program(PYTHON3_EXECUTABLE python REQUIRED) works now, too.

[jensboe]

Could Windows perhaps not install python3, only python?

The 'windows store python' did both. The 'real python' only python.

To sum this evening:

1. Backslashes in .\modm\cmake\ModmConfiguration.cmake must be normal slashes (or must be escaped?). I think slashes are the nicer way because their compatibility to unix systems.
2. pyelftools must be mention in the getting started guide for "windows and cmake only" (BTW: Why are they installed in newer ubuntu or are they just a dependency of Scons?)
3. A nice way for python3 (non-windows) and python (windows) command must be found.

[rleh]

• Backslashes in .\modm\cmake\ModmConfiguration.cmake must be normal slashes (or must be escaped?). I think slashes are the nicer way because their compatibility to unix systems.

👍🏽

• pyelftools must be mention in the getting started guide for "windows and cmake only" (BTW: Why are they installed in newer ubuntu or are they just a dependency of Scons?)

pyelftools are a dependency of the modm pip package that is installed in the getting started guide.

[sbx320]

I've looked a bit deeper into the Windows Store Python issues.

The Windows Store Python does something very odd. Rather than adding the python install folder to the PATH, %localappdata%/Microsoft/WindowsApps is added to the PATH, which contains zero byte "executables" (such as python.exe). Launching those executables causes the corresponding store package to be used, very similar to a symlink (yet it is not a proper symlink). This works just find for launching programs (which is also why replacing PYTHON3_EXECUTABLE with pythonworks as a workaround). However CMake does a bit more:

CMake's find_program walks all $PATH entries and checks whether the requested file exists in that directory. To validate whether a file exists, it first reads the file attributes and then - if the attributes indicates a NTFS reparse point [1] - it attempts to create a handle to that file to validate that the file at the end of the reparse point exists [0]. However this fails because you cannot create regular handles to those magic zero byte redirection files. Instead you can only execute them.

So I suppose that part is an upstream bug of either CMake or Windows rather than a modm bug.

---

[0]: See https://github.com/Kitware/CMake/blob/master/Source/kwsys/SystemTools.cxx#L1336
[1]: NTFS Reparse Points are essentially entry points for filesystem filters, which are then used to implement things like symlinks or mount point. Microsoft enjoys using those to bolt on various new things to NTFS.

[ghost]

Originally it was tested on windows and I can remember it was necessary to downgrade cmake to make it work. If build-in python libraries are use in lbuild to generate path names backslashes can also be a problem on Windows.

[ghost]

I have fund the following issues:

• Python installed from python.org works. The "Windows Store" version of Python does not work with CMake.
• CMake requires that Python has the same build architecture, don't mix 32 and 64 bits versions
• Backslashes: lbuild uses os.path and maybe a major rewrite to pathlib is needed. (not examined properly) but workaround exists, see below.

Tested with following version:
CMake 3.22
Python 3.9.9
Ninja 1.10,2 (instead of nmake)
arm-none-eabi 10.3

This ain't pretty but it works. The following code can be placed in the main CMakeLists.txt and it will replace the backslashes automatically.

function (convert_backslash_path FILE_NAME)
  file(READ ${FILE_NAME} SRC)
  file(TO_CMAKE_PATH ${SRC} DST)
  file(WRITE ${FILE_NAME} ${DST})
endfunction()

convert_backslash_path("modm/CMakeLists.txt")
convert_backslash_path("modm/cmake/ModmConfiguration.cmake")

[CrustyAuklet]

I have been playing with modm on windows recently and also ran into this issue.

I use CMake a lot across different platforms and IMO one should always use / path seperators, even on windows! In fact I use Windows daily at work and I don't think I have ever really used a backslash. Powershell, CMake, etc all work fine with / and it keeps me sane to have things the same across systems.

As far as the weird windows store python, one of the first things I do when I have a fresh windows install is remove all those executables and remove that path from my PATH. Then I install the "real" python from python.org. Again, it's a sanity thing.

The changes I made to address things, taking into consideration the above two things, you can see in my modm fork here: CrustyAuklet@bf542fb

I kept changes to the python restricted to cmake generation code, and in the CMake I check for python in addition to python3 if it's windows. I could figure out some more complex checking logic if you want to support more edge cases? I have used the FindPython3 module before and it is probably a better choice.

[salkinium]

Feel free to change the CMake generator and path issues as you see fit! My only experience with Windows is inside a VM and only briefly, so these issues are really not intentional.

(Unfortunately changing lbuild to use pathlib exclusively breaks backwards compatibility since pathlib.Path doesn't behave like a string everywhere. It's been bugging me, but I can't change it anymore…)

[jensboe]

This issue can be closed. pathes are generated with / now.
I have some other issues with Cmake now but I think its better to create a new issue for that.