Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problems using lbuild with windows #67

Closed
amarokmclion opened this issue Aug 15, 2018 · 15 comments
Closed

Problems using lbuild with windows #67

amarokmclion opened this issue Aug 15, 2018 · 15 comments
Labels
hosted 🖥

Comments

@amarokmclion
Copy link
Contributor

amarokmclion commented Aug 15, 2018 •
edited by salkinium

Hi all,

I'm trying to use modm under windows 10 with anaconda3 64bit. I've followed the steps in the installation page and found a few mistakes.

  • "conda install" command from anaconda can't install github recipes like "pip install" can

  • pyelftools is not available from conda-forge thus I had to install it using pip

In order to solve this issues I used the following commands to procede with the installation:

conda create --name modm python=3 pip #this is needed because otherwise anaconda will use the global pip and not the one from the env
activate modm
conda install -c conda-forge git 
pip install jinja2 scons future pyelftools git+https://github.com/modm-io/library-builder

After doing this I had to configure a .bat file to run the lbuild file inside Scripts, something simple like this:

@echo off
set scriptname=%~dp0%~n0
python "%scriptname%" %*

After doing this I cloned recursively the repository and I could call lbuild inside an example directory, but I got this error:

c:\modm\examples\stm32f4_discovery\blink (develop)
(modm3) λ lbuild -v build
[INFO] lbuild.module: Found module 'cmsis'
[INFO] lbuild.module: Found module 'core'
[INFO] lbuild.module: Found module 'dsp'
[INFO] lbuild.module: Found module 'fatfs'
[INFO] lbuild.module: Found module 'freertos'
[INFO] lbuild.module: Found module 'device'
[INFO] lbuild.module: Found module 'ros'
[INFO] lbuild.module: Found module 'device'
[INFO] lbuild.module: Found module 'tlsf'
[INFO] lbuild.module: Found module 'architecture'
[INFO] lbuild.module: Found module 'accessor'
[INFO] lbuild.module: Found module 'adc'
[INFO] lbuild.module: Found module 'assert'
[INFO] lbuild.module: Found module 'atomic'
[INFO] lbuild.module: Found module 'block.device'
[INFO] lbuild.module: Found module 'can'
[INFO] lbuild.module: Found module 'clock'
[INFO] lbuild.module: Found module 'delay'
[INFO] lbuild.module: Found module 'gpio'
[INFO] lbuild.module: Found module 'gpio.expander'
[INFO] lbuild.module: Found module 'heap'
[INFO] lbuild.module: Found module 'i2c'
[INFO] lbuild.module: Found module 'i2c.device'
[INFO] lbuild.module: Found module 'interrupt'
[INFO] lbuild.module: Found module 'memory'
[INFO] lbuild.module: Found module '1-wire'
[INFO] lbuild.module: Found module 'register'
[INFO] lbuild.module: Found module 'spi'
[INFO] lbuild.module: Found module 'spi.device'
[INFO] lbuild.module: Found module 'uart'
[INFO] lbuild.module: Found module 'unaligned'
[INFO] lbuild.module: Found module 'board'
[INFO] lbuild.module: Found module 'al-avreb-can'
[INFO] lbuild.module: Found module 'arduino-uno'
[INFO] lbuild.module: Found module 'black-pill'
[INFO] lbuild.module: Found module 'blue-pill'
[INFO] lbuild.module: Found module 'disco-f051r8'
[INFO] lbuild.module: Found module 'disco-f072rb'
[INFO] lbuild.module: Found module 'disco-f100rb'
[INFO] lbuild.module: Found module 'disco-f303vc'
[INFO] lbuild.module: Found module 'disco-f407vg'
[INFO] lbuild.module: Found module 'disco-f429zi'
[INFO] lbuild.module: Found module 'disco-f469ni'
[INFO] lbuild.module: Found module 'disco-f746ng'
[INFO] lbuild.module: Found module 'disco-f769ni'
[INFO] lbuild.module: Found module 'disco-l476vg'
[INFO] lbuild.module: Found module 'nucleo-f031k6'
[INFO] lbuild.module: Found module 'nucleo-f042k6'
[INFO] lbuild.module: Found module 'nucleo-f103rb'
[INFO] lbuild.module: Found module 'nucleo-f303k8'
[INFO] lbuild.module: Found module 'nucleo-f401re'
[INFO] lbuild.module: Found module 'nucleo-f411re'
[INFO] lbuild.module: Found module 'nucleo-f429zi'
[INFO] lbuild.module: Found module 'nucleo-l432kc'
[INFO] lbuild.module: Found module 'nucleo-l476rg'
[INFO] lbuild.module: Found module 'olimexino-stm32'
[INFO] lbuild.module: Found module 'stm32f030_demo'
[INFO] lbuild.module: Found module 'communication'
[INFO] lbuild.module: Found module 'ros'
[INFO] lbuild.module: Found module 'sab'
[INFO] lbuild.module: Found module 'sab2'
[INFO] lbuild.module: Found module 'xpcc'
[INFO] lbuild.module: Found module 'container'
[INFO] lbuild.module: Found module 'debug'
[INFO] lbuild.module: Found module 'driver'
[INFO] lbuild.module: Found module 'ad7280a'
[INFO] lbuild.module: Found module 'ad7928'
[INFO] lbuild.module: Found module 'adc_sampler'
[INFO] lbuild.module: Found module 'memory_bus'
[INFO] lbuild.module: Found module 'lawicel'
[INFO] lbuild.module: Found module 'mcp2515'
[INFO] lbuild.module: Found module 'tcs3414'
[INFO] lbuild.module: Found module 'tcs3472'
[INFO] lbuild.module: Found module 'ea_dog'
[INFO] lbuild.module: Found module 'hd44780'
[INFO] lbuild.module: Found module 'max7219'
[INFO] lbuild.module: Found module 'nokia5110'
[INFO] lbuild.module: Found module 'parallel_tft_display'
[INFO] lbuild.module: Found module 'sdl_display'
[INFO] lbuild.module: Found module 'siemens_s65'
[INFO] lbuild.module: Found module 'siemens_s75'
[INFO] lbuild.module: Found module 'ssd1306'
[INFO] lbuild.module: Found module 'mcp23x17'
[INFO] lbuild.module: Found module 'pca8574'
[INFO] lbuild.module: Found module 'pca9535'
[INFO] lbuild.module: Found module 'hmc58x'
[INFO] lbuild.module: Found module 'hmc6343'
[INFO] lbuild.module: Found module 'itg3200'
[INFO] lbuild.module: Found module 'l3gd20'
[INFO] lbuild.module: Found module 'lis302dl'
[INFO] lbuild.module: Found module 'lis3dsh'
[INFO] lbuild.module: Found module 'lis3.transport'
[INFO] lbuild.module: Found module 'lsm303a'
[INFO] lbuild.module: Found module 'adns9800'
[INFO] lbuild.module: Found module 'drv832x_spi'
[INFO] lbuild.module: Found module 'vl53l0'
[INFO] lbuild.module: Found module 'vl6180'
[INFO] lbuild.module: Found module 'amsys5915'
[INFO] lbuild.module: Found module 'bme280'
[INFO] lbuild.module: Found module 'bmp085'
[INFO] lbuild.module: Found module 'hclax'
[INFO] lbuild.module: Found module 'max6966'
[INFO] lbuild.module: Found module 'pca9685'
[INFO] lbuild.module: Found module 'nrf24'
[INFO] lbuild.module: Found module 'ds1302'
[INFO] lbuild.module: Found module 'block.device'
[INFO] lbuild.module: Found module 'file'
[INFO] lbuild.module: Found module 'heap'
[INFO] lbuild.module: Found module 'mirror'
[INFO] lbuild.module: Found module 'spi.flash'
[INFO] lbuild.module: Found module 'fat'
[INFO] lbuild.module: Found module 'i2c.eeprom'
[INFO] lbuild.module: Found module 'ds1631'
[INFO] lbuild.module: Found module 'ds18b20'
[INFO] lbuild.module: Found module 'lm75'
[INFO] lbuild.module: Found module 'ltc2984'
[INFO] lbuild.module: Found module 'tmp102'
[INFO] lbuild.module: Found module 'tmp175'
[INFO] lbuild.module: Found module 'ads7843'
[INFO] lbuild.module: Found module 'ft6x06'
[INFO] lbuild.module: Found module 'ft245'
[INFO] lbuild.module: Found module 'io'
[INFO] lbuild.module: Found module 'math'
[INFO] lbuild.module: Found module 'matrix'
[INFO] lbuild.module: Found module 'filter'
[INFO] lbuild.module: Found module 'geometry'
[INFO] lbuild.module: Found module 'interpolation'
[INFO] lbuild.module: Found module 'saturated'
[INFO] lbuild.module: Found module 'utils'
[INFO] lbuild.module: Found module 'platform'
[INFO] lbuild.module: Found module 'adc'
[INFO] lbuild.module: Found module 'adc'
[INFO] lbuild.module: Found module '1'
[INFO] lbuild.module: Found module '2'
[INFO] lbuild.module: Found module '3'
[INFO] lbuild.module: Found module 'adc'
[INFO] lbuild.module: Found module 'canusb'
[INFO] lbuild.module: Found module 'can.common'
[INFO] lbuild.module: Found module 'can'
[INFO] lbuild.module: Found module 'socketcan'
[INFO] lbuild.module: Found module 'can'
[INFO] lbuild.module: Found module '1'
[INFO] lbuild.module: Found module '2'
[INFO] lbuild.module: Found module 'clock'
[INFO] lbuild.module: Found module 'clock.cortex'
[INFO] lbuild.module: Found module 'clock'
[INFO] lbuild.module: Found module 'clock'
[INFO] lbuild.module: Found module 'comp'
[INFO] lbuild.module: Found module 'core'
[INFO] lbuild.module: Found module 'core'
[INFO] lbuild.module: Found module 'core'
[INFO] lbuild.module: Found module 'dma'
[INFO] lbuild.module: Found module '1'
[INFO] lbuild.module: Found module '2'
[INFO] lbuild.module: Found module 'fault.cortex'
[INFO] lbuild.module: Found module 'fsmc'
[INFO] lbuild.module: Found module 'gpio'
[INFO] lbuild.module: Found module 'gpio.common'
[INFO] lbuild.module: Found module 'gpio'
[INFO] lbuild.module: Found module 'gpio'
[INFO] lbuild.module: Found module 'gpio'
[INFO] lbuild.module: Found module 'i2c'
[INFO] lbuild.module: Found module 'i2c.bitbang'
[INFO] lbuild.module: Found module 'i2c'
[INFO] lbuild.module: Found module '1'
[INFO] lbuild.module: Found module '2'
[INFO] lbuild.module: Found module '3'
[INFO] lbuild.module: Found module 'i2c'
[INFO] lbuild.module: Found module 'i2c'
[INFO] lbuild.module: Found module 'id'
[INFO] lbuild.module: Found module '1-wire.bitbang'
[INFO] lbuild.module: Found module 'random'
[INFO] lbuild.module: Found module 'spi'
[INFO] lbuild.module: Found module 'spi_uart'
[INFO] lbuild.module: Found module 'spi.bitbang'
[INFO] lbuild.module: Found module 'spi'
[INFO] lbuild.module: Found module 'spi'
[INFO] lbuild.module: Found module '1'
[INFO] lbuild.module: Found module '2'
[INFO] lbuild.module: Found module '3'
[INFO] lbuild.module: Found module 'uart.spi'
[INFO] lbuild.module: Found module '1'
[INFO] lbuild.module: Found module '2'
[INFO] lbuild.module: Found module '3'
[INFO] lbuild.module: Found module '6'
[INFO] lbuild.module: Found module 'spi'
[INFO] lbuild.module: Found module 'timer'
[INFO] lbuild.module: Found module 'timer'
[INFO] lbuild.module: Found module '1'
[INFO] lbuild.module: Found module '8'
[INFO] lbuild.module: Found module '6'
[INFO] lbuild.module: Found module '7'
[INFO] lbuild.module: Found module '2'
[INFO] lbuild.module: Found module '3'
[INFO] lbuild.module: Found module '4'
[INFO] lbuild.module: Found module '5'
[INFO] lbuild.module: Found module '9'
[INFO] lbuild.module: Found module '10'
[INFO] lbuild.module: Found module '11'
[INFO] lbuild.module: Found module '12'
[INFO] lbuild.module: Found module '13'
[INFO] lbuild.module: Found module '14'
[INFO] lbuild.module: Found module 'timer'
[INFO] lbuild.module: Found module 'uart'
[INFO] lbuild.module: Found module 'uart'
[INFO] lbuild.module: Found module 'uart'
[INFO] lbuild.module: Found module 'uart'
[INFO] lbuild.module: Found module '4'
[INFO] lbuild.module: Found module '5'
[INFO] lbuild.module: Found module '1'
[INFO] lbuild.module: Found module '2'
[INFO] lbuild.module: Found module '3'
[INFO] lbuild.module: Found module '6'
[INFO] lbuild.module: Found module 'uart'
[INFO] lbuild.module: Found module 'processing'
[INFO] lbuild.module: Found module 'protothread'
[INFO] lbuild.module: Found module 'resumable'
[INFO] lbuild.module: Found module 'rtos'
[INFO] lbuild.module: Found module 'scheduler'
[INFO] lbuild.module: Found module 'timer'
[INFO] lbuild.module: Found module 'button'
[INFO] lbuild.module: Found module 'color'
[INFO] lbuild.module: Found module 'ui'
[INFO] lbuild.module: Found module 'animation'
[INFO] lbuild.module: Found module 'display'
[INFO] lbuild.module: Found module 'gui'
[INFO] lbuild.module: Found module 'led'
[INFO] lbuild.module: Found module 'menu'
[INFO] lbuild.module: Found module 'time'
[INFO] lbuild.module: Found module 'utils'
[INFO] lbuild.module: Found module 'stdc++'
[INFO] lbuild.module: Found module 'unittest'
[INFO] lbuild.module: Found module 'test'
[INFO] lbuild.module: Found module 'architecture'
[INFO] lbuild.module: Found module 'communication'
[INFO] lbuild.module: Found module 'container'
[INFO] lbuild.module: Found module 'driver'
[INFO] lbuild.module: Found module 'io'
[INFO] lbuild.module: Found module 'math'
[INFO] lbuild.module: Found module 'platform'
[INFO] lbuild.module: Found module 'gpio'
[INFO] lbuild.module: Found module 'spi'
[INFO] lbuild.module: Found module 'processing'
[INFO] lbuild.module: Found module 'ui'
[INFO] lbuild.module: Found module 'stdc++'
[INFO] lbuild.module: Found module 'build'
[INFO] lbuild.module: Found module 'cmake'
[INFO] lbuild.module: Found module 'scons'
[INFO] lbuild.module: Found module 'docs'
[INFO] lbuild.module: Found module 'generator'

[INFO] lbuild.parser: Selected modules: modm:board:disco-f407vg, modm:build:scons
[INFO] lbuild.module: Prepare for build modm:platform:gpio
[INFO] lbuild.module: Prepare for build modm:cmsis:device
[INFO] lbuild.module: Build modm:platform:spi:1

ERROR: Failed to retrieve Template: src\modm\platform\spi\stm32\spi_hal.hpp.in

The file spi_hal.hpp.in is in my directory structure and I can't understand what's the problem,
I tryed running again the lbuild build command and it got stuck at this point this time:

ERROR: Failed to retrieve Template: src\modm\platform\i2c\stm32\i2c_master.cpp.in

This is my python environment and python version:

c:\modm\examples\stm32f4_discovery\blink (develop)
(modm3) λ pip list
Package Version

certifi 2018.8.13
future 0.16.0
gitdb2 2.0.4
GitPython 2.1.11
Jinja2 2.10
lbuild 0.1
lxml 4.2.4
MarkupSafe 1.0
pip 18.0
pyelftools 0.24
scons 3.0.1
setuptools 40.0.0
smmap2 2.0.4
wheel 0.31.1
wincertstore 0.2

c:\modm\examples\stm32f4_discovery\blink (develop)
(modm3) λ python
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 11:48:23) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

I hope this will help getting a better windows support.
@salkinium
Copy link
Member

Hm, stuck on the very first access to a template… sorry about this, I haven’t used Windows in over a decade, so not an expert.

Cc @dergraaf maybe a path encoding error? I know we had path issues in SCons on Windows before.

@amarokmclion
Copy link
Contributor Author

Hi again,

I did some debugging and found that Jinja2 doesn't like the \ path separator. I'll try to craft a fix and push it in a PR.

@salkinium
Copy link
Member

In xpcc, there was a .replace on the template path:
https://github.com/roboterclubaachen/xpcc/blob/develop/scons/site_tools/template.py#L134
But it’s missing in lbuild:
https://github.com/modm-io/library-builder/blob/develop/lbuild/environment.py#L241

Maybe that’s the issue?

@amarokmclion
Copy link
Contributor Author

Yes, I did something like that in order to make it work.
I've encountered also a problem with the generated SConscript, each env.File entry had a single backslash as a separator in the string and thus SCons couldn't read correctly the path string.
I manually changed each backslash in a forward slash and it worked.
This made me remember that windows 10 supports both forward and back slash as path separators even in the command line, maybe one general solution could be to forget the backslash altogether.
The other solution is to use the paths generated with pathlib instead of os.path.

@amarokmclion
Copy link
Contributor Author

amarokmclion commented Aug 18, 2018 •
edited

I've made some pull requests that will make life for windows users a little better. Allowing a basic usage of:

  • lbuild build
  • scons build and size
  • maybe cmake build (haven't tested it yet)
  • openocd programming with openocd -f modm/openocd.cfg -c "program_release" and openocd -f modm/openocd.cfg -c "program_debug",
  • scons program will not work.

I've tested the modifications trying to build and program the nucleo_f042 and the nucleo_f411 blink examples.

TODO:
making scons program work
making lbuild git tests pass
more testing

@salkinium
Copy link
Member

I've updated the install instructions with your suggestions: https://modm.io/guide/installation/#windows

However, I'm unclear about the .bat step. Is this still required, or did your lbuild PR fix this?

@amarokmclion
Copy link
Contributor Author

Yes, with that PR there is no need to the .bat.

@salkinium
Copy link
Member

cc @daniel-k also has to support Windows

@jnewcomb
Copy link

jnewcomb commented Dec 1, 2018 •
edited by salkinium

Hello folks,
I've failed to reach the first hurdle! Any tips / suggestions / workarounds are much appreciated.

There seems to be an issue in project.xml with all examples that use the : (semicolon)

For example, in modm:board:nucleo-f303k8 when running lbuild build.

(modm) C:\Users\im\modm\examples\nucleo_f303k8\blink>lbuild build
Traceback (most recent call last):
  File "c:\users\im\appdata\local\conda\conda\envs\modm\lib\runpy.py", line 193, in _run_module_as_main    "__main__", mod_spec)
  File "c:\users\im\appdata\local\conda\conda\envs\modm\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\im\AppData\Local\conda\conda\envs\modm\Scripts\lbuild.exe\__main__.py", line 9, in <module>
  File "c:\users\im\appdata\local\conda\conda\envs\modm\lib\site-packages\lbuild\main.py", line 410, in main output = run(args)
  File "c:\users\im\appdata\local\conda\conda\envs\modm\lib\site-packages\lbuild\main.py", line 395, in run builder = Builder(config=args.config, options=args.options)
  File "c:\users\im\appdata\local\conda\conda\envs\modm\lib\site-packages\lbuild\api.py", line 44, in __init__ config = ConfigNode.from_file(config)
  File "c:\users\im\appdata\local\conda\conda\envs\modm\lib\site-packages\lbuild\config.py", line 160, in from_file if cpath.exists():
  File "c:\users\im\appdata\local\conda\conda\envs\modm\lib\pathlib.py", line 1329, in exists self.stat()
  File "c:\users\im\appdata\local\conda\conda\envs\modm\lib\pathlib.py", line 1151, in stat  return self._accessor.stat(self)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 
'C:\\Users\\im\\modm\\examples\\nucleo_f303k8\\blink\\modm:board:nucleo-f303k8'

My setup
Windows10

(modm) C:\Users\im\modm\examples\nucleo_f303k8\blink>pip list
Package         Version
--------------- ----------
anytree         2.4.3
certifi         2018.11.29
colorama        0.4.1
colorful        0.4.4
CppHeaderParser 2.7.4
deepdiff        3.3.0
future          0.17.1
gitdb2          2.0.5
GitPython       2.1.11
Jinja2          2.10
jsonpickle      1.0
lbuild          1.5.0
lxml            4.2.5
MarkupSafe      1.1.0
pip             18.1
ply             3.11
pyelftools      0.25
scons           3.0.1
setuptools      40.6.2
six             1.11.0
smmap2          2.0.5
wheel           0.32.3
wincertstore    0.2

(modm) C:\Users\im\modm\examples\nucleo_f303k8\blink>python
Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 19:01:41) [MSC v.1900 64 bit (AMD64)] :: Anaconda, Inc. on win32

Thanks..

@salkinium
Copy link
Member

salkinium commented Dec 1, 2018 •
edited

The <extends> element works by first checking if a file exists with this name locally, before checking in the repositories. Here is seems like pathlib doesn’t properly escape : in file names for Windows, when lbuild calls cpath.exists(). That’s pretty bad, especially since you have an up-to-date Python version. We could fix it in lbuild, but I don’t really want to add platform specific workarounds on a Python project… Le Sigh, Windows is such a shit show.

@salkinium
Copy link
Member

Apparently, the symbol : (colon) is simply not allowed in paths on Windows.
I don’t understand though why this worked for some of our other Windows users and not for you, maybe this is new with Windows 10? We’ve had a lot of Windows paths issues, some with conflicting solutions, so I’m very uncertain about what the limitations of Windows paths actually are.

@jnewcomb
Copy link

jnewcomb commented Dec 2, 2018 •
edited

Time to build a box then..
I guess because windows maps drives using the : (colon) whereas Linux uses mounts.
So as well a / (slash) not being a valid char they have added the : (colon) too.
Never thought about that.. imagine the confusion if you called a directory c:mydir
Saying that, you call a directory my%20files and it doesn't seem to complain!
Thanks for a quick response,
Jon.

@operativeF
Copy link

For those on Windows looking for a quick fix to this, you can edit line 159 of config.py in lbuild with this:
cpath = Path(ConfigNode._rel_path(node.text.replace(":", "_"), configpath))
I also had a problem with multiprocessing running the examples_compile.py script (another windows related issue, probably deserves another Issue #) which was solved by adding entry point protection. See here: https://github.com/operativeF/modm/tree/fix/win_multiprocessing

@salkinium salkinium mentioned this issue Feb 13, 2019
Merged
@salkinium
Copy link
Member

Can you check if this change in lbuild fixes your issue too? I just wrapped this into an exception.

@operativeF
Copy link

Yup, that fixes it. Thanks

@salkinium salkinium mentioned this issue Apr 9, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
hosted 🖥
Milestone
No milestone
Development

No branches or pull requests
4 participants
@salkinium @jnewcomb @operativeF @amarokmclion