Move to ninja build system

[tannewt]

Prototype, WIP version of this is here: master...tannewt:ninja

[omsai]

For anyone working on this, switch to the ninja git branch, generate and run the build.ninja file using:

$ python3 tools/build/circuitpython.py feather_m0_express
$ ninja -C build-feather_m0_express

At the minute it fails with:

In file included from .././py/lexer.h:32,
                 from ../extmod/vfs.h:29,
                 from ../main.c:30:
.././py/qstr.h:42:10: fatal error: genhdr/qstrdefs.enum.h: No such file or directory
   42 | #include "genhdr/qstrdefs.enum.h"

...probably because it's not yet running the equivalent make command to generate the qstrdefs.enum.h target:

$ make -Bn -C ports/atmel-samd BOARD=feather_m0_express
...
: "GEN build-feather_m0_express/genhdr/qstrdefs.enum.h"
python3 ../../py/makeqstrdata.py build-feather_m0_express/genhdr/qstrdefs.preprocessed.h > build-feather_m0_express/genhdr/qstrdefs.enum.h
...

Note that you can't run the above make command on the ninja branch; you have to run it on the master branch.

[omsai]

Yesterday I read through the short ninja build manual and wonder if it's worth using the depfile feature that automatically discovers required header files by using gcc -MM -MF $out.d ... to write out all required headers for a given C file. Using depfile could help in several ways:

Besides eliminating the need to specify the implicit header dependencies for an given C file using | foo.h, it also makes the process more robust by ensuring no header files fall through. For example, the current generated ninja.build has:

rule compile
  command = arm-none-eabi-gcc -I../. -I../ports/atmel-samd -I../build-feather_m0_express -I../build-feather_m0_express/qstr -I../build-feather_m0_express/version -I../ports/atmel-samd/asf4/samd21 -I../ports/atmel-samd/asf4/samd21/CMSIS/Include -I../ports/atmel-samd/boards/feather_m0_express -DFFCONF_H=\"lib/oofatfs/ffconf.h\" -D__ATSAMD21G18A__ -DSAMD21 -DSPI_FLASH_FILESYSTEM -DEXTERNAL_FLASH_DEVICE_COUNT=2 -DEXTERNAL_FLASH_DEVICES="S25FL216K,GD25Q16C" -c $in -o $out

build qstr/pp/main.c: qstr_src_pp ../main.c | version/genhdr/mpversion.h

Using depfile this can be changed to:

rule compile
  command = arm-none-eabi-gcc -MM -MF $out.d -I../. -I../ports/atmel-samd -I../build-feather_m0_express -I../build-feather_m0_express/qstr -I../build-feather_m0_express/version -I../ports/atmel-samd/asf4/samd21 -I../ports/atmel-samd/asf4/samd21/CMSIS/Include -I../ports/atmel-samd/boards/feather_m0_express -DFFCONF_H=\"lib/oofatfs/ffconf.h\" -D__ATSAMD21G18A__ -DSAMD21 -DSPI_FLASH_FILESYSTEM -DEXTERNAL_FLASH_DEVICE_COUNT=2 -DEXTERNAL_FLASH_DEVICES="S25FL216K,GD25Q16C" -c $in -o $out
  depfile = $out.d

build qstr/pp/main.c: qstr_src_pp ../main.c

This is what the gcc generated depfile main.o.d looks like so far. Towards the bottom you can see mpversion.h as ../build-feather_m0_express/version/genhdr/mpversion.h because we're saving main.o under objs/ which modify the relative path:

main.o: ../main.c ../extmod/vfs.h ../py/lexer.h ../py/mpconfig.h \
 ../ports/atmel-samd/mpconfigport.h \
 ../ports/atmel-samd/asf4/samd21/include/sam.h \
 ../ports/atmel-samd/asf4/samd21/include/samd21g18a.h \
 ../ports/atmel-samd/asf4/samd21/CMSIS/Include/core_cm0plus.h \
 ../ports/atmel-samd/asf4/samd21/CMSIS/Include/cmsis_compiler.h \
 ../ports/atmel-samd/asf4/samd21/CMSIS/Include/cmsis_gcc.h \
 ../ports/atmel-samd/asf4/samd21/include/system_samd21.h \
 ../ports/atmel-samd/asf4/samd21/include/component/adc.h \
 ../ports/atmel-samd/asf4/samd21/include/component/dac.h \
 ../ports/atmel-samd/asf4/samd21/include/component/dmac.h \
 ../ports/atmel-samd/asf4/samd21/include/component/dsu.h \
 ../ports/atmel-samd/asf4/samd21/include/component/eic.h \
 ../ports/atmel-samd/asf4/samd21/include/component/evsys.h \
 ../ports/atmel-samd/asf4/samd21/include/component/gclk.h \
 ../ports/atmel-samd/asf4/samd21/include/component/hmatrixb.h \
 ../ports/atmel-samd/asf4/samd21/include/component/i2s.h \
 ../ports/atmel-samd/asf4/samd21/include/component/mtb.h \
 ../ports/atmel-samd/asf4/samd21/include/component/nvmctrl.h \
 ../ports/atmel-samd/asf4/samd21/include/component/pac.h \
 ../ports/atmel-samd/asf4/samd21/include/component/pm.h \
 ../ports/atmel-samd/asf4/samd21/include/component/port.h \
 ../ports/atmel-samd/asf4/samd21/include/component/rtc.h \
 ../ports/atmel-samd/asf4/samd21/include/component/sercom.h \
 ../ports/atmel-samd/asf4/samd21/include/component/sysctrl.h \
 ../ports/atmel-samd/asf4/samd21/include/component/tc.h \
 ../ports/atmel-samd/asf4/samd21/include/component/tcc.h \
 ../ports/atmel-samd/asf4/samd21/include/component/usb.h \
 ../ports/atmel-samd/asf4/samd21/include/component/wdt.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/ac.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/adc.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/dac.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/dmac.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/dsu.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/eic.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/evsys.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/gclk.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sbmatrix.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/i2s.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/mtb.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/nvmctrl.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/pac0.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/pac1.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/pac2.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/pm.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/port.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/rtc.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom0.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom1.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom2.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom3.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom4.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom5.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sysctrl.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tc3.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tc4.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tc5.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tcc0.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tcc1.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tcc2.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/usb.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/wdt.h \
 ../ports/atmel-samd/asf4/samd21/include/pio/samd21g18a.h \
 ../py/circuitpy_mpconfig.h \
 ../ports/atmel-samd/boards/feather_m0_express/mpconfigboard.h \
 ../supervisor/flash_root_pointers.h \
 ../supervisor/shared/external_flash/external_flash_root_pointers.h \
 ../ports/atmel-samd/peripherals/samd/dma.h ../py/qstr.h ../py/misc.h \
 ../py/mpconfig.h ../build-feather_m0_express/genhdr/qstrdefs.enum.h \
 ../py/reader.h ../py/obj.h ../py/mpprint.h ../py/runtime0.h \
 ../supervisor/shared/translate.h ../py/proto.h ../extmod/vfs_fat.h \
 ../lib/oofatfs/ff.h ../lib/oofatfs/ffconf.h ../extmod/vfs.h \
 ../build-feather_m0_express/version/genhdr/mpversion.h ../py/nlr.h \
 ../py/compile.h ../py/parse.h ../py/emitglue.h ../py/frozenmod.h \
 ../py/mphal.h ../ports/atmel-samd/mphalport.h \
 ../supervisor/shared/tick.h ../extmod/virtpin.h ../py/runtime.h \
 ../py/mpstate.h ../py/mpthread.h ../py/nlr.h ../py/objlist.h \
 ../py/objexcept.h ../py/objtuple.h ../py/pystack.h \
 ../supervisor/linker.h ../py/repl.h ../py/gc.h ../py/stackctrl.h \
 ../lib/mp-readline/readline.h ../lib/utils/pyexec.h \
 ../ports/atmel-samd/background.h ../supervisor/cpu.h \
 ../supervisor/memory.h ../supervisor/port.h \
 ../supervisor/shared/safe_mode.h ../supervisor/filesystem.h \
 ../extmod/vfs_fat.h ../supervisor/shared/autoreload.h \
 ../supervisor/shared/translate.h ../supervisor/shared/rgb_led_status.h \
 ../lib/utils/pyexec.h ../supervisor/port.h \
 ../supervisor/shared/rgb_led_colors.h ../supervisor/shared/safe_mode.h \
 ../supervisor/shared/status_leds.h ../supervisor/shared/stack.h \
 ../supervisor/memory.h ../supervisor/serial.h \
 ../ports/atmel-samd/boards/board.h

[omsai]

I've been experimenting by directly modifying the build.ninja file and can get past the genhdr/qstrdefs.enum.h error, but I don't trust the generated file is corrected because I haven't yet corrected for all the modified paths in the $include_pre, etc. Which is worrying: the ad-hoc sed commands, clearly are not fault tolerant and I think the sed functionality should be merged into the python later. What do you think?

ninja_required_version = 1.1

# These variables are procedurally generated by tools/build/circuitpython.py

includes = $
-I.. $
-I../build-feather_m0_express $
-I../build-feather_m0_express/qstr $
-I../build-feather_m0_express/version $
-I../ports/atmel-samd $
-I../ports/atmel-samd/asf4/samd21 $
-I../ports/atmel-samd/asf4/samd21/CMSIS/Include $
-I../ports/atmel-samd/boards/build-feather_m0_express $
-I../ports/atmel-samd/boards/feather_m0_express

defines = $
-DEXTERNAL_FLASH_DEVICES="S25FL216K,GD25Q16C" $
-DEXTERNAL_FLASH_DEVICE_COUNT=2 $
-DFFCONF_H=\"lib/oofatfs/ffconf.h\" $
-DSAMD21 $
-DSPI_FLASH_FILESYSTEM $
-D__ATSAMD21G18A__

# These variables were reverse engineered from make output and may need to be
# merged with the procedurally generated variables above.

includes_pre = $
-I. $
-I../.. $
-I../../lib/tinyusb/src $
-I../../supervisor/shared/usb $
-I../lib/mp-readline $
-I../lib/timeutils $
-Iasf4/samd21 $
-Iasf4/samd21/CMSIS/Include $
-Iasf4/samd21/hal/include $
-Iasf4/samd21/hal/utils/include $
-Iasf4/samd21/hpl/core $
-Iasf4/samd21/hpl/gclk $
-Iasf4/samd21/hpl/pm $
-Iasf4/samd21/hpl/port $
-Iasf4/samd21/hpl/rtc $
-Iasf4/samd21/hpl/tc $
-Iasf4/samd21/hri $
-Iasf4/samd21/include $
-Iasf4_conf/samd21 $
-Iboards/ $
-Iboards/feather_m0_express $
-Ibuild-feather_m0_express $
-Ifreetouch $
-Iperipherals

defines_pre = $
-DCFG_TUD_CDC_RX_BUFSIZE=128 $
-DCFG_TUD_CDC_TX_BUFSIZE=128 $
-DCFG_TUD_MIDI_RX_BUFSIZE=128 $
-DCFG_TUD_MIDI_TX_BUFSIZE=128 $
-DCFG_TUD_MSC_BUFSIZE=512 $
-DCFG_TUSB_MCU=OPT_MCU_SAMD21 $
-DCIRCUITPY_ANALOGIO=1 $
-DCIRCUITPY_AUDIOBUSIO=1 $
-DCIRCUITPY_AUDIOCORE=1 $
-DCIRCUITPY_AUDIOIO=1 $
-DCIRCUITPY_AUDIOIO_COMPAT=1 $
-DCIRCUITPY_AUDIOPWMIO=0 $
-DCIRCUITPY_BITBANGIO=0 $
-DCIRCUITPY_BITBANG_APA102=0 $
-DCIRCUITPY_BLEIO=0 $
-DCIRCUITPY_BOARD=1 $
-DCIRCUITPY_BUSIO=1 $
-DCIRCUITPY_CANARY_WORD=0xADAF00 $
-DCIRCUITPY_DIGITALIO=1 $
-DCIRCUITPY_DISPLAYIO=1 $
-DCIRCUITPY_FREQUENCYIO=0 $
-DCIRCUITPY_FULL_BUILD=1 $
-DCIRCUITPY_GAMEPAD=1 $
-DCIRCUITPY_GAMEPADSHIFT=0 $
-DCIRCUITPY_I2CSLAVE=0 $
-DCIRCUITPY_MATH=1 $
-DCIRCUITPY_MICROCONTROLLER=1 $
-DCIRCUITPY_NEOPIXEL_WRITE=1 $
-DCIRCUITPY_NETWORK=0 $
-DCIRCUITPY_NVM=1 $
-DCIRCUITPY_OS=1 $
-DCIRCUITPY_PEW=0 $
-DCIRCUITPY_PIXELBUF=1 $
-DCIRCUITPY_PS2IO=0 $
-DCIRCUITPY_PULSEIO=1 $
-DCIRCUITPY_RANDOM=1 $
-DCIRCUITPY_ROTARYIO=1 $
-DCIRCUITPY_RTC=1 $
-DCIRCUITPY_SAFE_RESTART_WORD=0xDEADBEEF $
-DCIRCUITPY_SAMD=0 $
-DCIRCUITPY_SOFTWARE_SAFE_MODE=0x0ADABEEF $
-DCIRCUITPY_STAGE=0 $
-DCIRCUITPY_STORAGE=1 $
-DCIRCUITPY_STRUCT=1 $
-DCIRCUITPY_SUPERVISOR=1 $
-DCIRCUITPY_TIME=1 $
-DCIRCUITPY_TOUCHIO=1 $
-DCIRCUITPY_TOUCHIO_USE_NATIVE=1 $
-DCIRCUITPY_UHEAP=0 $
-DCIRCUITPY_USB_HID=1 $
-DCIRCUITPY_USB_MIDI=1 $
-DCIRCUITPY_USTACK=0 $
-DEXPRESS_BOARD $
-DEXTERNAL_FLASH_DEVICES="S25FL216K, GD25Q16C" $
-DEXTERNAL_FLASH_DEVICE_COUNT=2 $
-DFFCONF_H=$$"lib/oofatfs/ffconf.h$$" $
-DLONGINT_IMPL_MPZ -Os $
-DNDEBUG $
-DSAMD21 $
-DSPI_FLASH_FILESYSTEM=1 $
-DUSB_AVAILABLE $
-D__SAMD21G18A__

warnings = $
-Wall $
-Wcast-align $
-Wdouble-promotion $
-Werror $
-Werror-implicit-function-declaration $
-Wfloat-equal $
-Wmissing-format-attribute $
-Wnested-externs $
-Wno-deprecated-declarations $
-Wno-endif-labels $
-Wno-error=lto-type-mismatch $
-Wshadow $
-Wsign-compare $
-Wstrict-prototypes $
-Wundef $
-Wunreachable-code $
-Wwrite-strings

cflags_pre = $
--param max-inline-insns-single=500 $
-fdata-sections $
-ffunction-sections $
-finline-limit=60 $
-flto -flto-partition=none $
-fno-strict-aliasing $
-fshort-enums $
-fsingle-precision-constant $
-mabi=aapcs-linux $
-mcpu=cortex-m0plus $
-mfloat-abi=soft $
-msoft-float $
-mthumb $
-nostdlib $
-std=gnu11

rule compile
  depfile = $out.d
  command = arm-none-eabi-gcc -MM -MF $out.d $includes $defines -c $in -o $out
  description = CC $out

rule gen-qstr-defs-pre
  command = cat ../py/qstrdefs.h $in | $
sed 's/^Q(.*)/"&"/' | $
arm-none-eabi-gcc -E $includes_pre $defines_pre $warnings $cflags_pre - | $
sed 's/^"\(Q(.*)\)"/\1/' > $out
  description = GEN $out

rule gen-qstr-data
  command = python3 ../py/makeqstrdata.py $in > $out
  description = GEN $out

rule gen-version-hdr
  command = python3 ../py/makeversionhdr.py $out
  description = GEN $out

build objs/main.o : compile ../main.c
build genhdr/qstrdefs.preprocessed.h : gen-qstr-defs-pre $
../ports/atmel-samd/qstrdefsport.h $
../ports/atmel-samd/build-feather_m0_express/genhdr/qstrdefs.collected.h
build genhdr/qstrdefs.enum.h : gen-qstr-data genhdr/qstrdefs.preprocessed.h
build version/genhdr/mpversion.h: gen-version-hdr

[tannewt]

Yesterday I read through the short ninja build manual and wonder if it's worth using the depfile feature that automatically discovers required header files by using gcc -MM -MF $out.d ... to write out all required headers for a given C file. Using depfile could help in several ways:

Besides eliminating the need to specify the implicit header dependencies for an given C file using | foo.h, it also makes the process more robust by ensuring no header files fall through. For example, the current generated ninja.build has:

I definitely want to use depfile for header dependencies. I don't think it'll work for mpversionhdr though because the file doesn't exist until built. I believe you need to be explicit for headers that are generated.

I've been experimenting by directly modifying the build.ninja file and can get past the genhdr/qstrdefs.enum.h error, but I don't trust the generated file is corrected because I haven't yet corrected for all the modified paths in the $include_pre, etc. Which is worrying: the ad-hoc sed commands, clearly are not fault tolerant and I think the sed functionality should be merged into the python later. What do you think?

The sed stuff should definitely be merged into one of the QSTR build scripts.

[tannewt]

@omsai I pushed a new version that completes the QSTR stuff. Unfortunately, the next step is evolving the dynamic linkerscript generation that SAMD does.

[durapensa]

Zephyr's multiple repo & build management tool, west, may be useful. It manages both ninja & cmake builds, and has a nice clean way to manage multiple git repos.
https://github.com/zephyrproject-rtos/west
Please disregard this comment if if it's not useful.

[tannewt]

Thanks for the tip @durapensa! Can it do ninja without cmake? I don't really know want cmake to be a piece we use.

[durapensa]

@tannewt Peeking at the west docs, it uses cmake to generate ninja build files & board files. But since it's Python, that part can probably be replumbed. There's also a flashing component; could that be handy for priming/repairing boards for usb disk?

[omsai]

@tannewt Can you again suggest some representative hardware target devices to focus this build system retooling effort? I don't mind buying several devices with bare minimum features if helps with quicker wins. A power outage damaged my Metro M4 Airlift which was plugged into a wall outlet at the time. Maybe a device that you're also using and that's still in production?