PX4 protected build pub #16931

jlaitine · 2021-02-23T07:06:02Z

This PR implements PX4 running on NuttX protected build. The split between kernel and user space libraries is in top level:

kernel:

All drivers
Full uORB
Parameters (but only loading at boot time)
high resolution timer
Anything that links directly to OS drivers or accesses hardware

user space:

Another uORB, mostly only uORBManager communicating with kernel side over ioctl
parameters
hrt interafce, using an own highest priority task for callbacks

build scripts:

Support for building nuttx kernel and user -side libraries, linking for this
Support for generating "builtins" for kernel side modules
Support for putting any module to eiither kernel or user side (adding "KERNEL" flag to CMakeLists.txt for the module). Restrictions apply; a module linking to nuttx kernel or accessing HW cannot be linked into user side..

LorenzMeier · 2021-02-23T08:37:09Z

Thanks! I will set some quality time apart to go through it. This capability is one of the key reasons why we chose NuttX 10 years ago and why some of the PX4 fundamentals have been architected around file descriptors (to be able to go through kernel space cleanly). We have since optimized on various fronts, so I'm sure it needs some refresh and polishing.

dagar · 2021-02-23T14:30:38Z

I'm very interested to dig into this and see exactly where (and how) you've split things between kernel and user space. I actually gave this a try with NuttX a few years ago, but quickly came to the conclusion the value was minimal if nearly all of PX4 was simply in user space (able to clobber itself).

jlaitine · 2021-04-23T16:18:09Z

If someone has had time to look into this bunch of stuff; I'd appreciate comments on it; basically which parts are implemented in a proper manner, and which could be done better or doesn't make sense at all. I could start chopping it in pieces and make individual smaller PR:s, for example

dagar · 2021-04-23T20:42:56Z

If someone has had time to look into this bunch of stuff; I'd appreciate comments on it; basically which parts are implemented in a proper manner, and which could be done better or doesn't make sense at all. I could start chopping it in pieces and make individual smaller PR:s, for example

I'm still very interested in this and would like to start getting pieces in incrementally as soon as we get v1.12 out.

jlaitine · 2021-05-25T06:30:09Z

Rebased against current master, + added the following:
on px4_fmuv5_protected target:

removed the micrortps agent from the build, it is not relevant for this work
changed flash and sram allocation between kernel and user space, doubled the kernel memories so that it is easier to drop more stuff to kernel
moved "sensors" to kernel side, it saves a lot of CPU when all the sensor data doesn't need to go many times between kenel-userspace barrier
minor cleanups

Didn't have time to clean up all the things I had in mind yet though

jlaitine · 2021-09-06T13:14:57Z

Just a re-base to keep this alive. The following items should be resolved (at least):

couple of "command not found" nsh errors on other targets than "protected-build"; "userspace_init" systemcmd called from rcS, which only exists for protected, and "kparam" for loading kernel side parameters at boot
all uORBs are allocated from user-side heap for protected-build, because of uORB performance optimization by inlining functions; these should be re-designed, especially the "advertised()" call. But I didn't want to change the functionality of uORB for now, it is complicated enough
one nuttx backport needed (protected build has broken in nuttx meanwhile)
need a nicer way to select what goes to kernel side and what to user side. Now the selection is done by adding keyword "KERNEL" into component's CMakeLists.txt. But it would be better to have a separate kernel side component list with the "board" configuration
fix the linking for other targets (again..)

jlaitine · 2022-03-08T21:35:54Z

Dropped everything that is not absolutely needed to compile and run this; also tested flying it.

jlaitine · 2022-03-09T06:26:32Z

Just re-based to see if the weird CI fail goes away

jlaitine · 2022-03-15T14:02:51Z

@dagar , @bkueng , this PR now has left the following

initializing the userspace in boot
some random boardctl things like reading vbus status into userspace
boardctl call to launch modules in kernel side
px4_fmu-v5_protected target, which is tested to build, flash and fly

Would you like me to further split this into smaller pieces, or is this reviewable as is? This brings in a working target.

bkueng · 2022-03-16T07:56:28Z

platforms/nuttx/src/px4/common/px4_userspace_init.cpp

@@ -0,0 +1,20 @@
+#include <drivers/drv_hrt.h>


Missing copyright header

Thanks, added

The px4_userspace_init function is called from userspace entrypoint before starting NSH Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

For protected/kernel builds the cxx static initializations needs to be done also in kernel side, since px4 creates c++ objects in kernel Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

…utdown lockout Add kernel side ioctls and handlers for vbus state and managing shutdown lockout Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

Add launch_kmod command to start/execute px4 modules in kenel space in NuttX protected build Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

This is needed for DMA capable memory for fat also in the user side; the file system doesn't seem to work reliably without. Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

Add a target for nuttx protected build development Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

In memory protected builds these perform nuttx boardctl ioctl calls to kerenel Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

Direct gpio read is not possible from user side applications, so use boardctl interface instead. Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

It accesses kernel internal structures directly; this needs to be worked out with some proper userspace-kernel interface Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

…build Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

… protected build target This info is needed on both kernel and user sides, and is just data. Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

It is a performance issue to run it on user side Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

bkueng

You've made it through... with a lot of patience :D
Nice work.

jlaitine · 2022-03-17T09:58:03Z

Thank you for all the reviews and help with this! This is now a good base for gradual improvements in small steps, for which we already have some ideas.

jlaitine marked this pull request as draft February 23, 2021 07:26

dagar requested review from dagar and davids5 February 26, 2021 17:44

dagar added Admin: Enhancement (improvement) 💡 NuttX (OS) labels Feb 26, 2021

mrpollo changed the title ~~Px4 protected build pub~~ PX4 protected build pub Mar 10, 2021

jlaitine force-pushed the px4_protected_build_pub branch from 62f16ba to 30f70b3 Compare May 25, 2021 06:26

jlaitine force-pushed the px4_protected_build_pub branch 16 times, most recently from 1a81cae to 4f64a3a Compare May 27, 2021 20:10

jlaitine force-pushed the px4_protected_build_pub branch from 4f64a3a to b1cf57b Compare September 6, 2021 13:08

jlaitine force-pushed the px4_protected_build_pub branch 3 times, most recently from e5f7438 to eec3194 Compare March 1, 2022 20:52

jlaitine mentioned this pull request Mar 7, 2022

Protected build parameters #19256

Merged

jlaitine force-pushed the px4_protected_build_pub branch from eec3194 to 5dd27ed Compare March 8, 2022 21:20

jlaitine marked this pull request as ready for review March 8, 2022 21:22

jlaitine force-pushed the px4_protected_build_pub branch from 5dd27ed to fc1933b Compare March 9, 2022 06:26

jlaitine force-pushed the px4_protected_build_pub branch 2 times, most recently from 48d397b to 5fefddc Compare March 15, 2022 13:59

bkueng reviewed Mar 16, 2022

View reviewed changes

jlaitine added 12 commits March 16, 2022 13:40

Add a px4 userspace initialization in protected build

13e0cf9

The px4_userspace_init function is called from userspace entrypoint before starting NSH Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

Initialize cxx static variables also in kernel side

a32e59f

For protected/kernel builds the cxx static initializations needs to be done also in kernel side, since px4 creates c++ objects in kernel Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

platforms/nuttx/src/px4/common/board_ctrl: Add handlers for vbus & sh…

d35a729

…utdown lockout Add kernel side ioctls and handlers for vbus state and managing shutdown lockout Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

Add a simple way to launch kernel side modules

269a6aa

Add launch_kmod command to start/execute px4 modules in kenel space in NuttX protected build Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

Add gran allocator also to user side in protected build

b8c4698

This is needed for DMA capable memory for fat also in the user side; the file system doesn't seem to work reliably without. Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

Add px4_fmu-v5_protected target

6f5e42a

Add a target for nuttx protected build development Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

px4_fmuv5: Implement BOARD_*_LOCKOUT_STATE in protected/kernel builds

8076c80

In memory protected builds these perform nuttx boardctl ioctl calls to kerenel Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

Use IOCTL for board_read_VBUS_state in NuttX protected builds

1ce1f3a

Direct gpio read is not possible from user side applications, so use boardctl interface instead. Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

logger/watchdog: Disable watchdog for NuttX protected/kernel builds

1836696

It accesses kernel internal structures directly; this needs to be worked out with some proper userspace-kernel interface Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

uORB: Build uorb systemcmd always in kernel side in memory protected …

9e1f9cf

…build Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

Separate i2c and spi board bus configuration into and own library for…

9380a7f

… protected build target This info is needed on both kernel and user sides, and is just data. Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

Put gyro_fft in kernel by default in protected build

53bdb15

It is a performance issue to run it on user side Signed-off-by: Jukka Laitinen <jukkax@ssrc.tii.ae>

jlaitine force-pushed the px4_protected_build_pub branch from 5fefddc to 53bdb15 Compare March 16, 2022 11:40

bkueng approved these changes Mar 17, 2022

View reviewed changes

bkueng merged commit 3d35929 into PX4:master Mar 17, 2022

jlaitine deleted the px4_protected_build_pub branch April 22, 2022 09:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PX4 protected build pub #16931

PX4 protected build pub #16931

jlaitine commented Feb 23, 2021 •

edited

LorenzMeier commented Feb 23, 2021

dagar commented Feb 23, 2021

jlaitine commented Apr 23, 2021

dagar commented Apr 23, 2021

jlaitine commented May 25, 2021

jlaitine commented Sep 6, 2021 •

edited

jlaitine commented Mar 8, 2022

jlaitine commented Mar 9, 2022

jlaitine commented Mar 15, 2022

bkueng Mar 16, 2022

jlaitine Mar 16, 2022

bkueng left a comment

jlaitine commented Mar 17, 2022

PX4 protected build pub #16931

PX4 protected build pub #16931

Conversation

jlaitine commented Feb 23, 2021 • edited

LorenzMeier commented Feb 23, 2021

dagar commented Feb 23, 2021

jlaitine commented Apr 23, 2021

dagar commented Apr 23, 2021

jlaitine commented May 25, 2021

jlaitine commented Sep 6, 2021 • edited

jlaitine commented Mar 8, 2022

jlaitine commented Mar 9, 2022

jlaitine commented Mar 15, 2022

bkueng Mar 16, 2022

Choose a reason for hiding this comment

jlaitine Mar 16, 2022

Choose a reason for hiding this comment

bkueng left a comment

Choose a reason for hiding this comment

jlaitine commented Mar 17, 2022

jlaitine commented Feb 23, 2021 •

edited

jlaitine commented Sep 6, 2021 •

edited