dmesg functionality (console buffer)

[bkueng]

Often it would be helpful to see the boot output, but the console is not available. This PR adds all console output to a buffer (except for NuttX syslog output), so that it can be printed later on via a new dmesg utility.

It is implemented by redirecting stdout to a custom device that keeps a buffer and outputs to stderr (which still goes to the original console). I looked at different options, and this seems the best one and most efficient.
Alternatives:

• using up_putc instead of stderr is considerably less efficient
• using NSH_ALTCONDEV does not work, because the mavlink shell would be output to that console as well.
• I tried redirecting via shell scripting (e.g. sh /etc/init.d/rcS2 > /dev/ramlog), but that did not work properly.
• NuttX's ramlog is not appropriate because it clears the buffer on every read
• NuttX provides a tee command, but that uses an additional 1 KB buffer

Since it requires some RAM, I only enabled it for v5 targets. The buffer is 4 KB, and typical boot output is 3 KB. We can reduce that a bit by cleaning up the boot output and maybe reduce the ver all verbosity.

Example

You can also keep it running in the background:

dmesg -f &

TODO

• find a better way for the v5 ifdef
• add output to the log as well

[LorenzMeier]

Awesomeness!

[julianoes]

This is awesome.

[davids5]

@bkueng

How much of the motivation for this is wanting a PX4 portable way to do this?

I tried redirecting via shell scripting (e.g. sh /etc/init.d/rcS2 > /dev/ramlog), but that did not work properly.
NuttX's ramlog is not appropriate because it clears the buffer on every read

I am assuming you are referring to the syslog options:

http://nuttx.org/doku.php?id=wiki:nxinternal:syslog Ramlog and dmesg?

Could we just add a backing file store to dmesg? If dmseg RAM has content append it to a file and then cat the file?

[bkueng]

Could we just add a backing file store to dmesg? If dmseg RAM has content append it to a file and then cat the file?

That would either be the ramlog option (which clears on read, so not what we want), or add a read interface to our console_buf device. We could do the second one, but need a way to handle the follow/not-follow reads. The added abstraction would also add overhead.

[davids5]

@bkueng

Would you agree the output to the console has different significance to different developers?

My concern is the loss of information: I.e the diff from this to the actual console output.

There are some early boot messages that are critical to diagnosis an issue. I would like to see these in the output, but as the system boots so grows the capabilities to log. I.E there is no microsd fs in stm32_boardinitialize(). and some fs capabilities at board_app_initialize() there is full capabilities after the script completes in board_app_finalinitialize().

[davids5]

@bkueng - thank you!
Let's add to the Todo list what we discussed on the call:

1. A plan to enhance the ver command, to capture the Manifest index(HW rev,ver) and Core rev/patch.
2. consider using the a stream in BBSRAM for the dmsg buffer to not lose the high level information in a fault.

[dagar]

Questions

1. Can you add this to the log?
2. What should we do with current syslog usage? https://github.com/PX4/Firmware/blob/4907b8788267240d665a923a2166fd1d3d77079b/boards/px4/fmu-v5/src/init.c#L294
3. Ideas for capturing stdout on posix and qurt as well?

[bkueng]

Can you add this to the log?

Yes, that was already on the way. See e0cccce

What should we do with current syslog usage?

There's only 2: board version (which is also in ver all) and some hardfault log status output. Everything we want should be added to a printf.

Ideas for capturing stdout on posix and qurt as well?

Yes we can use a pipe.

[bkueng]

Rebased & tested again. Should be good to go if CI passes.

There is one thing to note: output from NuttX's work queues is not captured, because they are started earlier at boot. But since we move away from them, it won't be an issue.

During testing I hit the maximum task limit of 32 (mavlink shell + 1 bg task). This is likely already an issue for VTOL and certain other configurations, due to the additional work queue tasks. @dagar fyi, maybe you're aware already.

[dagar]

During testing I hit the maximum task limit of 32 (mavlink shell + 1 bg task). This is likely already an issue for VTOL and certain other configurations, due to the additional work queue tasks. @dagar fyi, maybe you're aware already.

No, I didn't realize we were that close on actual configurations. I'll accelerate plans to consolidate some of our tasks, kill off HPWORK/LPWORK (unless used by a NuttX level dependency), and shutdown any unused WQ threads (created probing empty buses at startup).

[julianoes]

I had another glance over this after my previous review, and I'm assuming that @dagar and @davids5 are not opposed to merging this now.