[sd logger] Version 2.0 release, direct SPI logging, UART download #1392

bartslinger · 2015-10-27T13:31:08Z

This module provides a tool for high-speed logging where weight- and size-limitations play part.
It only requires a micro-SD card which is soldered directly to the SPI-bus, for example on a lisa/s.

The logger is activated by flipping a switch on the transmitter. Up to 42 logs can be recorded during one session (battery kept connected). The data remains on the card when the battery is disconnected, so that you can replace the battery before downloading data to the computer. Previous logs will be overwritten as soon as the first log is initiated after boot.

The messages to be logged can be selected in the telemetry config file, under the Logger process. Next to that, it is possible to call pprz_msg_send_xxx from anywhere in the code, using a reference to the sdlogger.

Telemetry config example:

<process name="Logger">
  <mode name="default">
    <message name="ROTORCRAFT_STATUS"      period="1.2"/>
    <message name="DL_VALUE"               period="0.5"/>
    <message name="ALIVE"                  period="2.1"/>
    <message name="IMU_GYRO_SCALED"        period=".002"/>
    <message name="IMU_ACCEL_SCALED"       period=".002"/>
    <message name="IMU_MAG_SCALED"         period=".1"/>
  </mode>
</process>

A tool is provided to download the data at high speed over UART:

Limitations: For now, it can only be used with uart downlink. This is perfect for lisa/s where UART pins are broken out.

podhrmic · 2015-10-27T16:21:43Z

Looks really cool! Does it work with regular (==non-ChibiOS) Lisa M/MX board?

bartslinger · 2015-10-27T17:52:08Z

@podhrmic I use it without ChibiOS on a Lisa/S. It is designed to work on Lisa M/MX as well (but not tested).

podhrmic · 2015-10-27T18:34:02Z

Great. I ll test it on Lisa MX

podhrmic · 2015-10-28T00:13:10Z

Hi @bartslinger - sadly it doesn't work on Lisa MX. I have the SD card connected at SPI1, I am using a Quad_Lisa_MX example with the logger module, and it never writes anything on the card (no matter what position is the radio switch). Any ideas?

bartslinger · 2015-10-28T07:43:50Z

@podhrmic It does not work with the logger module. The card needs to be connected directly to the spi port like this:

bartslinger · 2015-10-28T08:51:07Z

@podhrmic Could you try if the LED goes on when you flip the switch? For the Lisa/S, I had to disable another LED in the board makefile. Then, in your airframe config it should be something like this

 <modules main_freq="512">
   <load name="logger_sd_spi_direct.xml">
     <configure name="LOGGER_CONTROL_SWITCH" value="RADIO_AUX2"/>
     <configure name="LOGGER_LED" value="3"/>
   </load>
   ...
 </modules>

podhrmic · 2015-10-28T16:15:18Z

I am using SD card breakout from Sparkfun: https://www.sparkfun.com/products/544
soldered directly to the SPI bus.
By logger module I mean "logger_sd_spi_direct.xml" module.
I ll give it another shot today.

bartslinger · 2015-10-28T16:44:06Z

Ok that should be fine. Please note that it does not use a filesystem like FAT, so you won't be able to see the logs if you insert the card in the computer directly.

An SD Logger settings tab should also appear in the ground station. Could you check which value sdcard1.status and sdlogger_spi.status have?

podhrmic · 2015-10-28T16:57:43Z

ah I see. So in that case how do I download the data? Do I just plug the autopilot into USB to get the ACM device?

podhrmic · 2015-10-28T17:06:33Z

Also, here is my status tab

podhrmic · 2015-10-28T17:15:23Z

Tested with different LED and different radio configuration. The radio SWITCH inputs gets through, but never triggers the logging or the LED. Any ideas?

bartslinger · 2015-10-28T19:44:19Z

The sdcard1.status = 1 means an error in (the connection to) the sd-card, probably already during initialization. I have not yet built in a way to see where it fails exactly, but can add this relatively easily. At 15 different locations in sdcard_spi.c the sdcard1.status is set to SDCard_Error. If you know exactly at which instance it fails, you have more information.

The LED only goes ON if the logging has actually started. That explains why it doesn't turn on.

Downloading can be done by connecting the boards UART port to the computer with FTDI or BlackMagicProbe.

bartslinger · 2015-10-28T20:09:50Z

I now added a setting to see where the error occurred. Let me know what it says.

podhrmic · 2015-10-28T22:06:31Z

Here it is - my guess is that it is SDCardError_InitializationNoResponse error so I will double check my connections and let you know

flixr · 2015-10-28T22:09:33Z

Without having looked at this in detail yet, it sounds like a really great modue/addition!
But I'm wondering why you are adding variables to settings in order to get debug info... it's not like you want to actually change those "settings"...

bartslinger · 2015-10-28T22:17:41Z

@flixr You're absolutely right about that. Is there another way to get those values on request?

podhrmic · 2015-10-28T22:20:05Z

Yep, still the same error. I have the connections as:
LISA MX -> SD_CARD
SPI1_SS -> CS
SPI1_SCK -> SCK
SPI1_MISO -> DO
SPI1_MOSI -> DI
3V3 -> VCC
GND -> GND

Using a scope I can't see any data coming through, nor even a toggle on CS or SCK. Are the pins properly defined on Lisa MX?

podhrmic · 2015-10-28T22:20:32Z

A dedicated telemetry message?

bartslinger · 2015-10-28T22:32:58Z

Do you have the following defines in var/generated//ap_srcs.list?

-DSPI_MASTER=1
-DUSE_SPI1=1
-DSDLOGGER_SPI_LINK_DEVICE=spi1
-DUSE_SPI_SLAVE1=1
-DSDLOGGER_SPI_LINK_SLAVE_NUMBER=SPI_SLAVE1

flixr · 2015-10-29T11:44:51Z

Yep, a dedicated telemetry message (or using the DEBUG message) is the other option...

@podhrmic do you use the correct SPI slave (SS pin), check against the lisa/mx board file.

podhrmic · 2015-10-30T22:51:41Z

OK, I have good news - adding these two lines into airframe config fixed the logging:

      <configure name="SDLOGGER_DIRECT_SPI" value="spi1"/>
      <configure name="SDLOGGER_DIRECT_SPI_SLAVE" value="spi_slave1"/>

So now the whole config for Lisa MX looks like this:

    <load name="logger_sd_spi_direct.xml">
      <configure name="LOGGER_CONTROL_SWITCH" value="RADIO_SWITCH_A"/>
      <configure name="LOGGER_LED" value="5"/>
      <configure name="SDLOGGER_DIRECT_SPI" value="spi1"/>
      <configure name="SDLOGGER_DIRECT_SPI_SLAVE" value="spi_slave1"/>
   </load>

(I am using Taranis SBUS radio).

The LED lights up when I flip the switch, so that looks good. Now, how do I download the data? I know there is the UART tool, but how do I use it?

Thanks!

bartslinger · 2015-10-30T23:23:35Z

You need to run sdlogger_download in sw/logalizer. The screenshots in the first post show an example. Make sure you use the same uart as is used for telemetry. That is currently the only supported method.

podhrmic · 2015-10-30T23:29:19Z

I am getting a seg fault:

./sdlogger_download -a 30 -p /dev/ttyUSB0 -b 57600
Segmentation fault (core dumped)

(the ac_id and the port are definitely correct)
Any ideas?

bartslinger · 2015-10-31T07:52:16Z

Damn, should have done that one test-driven as well. I will try to find and fix that, although its hard to debug since I don't have this problem.

gautierhattenberger · 2015-11-12T20:18:42Z

Correct, if it can find a GPS message in the log, sd2log will use it to rebuild the correct global date of the log. Otherwise the current time of the system that runs the command is used.

flixr · 2015-11-14T13:27:21Z

Would be great if you could add some more explanation/docs to the module xml file.
This should actually be preferred over wiki docs (which are more suitable for step-by-step examples, etc..).

flixr · 2015-11-14T13:30:20Z

sw/airborne/modules/loggers/sdlogger_spi_direct.h

+
+#include "std.h"
+#include "mcu_periph/link_device.h"
+#include "subsystems/radio_control.h"


only link_device and spi includes actually seem to be needed in this header.
Better to only include the reset in the .c file where they are actually needed.

flixr · 2015-11-14T13:42:45Z

Would be nice if this wouldn't depend on the radio control subsystem and make this optional (and protected with #ifdef RADIO_CONTROL).
So it would be nice to actually use the start and stop functions of the module, then you could also control the logging without RC from the GCS or the flight plan...

flixr · 2015-11-14T13:53:55Z

conf/modules/logger_sd_spi_direct.xml

+  <init fun="sdlogger_spi_direct_init()"/>
+  <periodic fun="sdlogger_spi_direct_periodic()" freq="512" start="sdlogger_spi_direct_start()" stop="sdlogger_spi_direct_stop()" autorun="TRUE"/>
+  <datalink message="SETTING" fun="sdlogger_spi_direct_command()"/>
+  <makefile>


Would make sense to set the target to ap here and merge the other makefile section into it, otherwise it will try to build the module for nps, sim, etc as well and fail....

flixr · 2015-11-14T14:05:35Z

Can you please explain the need for changing telemetry_mode_Main (and hence requiring a TELEMETRY_MODE_Main_empty)?
This also means that it will only work for rotorcraft firmware as it is...

flixr · 2015-11-14T14:06:19Z

sw/airborne/modules/loggers/sdlogger_spi_direct.h

+  uint8_t buffer[SDLOGGER_BUFFER_SIZE];
+  uint8_t idx;
+  uint32_t log_len;
+  uint8_t command;


What are these commands?

flixr · 2015-11-19T14:18:14Z

@bartslinger since this is very useful, would be great if we could merge it ASAP.
Do you have time to fix the xml file? Otherwise we can merge it and I will try to fix it up...
Triggering the logging via datalink (without RC) can be addressed later...

And it would be great if you could explain why you added an empty mode... why is that needed?
If it is to prevent other messages from being sent at the same time, I just want to point out that this will only stop the periodic messages (from whatever mode is otherwise active). You can't rely on that no other messages are sent (as other parts of the code/modules also directly send messages on certain events).

OpenUAS · 2015-11-19T15:33:45Z

Relevant Wiki documentation: https://wiki.paparazziuav.org/wiki/SDLogger_SPI_Direct to be extended over time

OpenUAS · 2015-11-19T15:35:42Z

It would be great to have it merged. Then we have +4 students using PPRZ master who will test it in real life and can make improvements, give feedback and add documentation and examples.

OpenUAS · 2015-11-19T15:45:34Z

Improvement and fixes to be done after pulled into master:

Fix in module description documentation to make it usable
Must be somehow also usable in basic form without Radio control
Fix code style
More extensive Wiki docs and examples
Needs real life testing
Reader tool unification: https://github.com/karlito139/lisa_s_logger_reader

bartslinger · 2015-11-19T16:24:21Z

Ok my priorities have shifted a bit, but I can spend some time on this tomorrow.

OpenUAS · 2015-11-19T20:24:02Z

@bartslinger , not that you need to fix the issues alone, it's more that if in master those are the point to improve soon afterwards. I will be of assistance and finish some task for that.

For now fixing the most important remarks of @flixr after the merge.

IMHO, yes plz let's merge it, we have commitment of multiple to improve soon after merge.

podhrmic · 2015-11-19T20:54:27Z

IMHO there is no point if merging an unfinished feature - the pull request can be tested as is, improved and then finally merged in master. I would like to use it too, these are just my 2 cents.

bartslinger · 2015-11-20T12:59:06Z

start/stop functionality

The initialization sequence is called on startup of the system. The initialization involves periods of waiting which is implemented by using periodic function. Therefore, the module is started from the beginning. I would have preferred RTOS for this since you then need to call initialization only once. It is possible to rebuild it such that the card initialization happens when the first logs starts, but that would delay the first measurement. However, that would allow usage of start/stop functions of the module. I'm not planning to implement this in this version.

Empty message

The downloading of data is implemented by putting all logged data on the uart bus. I only want to save logger messages, so therefore I disabled other messages with empty telemetry config. If you have other suggestions to disable all messages except from the sdcard, let me know.

command / settings

The command value is set by the 'setting', which calls the sdlogger_spi_direct_command function. <datalink message="SETTING" fun="sdlogger_spi_direct_command()"/>. command = 255 is called to receive the index block. Commands 1-42 are for downloading that log number.

flixr · 2015-11-20T13:38:47Z

@bartslinger thanks a lot for fixing the target, cleaning up and the clarification!

Regarding the periodic function:
You can have more than one that you should be able to start/stop separately. That could be one solution to starting/stopping logging via datalink later.

Regarding the empty periodic messages mode:
That is what I though, and I don't think we have any other mechanism to disable all periodic messages right now @gautierhattenberger ?
My main concern (apart that only this ties it to the rotorcraft firmware) was that this doesn't prevent other messages from being sent "directly", e.g. modules with x_SYNC_SEND enabled, etc...
So is it a problem if there are other messages sent while transferring the log, or did you just want to lower the data rate by disabling most other messages?

If the downloader can correctly handle some other messages in between now and then, I think this is good to merge. If not, we should address this.

bartslinger · 2015-11-20T14:02:40Z

Actually, I don't know, because the data is processed by sw/logalizer/sd2log which is written by @gautierhattenberger.

So it's possible to have multiple start/stop/periodic functions per module? How?

flixr · 2015-11-20T15:58:43Z

If that is a problem, then one way to deal with this would be to add a specific message for sending log data and only parse this message in the downloader on the ground... while that would incur some more overhead, I think it would be the way to go...
@gautierhattenberger what do you think?

You can add multiple periodic functions in the module xml and assign them different start/stop functions:
https://wiki.paparazziuav.org/wiki/Modules#In_the_module_file

gautierhattenberger · 2015-11-20T20:29:51Z

Maybe the mistake was to use the same start byte for pprz and logger format, therefore we can't correctly split them.
Actually, you can probably run sd2log on a file with both normal telemetry and log data, the messages from the normal telemetry will just fail on the checksum check. So it might work... @bartslinger did you tried ?

What a about a null device that we can set as default telemetry device to disable them ? Probably not relevant here since you couldn't change at runtime.

flixr · 2015-11-21T17:36:25Z

So if this is a problem, I guess it will some more work which should be addressed in a separate pull request.
I would say we merge this as is and then see how it goes.
Do you agree @gautierhattenberger ?

gautierhattenberger · 2015-11-21T20:46:00Z

yes, agreed

This module provides a tool for high-speed logging where weight- and size-limitations play part. It only requires a micro-SD card which is soldered directly to the SPI-bus, for example on a lisa/s. The logger is activated by flipping a switch on the transmitter. Up to 42 logs can be recorded during one session (battery kept connected). The data remains on the card when the battery is disconnected, so that you can replace the battery before downloading data to the computer. Previous logs will be overwritten as soon as the first log is initiated after boot. The messages to be logged can be selected in the telemetry config file, under the Logger process. Next to that, it is possible to call pprz_msg_send_xxx from anywhere in the code, using a reference to the sdlogger. See also http://wiki.paparazziuav.org/wiki/SDLogger_SPI_Direct

[sd logger] Version 2.0 release, direct SPI logging, UART download

ec15491

[sdlogger] added sdcard error status

eef767e

[sdlogger] removed some non-setting settings

5c84f12

OpenUAS added this to the v5.8 milestone Oct 31, 2015

OpenUAS added Airborne Enhancement Improve upon exiting functionality labels Oct 31, 2015

flixr reviewed Nov 14, 2015
View reviewed changes

bartslinger added 2 commits November 20, 2015 11:52

[sdlogger] changed includes, removed unused func decl

a841886

[sdlogger] conf added description, target to ap, consistent naming

c644e5c

flixr merged commit c644e5c into paparazzi:master Nov 22, 2015

bartslinger deleted the sdlogger_v2_release branch November 22, 2015 15:27

[sd logger] Version 2.0 release, direct SPI logging, UART download #1392

[sd logger] Version 2.0 release, direct SPI logging, UART download #1392

Conversation

bartslinger commented Oct 27, 2015

podhrmic commented Oct 27, 2015

bartslinger commented Oct 27, 2015

podhrmic commented Oct 27, 2015

podhrmic commented Oct 28, 2015

bartslinger commented Oct 28, 2015

bartslinger commented Oct 28, 2015

podhrmic commented Oct 28, 2015

bartslinger commented Oct 28, 2015

podhrmic commented Oct 28, 2015

podhrmic commented Oct 28, 2015

podhrmic commented Oct 28, 2015

bartslinger commented Oct 28, 2015

bartslinger commented Oct 28, 2015

podhrmic commented Oct 28, 2015

flixr commented Oct 28, 2015

bartslinger commented Oct 28, 2015

podhrmic commented Oct 28, 2015

podhrmic commented Oct 28, 2015

bartslinger commented Oct 28, 2015

flixr commented Oct 29, 2015

podhrmic commented Oct 30, 2015

bartslinger commented Oct 30, 2015

podhrmic commented Oct 30, 2015

bartslinger commented Oct 31, 2015

gautierhattenberger commented Nov 12, 2015

flixr commented Nov 14, 2015

flixr Nov 14, 2015

Choose a reason for hiding this comment

flixr commented Nov 14, 2015

flixr Nov 14, 2015

Choose a reason for hiding this comment

flixr commented Nov 14, 2015

flixr Nov 14, 2015

Choose a reason for hiding this comment

flixr commented Nov 19, 2015

OpenUAS commented Nov 19, 2015

OpenUAS commented Nov 19, 2015

OpenUAS commented Nov 19, 2015

bartslinger commented Nov 19, 2015

OpenUAS commented Nov 19, 2015

podhrmic commented Nov 19, 2015

bartslinger commented Nov 20, 2015

start/stop functionality

Empty message

command / settings

flixr commented Nov 20, 2015

bartslinger commented Nov 20, 2015

flixr commented Nov 20, 2015

gautierhattenberger commented Nov 20, 2015

flixr commented Nov 21, 2015

gautierhattenberger commented Nov 21, 2015