Support for data integrity checks

[willtoth]

Since typical UART/serial does not support correction of data errors it would be nice to be able to have some mechanism for this.

I would suggest two mechanisms:

1. Simple 8 or 16 bit checksum (simple but not perfect by any means)
2. 8, 16 or 32-bit CRC (configurable polynomial would be a bonus)

This would be appended to the end of the frame. If the checksum or CRC does not pass, drop the frame.

[alex-spataru]

Hi, I just implemented this feature (I still need to make tests though). You can select between crc8, crc16& crc32. To use this feature, simply add the checksum type and checksum data at the end of a frame. For example:

/*data_1, data_2, data_3, ..., data_n*/crc8:0x8A
/*data_1, data_2, data_3, ..., data_n*/crc16:0x7162
/*data_1, data_2, data_3, ..., data_n*/crc32:0xB45F5FBF

Note: the checksums must be added byte-by-byte, for example:

crc8: 0x8A
crc16: 0x71 0x62
crc32: 0xB4 0x5F 0x5F 0xBF

Also, the checksum should only account for the data inside the frame start/end sequences (do not compute the CRC with the /*and */).

The implementation of the CRC functions are available in src/IO/Checksum.cpp.

The verification procedure is defined here, I still need to create a test program on my microcontroller to validate that this implementation works correctly.

[alex-spataru]

Update: With the latest commit, checksum verification works without any issues (so far 😆).

Here's an Arduino program to test different checksum verifications with Serial Studio:

///
/// Frame buffer string
///
static char FRAME[100];

///
/// Strings to store ADC readings
///
char ADC1_STR[6];
char ADC2_STR[6];
char ADC3_STR[6];
char ADC4_STR[6];
char ADC5_STR[6];

///
/// Select CRC type
/// 
//#define CRC8
//#define CRC16
#define CRC32

///
/// CRC-8 implementation
///
#ifdef CRC8
uint8_t crc8(const char* data, const int length)
{
  uint8_t crc = 0xff;
  for (int i = 0; i < length; i++)
  {
    crc ^= data[i];
    for (int j = 0; j < 8; j++)
    {
      if ((crc & 0x80) != 0)
        crc = (uint8_t) ((crc << 1) ^ 0x31);
      else
        crc <<= 1;
    }
  }

  return crc;
}
#endif

///
/// CRC-16 implementation
///
#ifdef CRC16
uint16_t crc16(const char* data, const int length)
{
  uint8_t x;
  uint16_t crc = 0xFFFF;

  for (int i = 0; i < length; ++i)
  {
    x = crc >> 8 ^ data[i];
    x ^= x >> 4;
    crc = (crc << 8) ^ ((uint16_t)(x << 12)) ^ ((uint16_t)(x << 5)) ^ ((uint16_t)x);
  }

  return crc;
}
#endif

///
/// CRC-32 implementation
///
#ifdef CRC32
uint32_t crc32(const char* data, const int length)
{
  uint32_t mask;
  uint32_t crc = 0xFFFFFFFF;
  for (int i = 0; i < length; ++i)
  {
    crc = crc ^ data[i];
    for (int j = 8; j >= 0; j--)
    {
      mask = -(crc & 1);
      crc = (crc >> 1) ^ (0xEDB88320 & mask);
    }
  }

  return ~crc;
}
#endif

///
/// Reads voltage in the given @a pin and outputs it to the given @a str
///
inline void readAdc(const int pin, char* str) {
  const float voltage = analogRead(pin) * 5.0 / 1024;
  dtostrf(voltage, 4, 2, str);
}

///
/// MCU init function
///
void setup() {
  // Initialize serial port
  Serial.begin(115200);

  // Setup pins
  pinMode(A1, INPUT);
  pinMode(A2, INPUT);
  pinMode(A3, INPUT);
  pinMode(A4, INPUT);
  pinMode(A5, INPUT);
}

///
/// MCU loop function
///
void loop() {
  // Read ADC values and store them into strings
  readAdc(A1, ADC1_STR);
  readAdc(A2, ADC2_STR);
  readAdc(A3, ADC3_STR);
  readAdc(A4, ADC4_STR);
  readAdc(A5, ADC5_STR);

  // Create frame
  sprintf(FRAME, "%s,%s,%s,%s,%s", ADC1_STR, ADC2_STR, ADC3_STR, ADC4_STR, ADC5_STR);

  // Send frame to serial port
  Serial.print("/*");
  Serial.print(FRAME);
  Serial.print("*/");

  // Send CRC-8
#ifdef CRC8
  uint8_t crc = crc8(FRAME, strlen(FRAME));
  Serial.print("crc8:");
  Serial.write(crc);
#endif

  // Send CRC-16
#ifdef CRC16
  uint16_t crc = crc16(FRAME, strlen(FRAME));
  Serial.print("crc16:");
  Serial.write((crc >> 8) & 0xff);
  Serial.write((crc) & 0xff);
#endif

  // Send CRC-32
#ifdef CRC32
  uint32_t crc = crc32(FRAME, strlen(FRAME));
  Serial.print("crc32:");
  Serial.write((crc >> 24) & 0xff);
  Serial.write((crc >> 16) & 0xff);
  Serial.write((crc >> 8) & 0xff);
  Serial.write((crc) & 0xff);
#endif

  // Write end of line
  Serial.print("\n");

  // Wait 50 ms
  delay(50);
}

Here's the JSON map file that I used:

{"fe":"","fs":"","g":[{"d":[{"g":true,"max":5,"min":0,"t":"ADC 1","u":"V","v":"%1","w":"bar"},{"g":true,"max":5,"min":0,"t":"ADC 2","u":"V","v":"%2","w":"bar"},{"g":true,"max":5,"min":0,"t":"ADC 3","u":"V","v":"%3","w":"bar"},{"g":true,"max":5,"min":0,"t":"ADC 4","u":"V","v":"%4","w":"bar"},{"g":true,"max":5,"min":0,"t":"ADC 5","u":"V","v":"%5","w":"bar"}],"t":"ADC Readings","w":""}],"s":"","t":"ADC Test"}

And finally, here is a screenshot with the console output of the Arduino program:

@willtoth Please let me know what you think about this solution!

[willtoth]

I like it! Do invalid checksums get dropped/flagged?

[alex-spataru]

I like it! Do invalid checksums get dropped/flagged?

Hi! For the moment, frames with invalid checksums get dropped.

[alex-spataru]

I just created a new release that implements this feature. Please let me know how it works for you!