TeensyDB

Remember the infamous DBase III from the early '80's? Well here it is for your Teensy microcontroller (Arduino / ESP are not supported) and select flash chips. This library is a database system for SPI-based flash memory chips and uses a field/record approach in saving data to a chip. Some advantages of saving data to a flash-type chip are no physical contacts to loose connection, but have very fast writes and reads. This driver can save ~50 bytes in under 1ms. Since this library has no "close" method to finalize a write transation, saved data can be read even if power was lost during data collection. While you can save data to an SD card, the classic open/write/close has a huge overhead the can take in the 100's of ms to execute. This claim is based on open/write/close repeat. Having Open and Close outide the save loop is much faster but risks loosing data in cases of unexpected power loss.

This library has been tested with
MCU
Teensy 3.2
Teensy 4.0

Flash Chips
Microchip SST25F040C (https://ww1.microchip.com/downloads/en/DeviceDoc/SST25PF040C-4-Mbit-3.3V-SPI-Serial-Flash-Data-Sheet-DS20005397E.pdf)
Winbond 25Q64JVSIQ (https://www.winbond.com/resource-files/w25q64jv%20revj%2003272018%20plus.pdf)

This driver is intended for data acquistion systems where known data is to be stored. Namely because fields must be established before any operations, and that field list must match any existing database schemes on the chip. Data types is stored in fields (similar to a columns in a spreadsheet, and each record of data values are store in rows. You can easily save measurements such as volts in a volt field, temperature in a temp field, etc. Since a known scheme is required, it's not intended for saving video, images, or "random" data. Because of the database scheme, you cannot change the fields unless you erase the chip first.

If you are not familiar with fields and records, fields are the columns, and records are the rows. Similar to:

Recordset	Datapoint	Time	Temp1	Temp2	Temp2
1	1	10:00	23.4	45.2	63.1
1	2	10:01	23.6	45.1	65.4
1	3	10:02	23.7	45.0	67.2

This driver lets you create fields of specified data types, then in some measurement loop add a new record, save a record, and repeat. As with many flash chips you CANNOT write to an address unless it's in the erased state hence this library requires data to be written and stored contiguous. This driver will find the next available writable address so if you power up your system, and start saving data, you can be sure you will be writing to valid addresses. The field definition process passes pointers into the library so the save process simply looks at the data you already have in memory. This design keeps you from having to populate and save a bunch of fields. One addRecord() call and one saveRecord() call is all that is needed to save your data to the chip.

Library highlights

1. relatively small footprint
2. very fast write times (approx 32 microseconds per byte)
3. ability to add up to 255 fields
4. ability to add a new record (required for each record save)
5. ability to save a record with a single call
6. ability to goto a record
7. ability to read the data out of a field
8. ability to get total records so save can start at a valid address
9. ability to save bytes, ints, uint32_t, floats, char[fixed_length], doubles, more... But sorry STRING is not supported.
10. ability to get chips stats (JEDEC codes, and used space)
11. ability to erase a sector or the entire chip (caution: the chip requires contiguous memory for writing so memory in the middle cannot only be erased)
12. Only 1 field scheme is allow between chip erases
13. a concept of a field called "RecordSet" could be used to distinguish one set of readings from another--similar to a file number
14. this library writes data to the chip byte by byte and not byte arrays. This does impede performance, but improves write reliability.

Pin Connection

flash chip	CE	SO/SIO1	WP	VSS	VDD	HOLD	SCK	SI/SIO0
Teensy	6 (user selection)	12 (MISO)	3v3	GND	3v3	3v3	13 (SCK)	11 (MOSI)

General implementation

1. include the library
#include "TeensyDB.h"

2. create the chip object
TeensyDB YOUR_OBJECT_NAME(THE_CHIP_SELECT_PIN);

3. create variables
float MyVolts = 0.0;
int MyVoltsID = 0;
uint32_t LastRecord = 0, i = 0;

4. In setup, create data fields
MyVoltsID = YOUR_OBJECT_NAME.addField("Volts", &MyVolts);

5. In setup, get the last writable record

LastRecord = YOUR_OBJECT_NAME.findFirstWritableRecord();
YOUR_OBJECT_NAME.gotoRecord(LastRecord);

6. In some measurement loop
MyVolts = analogRead(A0);

7. Add a new record
YOUR_OBJECT_NAME.addRecord();

8. Save the record
YOUR_OBJECT_NAME.saveRecord();

9. when you are ready to read the data...
LastRecord = YOUR_OBJECT_NAME.getLastRecord();
for (i = 1; i <= LastRecord; i++) {
  Serial.print("Record: ");
  Serial.print(i);
  Serial.print(" - ");
  Serial.print(YOUR_OBJECT_NAME.getField(MyVolts, MyVoltsID ));
  Serial.print(", ");
}


Performance comparison between 3 different storage options 1) flash chip using this library 2) standard SD card using SdFat 3) flash chio using LittleFS. Users can chose when to close files with SD cards and LittleFS, either close after all data is collected, or close after each datapoint is collected. There are several cases that require the latter, namely when power down is unpredictable in which results in data loss due to lack of file closure. Since this library has no concept of opening and closing, there is no chance of lost data in the even of an unplanned loss of power.

The fastest performance is with a flash chip and LittleFS in a open once / close once scenario. Howerver this configuration becomes very slow when open / close is performed for each data write. SD cards are fast again only in open / close once scenarios. This library offers high performance withought sacrificing data write integridy.



Testing shows very reliable record writes and reads. This images shows 500,000 writes and reads with no issues.

This image shows the inspiration for developing this library. Some 55 bytes are being written every second, and on occassion, the SD write proceess corrupts data. Notice the incorrect data highlighted. This error rarely occurrs (estimated at 1 data point / 10,000) and is not reproducable but not acceptable. During testing and usage, this library has written 500 mb bytes without losing 1 bit.