SD card with ESP8266 for data logging

[pkkrusty]

After muddling through a number of discussions, but no real comprehensive guides for SD card and ESP8266, I got my setup to work, so I'm writing it here for posterity. ESP32 boards are easily capable of dealing with SD cards, but it wasn't clear how many folks have successfully used SD cards with 8266, especially since April 2021 was the earliest one could even compile successfully after the fix from #11724

Goal is to log sensor data to a microSD card while also doing all the normal Tasmota stuff like InfluxDB writes, MQTT data, Syslog messages which are all going over the network to another device (RPi 4b). In case of network/power failure, the (battery-backed-up) ESP8266 would still log sensor data, which could be referenced.

SPI pins are all default, not sure if changing from the hardware SPI pins affects success or not. DS18B20 was used just to have some sensor data. Make sure to power the SD card board with 5v, or you'll bang your head against the wall for two days while your firmware builds fail to see the SD card.

user_config_override.h has the following #defines, since I wanted to use scripting for actual writing of sensor data:

// Select none or only one of the below defines USE_RULES or USE_SCRIPT
#ifdef USE_RULES 
#undef USE_RULES                                // Add support for rules (+8k code)
#endif

#ifndef USE_SCRIPT
#define USE_SCRIPT
#endif
#define SDCARD_DIR //Not essential I think
#define USE_SCRIPT_FATFS_EXT //Not essential I think
#define USE_SCRIPT_FATFS //Not essential I think
#define USE_UFILESYS
#define USE_SDCARD
#define SDCARD_CS_PIN 4 //Not strictly necessary since the same #define happens in xdrv_50_filesystem.ino

//Further down I define SPI

#define USE_SPI

In platform_override.ini, I commented out
;board = ${common.board}
and uncommented
board = esp8266_4M2M
to have the option of using the extra 2mb flash on the Wemos knockoff chip I'm using. Also not essential for the SD card to work.

This all compiled correctly and I flashed to my board. Defined the pins and web GUI works as intended.

Update Using esptool to flash firmware will result in ufilesystem failure unless you change the command from esptool.py --port /dev/tty.wchusbserialfa130 write_flash -fs 1MB -fm dout 0x0 tasmota.bin to esptool.py --port /dev/tty.wchusbserialfa130 write_flash -fs 4MB -fm dout 0x0 tasmota.bin. Uploading firmware via the Tasmota GUI works correctly. (Minimal then full)

Note that clicking "Manage File System" will take a few seconds to load as the firmware calculates used and available space of a large SD card.

Sensor logging script taken from the cookbook works great after it's been tweaked to match my setup.

>D 48
temp=0
fr=0
res=0
cnt=0
; moving average for 60 seconds, also possible to do median value with lower case m, which seems to help when
; 8266 has heavy loads, the temp sensor data doesn't get read correctly (reads as 0), pulling the average down
M:mtemp=0 60
str=""

>B
; set sensor file download link
;fl1("slog.txt")
; delete file in case we want to start fresh
;fd("slog.txt")

; list all files in root directory
fr=fo("/" 0)
for cnt 1 20 1
res=fr(str fr)
if res>0
then
print %cnt% : %str%
else
break
endif
next
fc(fr)

>T
; get sensor values
temp=DS18B20#Temperature

>S
; average sensor values every second
mtemp=temp

; write average to sensor log every minute
if upsecs%60==0
then
; open file for write (note that in the guide, the file open command has a 1 instead of 2. 
;I want to append sensor values instead of overwriting each time)
fr=fo("slog.txt" 2)
; compose string for tab delimited file entry
str=tstamp+"\t"+s(mtemp)+"\n"
; write string to log file
res=fw(str fr)
; close file
fc(fr)
endif

>R

[TrDA-hab]

Cool !!!

[plewka]

I'm more on the ESP32, but this is great logging option.
Didn't know it is even possible with the ESP8266.
Thanks implementation and for details how to do!

[sfromis]

As long as you have 4MB of flash, you can also get a file system on ESP8266 without adding a SD card.

[pkkrusty]

For the 2mb you get with a Mini D1, I calculated I could get around 50 days of temp data writing values every minute. Could get quite a bit longer by logging every 5 minutes and using a shorter date format.

[prabakaran3787]

I'm using esp32 with sdcard adapter. Since Iam uploading the above script, it results as error.

I have connected the pins of esp32 with sdcard adapter as follows,
CS ---‐> D5
SCL---->D18
MISO--'>D23
MOSI--->D19

[ascillato]

Please, for support, can you open a separated issue and show the error and logs? Thanks.

[prabakaran3787]

Have to connect Esp32 device along with sdcard adapter and temperature sensor. How to store the temperature data in sdcard Data is not get stored in sdcard by trying your said method I'm follow this link (#13938)

…

On Fri, 24 Dec 2021, 11:19 pm pkkrusty, ***@***.***> wrote: For the 2mb you get with a Mini D1, I calculated I could get around 50 days of temp data writing values every minute. Could get quite a bit longer by logging every 5 minutes and using a shorter date format. — Reply to this email directly, view it on GitHub <#13938 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUISF4GYMO7I5X5NTGQOMMDUSSXBHANCNFSM5JPW475Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[pkkrusty]

Did you compile custom firmware with correct define statements? What temp sensor are you using? What does the console tell you when the ESP32 starts up?

This guide is meant for ESP8266, I haven't tried with ESP32.

[prabakaran3787]

Thanks for your reply and support sir.. Yes sir, I have compiled custom firmware using below statements. // Select none or only one of the below defines USE_RULES or USE_SCRIPT #ifdef USE_RULES #undef USE_RULES // Add support for rules (+8k code) #endif #ifndef USE_SCRIPT #define USE_SCRIPT #endif #define SDCARD_DIR //Not essential I think #define USE_SCRIPT_FATFS_EXT //Not essential I think #define USE_SCRIPT_FATFS //Not essential I think #define USE_UFILESYS #define USE_SDCARD * #define SDCARD_CS_PIN 5 *//Not strictly necessary since the same #define happens in xdrv_50_filesystem.ino //Further down I define SPI #define USE_SPI I am using ( DTH11) temperature sensor. While starting ESP32, it is showing as " file does not exist". We are using ESP32 only. So, I am expecting statements related to that sir. ESP32 connection sdcard adapter and temp: SD Card Interface MOSI GPIO-23 MISO GPIO-19 SCK GPIO-18 CS GPIO-5 DHT11 Interface OUT GPIO-25 and script alos uploaded for below statement

D 48

temp=0 fr=0 res=0 cnt=0 ; moving average for 60 seconds, also possible to do median value with lower case m, which seems to help when ; 8266 has heavy loads, the temp sensor data doesn't get read correctly (reads as 0), pulling the average down M:mtemp=0 60 str=""

B

; set sensor file download link ;fl1("slog.txt") ; delete file in case we want to start fresh ;fd("slog.txt") ; list all files in root directory fr=fo("/" 0) for cnt 1 20 1 res=fr(str fr) if res>0 then print %cnt% : %str% else break endif next fc(fr)

T

; get sensor values temp=DHT11#Temperature

S

; average sensor values every second mtemp=temp ; write average to sensor log every minute if upsecs%60==0 then ; open file for write (note that in the guide, the file open command has a 1 instead of 2. ;I want to append sensor values instead of overwriting each time) fr=fo("slog.txt" 2) ; compose string for tab delimited file entry str=tstamp+"\t"+s(mtemp)+"\n" ; write string to log file res=fw(str fr) ; close file fc(fr) endif

…

R On Mon, Dec 27, 2021 at 1:29 AM pkkrusty ***@***.***> wrote: Did you compile custom firmware with correct define statements? What temp sensor are you using? What does the console tell you when the ESP32 starts up? This guide is meant for ESP8266, I haven't tried with ESP32. — Reply to this email directly, view it on GitHub <#13938 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUISF4AN5SRETBWZ3Y6TKN3US5XYXANCNFSM5JPW475Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

-- With Regards, Prabakaran S

[pkkrusty]

Do you see the web interface for the File System? If that is working, then it sounds like the script is the problem. Since it says "File not defined", I think you need to uncomment the line that defines the log file in the >B section. Or just upload a blank file with the name slog.txt.

; set sensor file download link
fl1("slog.txt")

instead of

; set sensor file download link
;fl1("slog.txt")

[pkkrusty]

However, I'm checking my setup and it works fine the way it's written, leaving both things commented out. So I'm not sure what the issue is.

[prabakaran3787]

Hello sir, Iam using this ()bin file and configuration file. But not getting output. Script also uploaded as you said sir. But again it's showing as "file opened failed". So, please send bin file and configuration setup for ESP32.

…

On Mon, Dec 27, 2021 at 6:44 PM pkkrusty ***@***.***> wrote: However, I'm checking my setup and it works fine the way it's written, leaving both things commented out. So I'm not sure what the issue is. — Reply to this email directly, view it on GitHub <#13938 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUISF4CUKM7JU63WAWAZYW3UTBP5JANCNFSM5JPW475Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

-- With Regards, Prabakaran S

[pkkrusty]

I only have made it work with esp8266, I don't know exactly how to make esp32 work. Sorry.

[prabakaran3787]

reply pls On Thu, Jul 21, 2022 at 11:56 AM praba karan ***@***.***> wrote:

…

Good morning Sir, I need one help sir. What is the procedure to convert automation industrial Current loop 4 - 20 mA to Voltage in esp8266 or esp32. How to use this in Tasmota. On Wed, Dec 29, 2021 at 1:32 PM pkkrusty ***@***.***> wrote: > I only have made it work with esp8266, I don't know exactly how to make > esp32 work. Sorry. > > — > Reply to this email directly, view it on GitHub > <#13938 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AUISF4FTGUCIXODGF6H3YMDUTK6BBANCNFSM5JPW475Q> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > You are receiving this because you commented.Message ID: > ***@***.***> > -- With Regards, Prabakaran S

-- With Regards, Prabakaran S

[pkkrusty]

Like I said before, I'm not sure how to make it work with ESP32. Haven't taken the time to get everything hooked up to test.