ESP8266 issue

[luc-github]

Hi I use latest git version of SdFat with my ESP8266 and got error when trying to download "big" files.
File 451B is ok but with 174KB and up I got this :

Panic C:\Users\user\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.3.0\cores\esp8266\core_esp8266_main.cpp:98 __yield

ctx: sys 
sp: 3ffffbc0 end: 3fffffb0 offset: 01b0

>>>stack>>>
3ffffd70:  00000003 0140b7be 00000000 00000000  
3ffffd80:  3ffed7ee 1c01a8c0 00000003 4021b0bd  
3ffffd90:  3fff0b80 3fff09c0 3fff09e4 40217735  
3ffffda0:  3fff0b80 3fff09bf 3fff09e4 00000000  
3ffffdb0:  00000000 00000011 3fff09e4 402178b1  
3ffffdc0:  00007653 00000001 3fff09e4 402176d3  
3ffffdd0:  00000000 3fff07c0 3fff09e4 40217aad  
3ffffde0:  4023ab43 00007653 3fff07b4 40208043  
3ffffdf0:  00000000 00007653 3fff07b4 40217210  
3ffffe00:  3fff2000 00000001 00000001 402162d8  
3ffffe10:  00000001 3ffffe40 00000013 3fff27c0  
3ffffe20:  3fff27c4 3fff52cc 3fff27bc 000005b4  
3ffffe30:  3fff5fe4 3fff1ff0 3fff1ff0 40208b54  
3ffffe40:  0000002e 3ffeacb9 00000001 4010068c  
3ffffe50:  3fff2e7c 00000000 3fff1ff0 40219fa7  
3ffffe60:  3fff109c 000004f0 000003c8 40219feb  
3ffffe70:  3fff514c 3fff27c8 3fff514c 3fff27bc  
3ffffe80:  3fff52cc 000005b4 3fff53ac 40211c37  
3ffffe90:  3fff514c 000005b4 3fff523c 40211de7  
3ffffea0:  40230000 3fff52d0 3fff52cc 3fff27bc  
3ffffeb0:  3fff52cc 3fff27c4 3fff523c 40211f14  
3ffffec0:  3ffeb698 c162de08 3fff27c0 40211f34  
3ffffed0:  3fff52cc 3fff27c4 3fff27c0 4023b059  
3ffffee0:  00000000 1c01a8c0 00000010 00000000  
3ffffef0:  00000000 00000010 00000102 401075f8  
3fffff00:  40240000 00000000 0000007d 3fff2864  
3fffff10:  3ffed7da 3fff286c 3fff5414 40239479  
3fffff20:  3fff26b8 3fff2e7c 3fff2e7c 3ffefb30  
3fffff30:  00000000 3fff5414 0000001c 3fff2e7c  
3fffff40:  3ffed7cc 00000000 3fff5414 40238875  
3fffff50:  8701a8c0 3ffeadb4 00dcec39 00000000  
3fffff60:  40104b62 00dcec39 3fff0180 60000600  
3fffff70:  40241b9c 3fff0158 3fff0180 10e0b48f  
3fffff80:  40241bc2 3fffdab0 00000000 3fff2edc  
3fffff90:  3fffdc80 00000000 3fff5414 402417bf  
3fffffa0:  40000f49 3fffdab0 3fffdab0 40000f49  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(1,7)


 ets Jan  8 2013,rst cause:4, boot mode:(1,7)

wdt reset

Using https://github.com/tannewt/SdFat fork, which is pre-SDFat-Beta and has some patch for ESP8266, this issue does not happen, I can even download 10M files, but tannewt one crash when using clustercount functions:

uint32_t volTotal = SD.vol()->clusterCount();
uint32_t volFree = SD.vol()->freeClusterCount();

when yours do not, even answer is slow there is no watchdog issue.

I have applied same patch @tannewt did on his fork to your git version but this did not help.

Anything I can do to help to debug this issue ?

Thanks
I add decoded

[greiman]

Let me know when you find the problem. I really depend on users to support ESP8266.

I ran the bench example on a Adafruit HUZZAH ESP8266 Breakout and it will write and read a 5 MB file.

Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0X1B
OEM ID: SM
Product: 00000
Version: 1.0
Serial number: 0X725D8465
Manufacturing date: 10/2015

File size 5 MB
Buffer size 512 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
445.84,32564,989,1147
441.20,33692,988,1159

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
972.51,1395,519,525
972.13,2198,518,525

[luc-github]

you mean I provide you a test sketch to reproduce issue ?

[greiman]

No you must find the problem since I don't want to dig through your code and my examples work.

I tried the VolumeFreeSpace example and it also works.

Type any character to start

First call to freeClusterCount scans the FAT.

freeClusterCount() call time: 4996441 micros
freeClusters: 976358
freeSpace: 31993.299 MB (MB = 1,000,000 bytes)

[luc-github]

Yes it is what I wrote VolumeFreeSpace works on your Git version but not on https://github.com/tannewt/SdFat fork.
Open small file with your git works
Open big file with your git generate the issue mentioned above
Use Volume free space with your git works

but using https://github.com/tannewt/SdFat which is based on your git before last big changes
Open small file works
Open big file works
Use Volume free space do not works

I did not asked you to dig in the my code I wrote :

Anything I can do to help to debug this issue ?

I was looking for advice to find the problem - seems not possible so I won't bother you anymore - no problem

[greiman]

Why did you expect me to debug the tannewt fork? It is almost two years old.

[luc-github]

No but you did several changes since and you may know which could lead me to search - anyway please do not bother - I can handle it

[luc-github]

I solved my issue by changing ::yield(); by delay(0); in SysCall.h
https://github.com/greiman/SdFat/blob/master/SdFat/src/FatLib/SysCall.h#L54-L58

#if defined(ARDUINO)
inline void SysCall::yield() {
  // Use the external Arduino yield() function.
  //::yield();
  delay(0);
}

now big files can also be read without generating an error and no side effects - issue solved

[greiman]

Strange, must be a bug in the ESP8266 you are using. yield() works for me.

See this:

https://github.com/esp8266/Arduino/blob/master/doc/reference.md

There is also a yield() function which is equivalent to delay(0). The delayMicroseconds function, on the other hand, does not yield to other tasks, so using it for delays more than 20 milliseconds is not recommended.

I use yield() since it also works for schedulers in other systems. Also yield() is a weak function so if you replace it with your own version you can do other scheduler stuff.

[greiman]

I tested bench with this version of SysCall::yield() and it works.

inline void SysCall::yield() {
  // Use the external Arduino yield() function.
  ::yield();
}

If I comment out ::yield(); it crashes so something you are using replaces yield or there is a bug in your version of ESP8266.

I used yield instead of delay(0) since it allow more flexible scheduling.

[luc-github]

I use https://github.com/esp8266/Arduino, 2.3.0 and git version - I did not change anything in core version

[greiman]

That's the version I am using.

You don't need to change anything in the core version to have yield replaced. If some other code/library you are using defines yield, it replaces the core yield. I assume you know what a weak function is.

You may still have a problem. Some SD cards have long latencies so another yield call may be needed in SdFat.

[greiman]

I ran this test and the WDT timeout is about 2100 ms so no SD card should cause a problem.

void setup() {
  Serial.begin(9600);
  uint32_t ms = 0;
  while(1) {
    Serial.println(ms);
    uint32_t m = millis();
    while (millis() < (m + ms)) {}
    delay(500);
    ms += 100;
  }
}
void loop() {}

Output at crash

2000
2100

Soft WDT reset

If you wish, you can set this to less than 10,000 microseconds by editing SdFatConfig.h in the current SdFat.

// If Particle device or ESP8266 call yield.
#define WDT_YIELD_TIME_MICROS 100000

[luc-github]

Yes I know weak function but SdFat is the only library I am using, no other external code - I have no idea where the issue come from - I just shared what issue I experienced, asked for advice, and shared how I solved, that is all.

[luc-github]

Using optimistic_yield(10000); instead of delay(0); works also for me

like set in 2.30 for embedded SD: esp8266/Arduino@9172033

[greiman]

That means something is likely wrong with the weak function linking.

optimistic_yield calls yield.

Here is the code for all the yield functions.

extern "C" void __yield() {
    if (cont_can_yield(&g_cont)) {
        esp_schedule();
        esp_yield();
    }
    else {
        panic();
    }
}

extern "C" void yield(void) __attribute__ ((weak, alias("__yield")));

extern "C" void optimistic_yield(uint32_t interval_us) {
    if (cont_can_yield(&g_cont) &&
        (system_get_time() - g_micros_at_task_start) > interval_us)
    {
        yield();
    }
}

I have seen something like this before with Arduino. I may just call delay(0) for ESP8266. Sad since the reason for a weak yield is so you can implement a custom scheduler.

delay(0) is the same as __yield() which should be the same as yield().

One more thing. I found a version of yield() in some ESP8266 test code that would cause the problem if it were to replace the weak version.

It's in this file.

esp8266\hardware\esp8266\2.3.0\tests\host\common\Arduino.cpp

Here is the function:

extern "C" void yield()
{
}

[greiman]

Everything above is wrong. I just noticed that the correct yield is called but __yield calls panic in your code.

Panic C:\Users\user\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.3.0\cores\esp8266\core_esp8266_main.cpp:98 __yield

It's not a WDT problem.

delay(0) is the same as yield() except it doesn't call panic, it just returns if this is not true.

cont_can_yield(&g_cont)

More terrible code int ESP8266. So it is a bug in ESP8266.

optimistic_yield also just returns.

[luc-github]

wow !!! thanks so actually delay() and optimistic_yield hide the problem but issue is elsewhere

[greiman]

I updated github with this change

#if defined(ESP8266)
inline void SysCall::yield() {
  // Avoid ESP8266 bug
  delay(0);
}
#elif defined(ARDUINO)

[luc-github]

should this reported to ESP8266 github ? as issue is on their side ?

Thanks for you help on this - I learn something today

[greiman]

I think it should be since there are many ported libraries that use yield() in the ESP8266 software.

I develop for more than 10 platforms and stuff like this is killing my time but reporting and following up takes even more time.

Recently I spent two days finding a bug in the Particle Electron SPI driver.

So I wish you would report it.

[luc-github]

I have started to share the problem on gitter, as seems other people do not have the issue which is weird