Constraints on SD card access: no free file descriptors

[eliosmoreira]

Platform

ESP32

IDE / Tooling

PlatformIO

What happened?

Hi, I am using Arduino-audio-tools with this library but in different RTOS tasks.
I made a custom class networkManager which simplifies the user experience in the main setup/loop functions for this library. I made a callback function _jsProcessorCallback so whenever the user needs to process a javascript file variables, he can just attach his own templateProcessor.

The error pops when an audio file is already being played (not in the same directory) and the AWS has a request for its index: simple html, css and js files (Using minified versions from 450 lines of code in total).
In the user template processor callback, the variable "%audio_files%" triggers a process of accessing SD and retrieving a list of files in a directory. (~25 files in total).
I cannot find a clear answer but i believe this has something to do with AWS internal queuing. If i use AWS method for accessing SD directly, it probably takes too long to close the files so i run out of file descriptors and so i cannot get the list.

User callback:

String jsProcessor(const String &var) {
    if (var.equals("audio_files")){
        String path = "/music/background";
        File dir = SD.open(path);
        if (!dir || !dir.isDirectory()) {
            Serial.println("Failed to open directory");
            dir.close();
            return String();
        }

        String fileList;
        File file = dir.openNextFile();

        while (file) {
            if (!file.isDirectory()) {
                LOG.debug(file.name());
                fileList += "\"" + String(file.name()) + "\",";
            }
            file.close();
            file = dir.openNextFile();
        }
        dir.close();
        LOG.debug(fileList.c_str());
        return fileList;
    }
}

The buggy way (works if audio player is not being used):

// ---- part of onNotFound callback ----
// Send processed JS
if(_jsProcessorCallback && mime == "text/javascript") {
    request->send(SD, filePath, String(), false,
            [this](const String& var) -> String { return this->_jsProcessorCallback(var); });
    return;
}

// Send non-processed
request->send(SD, filePath, mime);
return;

But if i get the file, store the content into a String and send it, the error disappears (always works):

// ---- part of onNotFound callback ----
// Get file content
String content;
File file = SD.open(filePath);

if(file && !file.isDirectory()){
    size_t size = file.size();
    char* buffer = (char*)malloc(size + 1);
    file.readBytes(buffer, size);
    buffer[size] = '\0';
    content = String(buffer);
    free(buffer);
}
file.close();

// Send processed JS
if(_jsProcessorCallback && mime == "text/javascript") {
    request->send(200, mime, content,
        [this](const String& var) -> String {return this->_jsProcessorCallback(var);});
    return;
}

// Send non-processed
request->send(200, mime, content);
return;

Stack Trace

E (9795) vfs_fat: open: no free file descriptors
[  9819][E][vfs_api.cpp:301] VFSFileImpl(): fopen(/sd/music/background/B25.wav) failed

Minimal Reproductible Example (MRE)

Cannot get an MRE at the moment

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of ESPAsyncWebServer (and AsyncTCP for ESP32).

[eliosmoreira]

In the meantime, incrementing file descriptors in SD.begin() mitigates the error but still, why so many files opened at the same time? What are the underlying processes that make this happen and is there any way of reducing its use?
And, maybe this should be a wiki topic? 🤔