Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poor SD read speed of fread() (IDFGH-903) #3249

Closed
keelung-yang opened this issue Apr 3, 2019 · 6 comments
Closed

Poor SD read speed of fread() (IDFGH-903) #3249

keelung-yang opened this issue Apr 3, 2019 · 6 comments

Comments

@keelung-yang
Copy link

@keelung-yang keelung-yang commented Apr 3, 2019

I'm using the last master branch since fopen() failed in v3.3 while opening large file which more than 7MB.

Test results on Kingston 32GB 90MB/s TF(Micro SD) Class10 UHS-I:

esp_err_t sd_connect()
{
    // On esp-idf v4.0-dev-225-g2f8b6cfc7
    //      SPI SDMMC_FREQ_DEFAULT   :   585 KB/s
    //      SPI SDMMC_FREQ_HIGHSPEED :   594 KB/s
    //      MMC(1-line) SDMMC_FREQ_DEFAULT   :   673 KB/s
    //      MMC(1-line) SDMMC_FREQ_HIGHSPEED :   775 KB/s
#if 0
    ESP_LOGI(LOG_TAG, "Using SPI peripheral");
    sdmmc_host_t host = SDSPI_HOST_DEFAULT();
    sdspi_slot_config_t slot_config = SDSPI_SLOT_CONFIG_DEFAULT();
#else
    ESP_LOGI(LOG_TAG, "Using SDMMC peripheral");
    sdmmc_host_t host = SDMMC_HOST_DEFAULT();
    sdmmc_slot_config_t slot_config = SDMMC_SLOT_CONFIG_DEFAULT();
    slot_config.width = 1;
#endif
    // host.max_freq_khz = SDMMC_FREQ_HIGHSPEED;
    // ...
}

And my test code:

int64_t sd_read_file(const char * path, uint8_t * buf, long int pos, size_t size)
{
    FILE * file = NULL;
    int ret = sd_cached_file(path, &file);  // To avoid call fopen() frequently
    if (ret != 0)
        return ret;
    
    ret = fseek(file, pos, SEEK_SET);
    if (ret != 0)
        return ret > 0 ? -ret : ret;
    else
        return fread(buf, 1, size, file);
}

double sd_test_read(const char * path, size_t blocksize)
{
    uint8_t * buf = (uint8_t *)malloc(blocksize);
    if (buf == NULL)
        return -1.0;
    
    int64_t bytes = 0;
    int64_t from = esp_timer_get_time();
    for (int64_t i = 0; ; i += blocksize)
    {
        int64_t ret = sd_read_file(path, buf, i, blocksize);
        if (ret > 0)
            bytes += ret;
        else
            break;
    }
    int64_t to = esp_timer_get_time();

    free(buf);
    
    return 1000000.0 * bytes / (to - from);
}

double speed = sd_test_read("/sdcard/37m.zip", 64*1024);
printf("---------------SD SPEED: %.2f KB/s\n", speed/1000);
@github-actions github-actions bot changed the title Poor SD read speed of fread() Poor SD read speed of fread() (IDFGH-903) Apr 3, 2019
@igrr
Copy link
Member

@igrr igrr commented Apr 5, 2019

Hi @keelung-yang, what read speed do you need to achieve?
If you use SD interface in 4-line mode, instead of SPI, you can get read speed around 3MB/s, from a FAT partition, and around 12MB/s if you read from card as a raw block device.

@keelung-yang
Copy link
Author

@keelung-yang keelung-yang commented Apr 6, 2019

Hi @igrr , > 50KB/s is enough to feed our 500kbps CAN bus application in theory. Since I've built a task which read data from SD card to a Ring Buffer, and then our CAN application fetch data from the Ring Buffer.

But, I found this while struggling with read spead: https://esp32.com/viewtopic.php?t=1942

esp_vfs_read() is being called to read 128 bytes at a time

I can write into SD card with the speed of ~6MB using fwrite but the reading speed from the SD card (fread) is aroung 1MB!!

It's not appropriate.
The chunk size 128 bytes should be configured by sdkconfig, but it's not.

@igrr
Copy link
Member

@igrr igrr commented Apr 8, 2019

Hi @keelung-yang, the default file buffer size is a compile-time constant of the C library (newlib). Since we are not building newlib as part of IDF build process, it is not easily possible to modify it using sdkconfig.

Currently you can do one of the following things to improve the read speed:

  1. Increase the buffer size used by C library for the given FILE, after opening the file. This can be done using C function setvbuf. The C library will then perform reads from the VFS in chunks equal to the new buffer size.

  2. Bypass the C library and its buffering layer, and read directly from the file descriptor using read(fileno(file), buf, buf_size) (where file is your FILE* stream).

We will also look into enabling the unbuffered stream read optimization in the next release of the toolchain. This will not change the behavior in the default (buffered) case, but will allow the fread function to read directly into user-provided buffer (without chunking) if buffering has been disabled with setvbuf.

@ruffle-b
Copy link

@ruffle-b ruffle-b commented Apr 8, 2019

FWIW, if anyone else trips across this, I've just been doing some testing (SPI 1 bit on a Wemos D32-Pro) and with the standard buffering it was taking some 60 seconds to read a 50MB file.

With a 16k buffer, that's cut down to ~29 seconds. I tried 32 and 64k but that didn't improve it any more and an 8k buffer was slower.

Still not brilliant but twice as good as before.

HTH

@InfiniteYuan
Copy link
Contributor

@InfiniteYuan InfiniteYuan commented May 17, 2019

@igrr What is the highest rate if use SPI mode?

@tom-borcin
Copy link
Collaborator

@tom-borcin tom-borcin commented Feb 25, 2021

Hello @keelung-yang,

thank you for reporting this issue. At Espressif we really appreciate it when our users report issues as it helps us make our products better. Unfortunately there hasn't been any progress with this issue for the last 2 years and it looks like it's been explained and there are workarounds proposed. Because of that we are closing it to keep our backlog manageable.

If you don't agree with this decision, please feel free to reopen the issue or contact me directly.

Regards,

Tomas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants