[TW#17077] SDCard format leads to speed decrease

[jkearins]

I have SD card using 4-line SD mode (SDMMC peripheral). SD is formatted under Windows 7 with exFAT (cluster 16k). I measured SD Card speed writing 1mb file in circle. The speed depends of buffer size, as expected. Results are:
1kb buffer: 0.3-0.4 Mb/s
2kb buffer: 0.6-0.9 Mb/s
4kb buffer: 0.9-1.9 Mb/s
8kb buffer: 1.0-3.8 Mb/s
16kb buffer: 1.4-4.1 Mb/s

If I format SD using esp32 than I have significant decrease of speed. The speed does not depend of buffer size:
1kb buffer: 0.14-0.16 Mb/s
2kb buffer: 0.14-0.16 Mb/s
4kb buffer: 0.14-0.16 Mb/s
8kb buffer: 0.14-0.16 Mb/s
16kb buffer: 0.14-0.16 Mb/s

FATFS configuration is:

CONFIG_FATFS_CODEPAGE=866
CONFIG_FATFS_LFN_HEAP=y
CONFIG_FATFS_MAX_LFN=255
#define FF_FS_EXFAT		1
#define FF_LFN_UNICODE	2
#define FF_STRF_ENCODE	3

I use the following format function:

static sdmmc_card_t* s_card = NULL;
static uint8_t s_pdrv = 0;
static char * s_base_path = NULL;

esp_err_t MySD::_format(String &sLog) {
    // input parameters
    const char* base_path = (char*)MOUNT_POINT;
    const sdmmc_host_t* host_config = &m_host;
    const void* slot_config = &m_slot_config;
    const esp_vfs_fat_mount_config_t* mount_config = &s_mount_config;

    esp_err_t err = ESP_OK;
    const size_t workbuf_size = FF_MAX_SS; //f_fdisk needs FF_MAX_SS
    void* workbuf = NULL;
    FATFS* fs = NULL;
    sLog = "";
    BYTE pdrv;
    FRESULT res;
    DWORD plist[] = {100, 0, 0, 0};
    char drv[3];

    if (s_card != NULL) {
        sLog = "Previous unmount() is not fully completed";
        return ESP_ERR_INVALID_STATE;
    }

    // connect SDMMC driver to FATFS
    pdrv = 0xFF;
    if (ff_diskio_get_drive(&pdrv) != ESP_OK || pdrv == 0xFF) {
        sLog = "The maximum count of volumes is already mounted";
        return ESP_ERR_NO_MEM;
    }

    s_base_path = strdup(base_path);
    if(!s_base_path){
        sLog = "Could not copy base_path, not enough momory";
        return ESP_ERR_NO_MEM;
    }

    s_card = (sdmmc_card_t*)malloc(sizeof(sdmmc_card_t));
    if (s_card == NULL) {
        sLog = "Could not allocate memory for sdmmc_card_t";
        err = ESP_ERR_NO_MEM;
        goto fin;
    }

    err = (*host_config->init)();
    if (err != ESP_OK) {
        sLog.printf("Host init returned err=0x%x", err);
        goto fin;
    }

    // configure SD slot
    if (host_config->flags == SDMMC_HOST_FLAG_SPI) {
        err = sdspi_host_init_slot(host_config->slot,
                (const sdspi_slot_config_t*) slot_config);
    } else {
        err = sdmmc_host_init_slot(host_config->slot,
                (const sdmmc_slot_config_t*) slot_config);
    }
    if (err != ESP_OK) {
        sLog.printf("Slot init returned err=0x%x", err);
        goto fin;
    }

    // probe and initialize card
    err = sdmmc_card_init(host_config, s_card);
    if (err != ESP_OK) {
        sLog.printf("Card init returned err=0x%x", err);
        goto fin;
    }

    ff_diskio_register_sdmmc(pdrv, s_card);
    s_pdrv = pdrv;
    ESP_LOGD(TAG, "%s() using pdrv=%i", __func__, pdrv);
    //char drv[3] = {(char)('0' + pdrv), ':', 0};
    drv[0] = (char)('0' + pdrv);
    drv[1] = ':';
    drv[2] = '\0';

    // connect FATFS to VFS
    err = esp_vfs_fat_register(base_path, drv, mount_config->max_files, &fs);
    if (err == ESP_ERR_INVALID_STATE) {
        // it's okay, already registered with VFS
    } else if (err != ESP_OK) {
        sLog.printf("FAT register returned err=0x%x", err);
        goto fin;
    }

    ESP_LOGW(TAG, "%s() partitioning card...", __func__);
    workbuf = malloc(workbuf_size);
    ESP_LOGW(TAG, "%s() workbuf_size=%d workbuf=0x%x", __func__, workbuf_size, (unsigned int)workbuf);
    if (workbuf == NULL) {
        sLog = "Could not allocate memory for workbuf";
        err = ESP_ERR_NO_MEM;
        goto fin;
    }

    ESP_LOGW(TAG, "%s() fdisking card...", __func__);
    res = f_fdisk(s_pdrv, plist, workbuf);
    if (res != FR_OK) {
        sLog.printf("f_fdisk returned err=0x%x", res);
        err = ESP_FAIL;
        goto fin;
    }
    ESP_LOGW(TAG, "%s() formatting card...", __func__);
    res = f_mkfs(drv, FM_ANY, s_card->csd.sector_size, workbuf, workbuf_size);
    ESP_LOGW(TAG, "%s() formatting done", __func__);
    if (res != FR_OK) {
        sLog.printf("f_mkfs returned err=0x%x", res);
        err = ESP_FAIL;
        goto fin;
    }
    free(workbuf);
    workbuf = NULL;

    ESP_LOGW(TAG, "%s() mounting again", __func__);
    res = f_mount(fs, drv, 0);
    if (res != FR_OK) {
        sLog.printf("f_mount after format returned err=0x%x", res);
        err = ESP_FAIL;
        goto fin;
    }
    //return ESP_OK;

fin:
    if (sLog.length() > 0) {
      ESP_LOGE(TAG, "%s() %s", __func__, sLog.c_str());
    }
    sdmmc_host_deinit();
    free(workbuf);
    if (fs) {
        f_mount(NULL, drv, 0);
    }
    esp_vfs_fat_unregister_path(base_path);
    ff_diskio_unregister(pdrv);
    free(s_card);
    s_card = NULL;
    return err;
}

FatFS is unmounted before calling format function. Just after format, Windows 7 can not read SD card. When I insert an adapter with SD card it is blinking endless, drive letter is appeared but system hangs on trying to open it. If esp32 formats SD card with FAT, the situation is the same. I used 5 different SD cards (2gb, 8gb, 32gb). To recover SD card I insert it into phone with android, format with FAT, than format with exFAT under Windows.

[igrr]

That probably has to do with the allocation unit size set to sector size in

res = f_mkfs(drv, FM_ANY, s_card->csd.sector_size, workbuf, workbuf_size);

According to the FATFS manual, "The value must be power of 2 and between the sector size and 128 * sector size".

Setting the allocation unit equal to 64kB will result in the best performance.

[jkearins]

1. You are right, the third parameter in
   res = f_mkfs(drv, FM_ANY, s_card->csd.sector_size, workbuf, workbuf_size);
   not a sector size, but a cluster size. After setting allocation unit to 64k and forcing exFAT by FM_EXFAT I got expected speed:

const DWORD allocation_unit = 65536;
res = f_mkfs(drv, FM_EXFAT | FM_SFD, allocation_unit, workbuf, workbuf_size);

2. According to the source file vfs_fat_sdmmc.c, if mount fails and flag format_if_mount_failed is set, than format will be called as:
   res = f_mkfs(drv, FM_ANY, s_card->csd.sector_size, workbuf, workbuf_size);
   and we see that allocation unit will be 512 bytes. Maybe it should be like this?:
   res = f_mkfs(drv, FM_ANY | FM_SFD, workbuf_size, workbuf, workbuf_size);
   just like in vfs_fat_spiflash.c, at least allocation unit will be 4096 bytes. By the way, file system in this case will be FAT32, not exFAT (type is calculated according to overall size of sd card, for very large cards it can be exFAT).

[igrr]

It does make sense to make the allocation unit size configurable through esp_vfs_fat_mount_config_t, agree on that. FM_SFD flag should not be used in case of an SD card though, as it indicates that the media should not be partitioned, with the FAT volume starting from sector 0. This is something that makes sense for FATFS in flash (it is never read by other devices), but for an SD card this may cause compatibility issues.

[jkearins]

Thanks for the hint on FM_SFD. I removed this flag.