[FATFS] calling stat() on every file in a folder is too slow (IDFGH-8788)

[chipweinberger]

Is your feature request related to a problem?

Goal: get all paths in a folder that are directories (and ideally their filesize too)

stat() is a very slow function. It takes 10+ seconds to call stat() on 300+ paths in the same directory.

But! it takes less than a second to call readdir on every path in the same directory.

This is unfortunate because in the code , vfs_fat_readdir_r already has all the stat() information and just throws it away!

Is there anything I can do to speed this up?

Describe the solution you'd like.

Not sure.

Perhaps I can call f_readdir myself directly? Some guidance here would be helpful.

FRESULT f_readdir (
FF_DIR* dp, /* Pointer to the open directory object /
FILINFO fno /* Pointer to file information to return */
)

[chipweinberger]

So I may be pretty dumb! It looks like dirent has a entry->d_type = DT_DIR field that tells me exactly the information I need!

But, if I want to get filesize quickly as well, I would need to do something like this (reimplement readdir):

edit: Just btw, to anyone that finds this thread, this code does work! it is so much faster!

typedef struct {
    DIR dir;
    long offset;
    FF_DIR ffdir;
    FILINFO filinfo;
    struct dirent cur_dirent;
} vfs_fat_dir_t;

// returns the FILEINFO as well!
int better_vfs_fat_readdir_r(void* ctx, DIR* pdir,
        struct dirent* entry, struct dirent** out_dirent, FILEINFO* out_fileinfo)
{
    assert(pdir);
    vfs_fat_dir_t* fat_dir = (vfs_fat_dir_t*) pdir;
    FRESULT res = f_readdir(&fat_dir->ffdir, &fat_dir->filinfo);
    if (res != FR_OK) {
        *out_dirent = NULL;
        ESP_LOGD(TAG, "%s: fresult=%d", __func__, res);
        return fresult_to_errno(res);
    }

    // copy fileinfo to output
    *out_fileinfo = fat_dir->filinfo;

    if (fat_dir->filinfo.fname[0] == 0) {
        // end of directory
        *out_dirent = NULL;
        return 0;
    }
    entry->d_ino = 0;
    if (fat_dir->filinfo.fattrib & AM_DIR) {
        entry->d_type = DT_DIR;
    } else {
        entry->d_type = DT_REG;
    }
    strlcpy(entry->d_name, fat_dir->filinfo.fname,
            sizeof(entry->d_name));
    fat_dir->offset++;
    *out_dirent = entry;
    return 0;
}

// returns the FILEINFO as well!
static struct dirent* better_vfs_fat_readdir(void* ctx, DIR* pdir, FILEINFO* out_fileinfo)
{
    vfs_fat_dir_t* fat_dir = (vfs_fat_dir_t*) pdir;
    struct dirent* out_dirent;
    int err = better_vfs_fat_readdir_r(ctx, pdir, &fat_dir->cur_dirent, &out_dirent, out_fileinfo);
    if (err != 0) {
        errno = err;
        return NULL;
    }
    return out_dirent;
}


main() {
  DIR *dir;
  struct dirent *entry;
  if (dir = opendir("⁄")) { 
    FILEINFO info;
    while ((entry = better_vfs_fat_readdir(NULL, dir, &info)) != NULL) {
      printf("  %u\n", info-> fsize);
    }
    closedir(dir);
  }
}

[igrr]

While it is unlikely we would add a non-standard API like better_vfs_fat_readdir, we can do an optimization for this specific case. readdir can cache the FILINFO structure for the current file and stat can peek into that cached structure. This will speed up the operation in the common case when you are calling stat for every directory entry, while iterating over the directory.

If implemented, this can close #9570 which I think was asking for a similar feature.

[chipweinberger]

That's a clever solution.

Yes, that would be a great idea.

[chipweinberger]

When / if you implement caching, please consider adding it to the v4.4 branch as well. It'll be a massive improvement.

[svofski]

In my project it can get up to 3-4 seconds per file just to query their size. This can't be right.

Thanks @chipweinberger for your workaround!

[chipweinberger]

wow very impressive work @RathiSonika