Support directories with millions of files.

[davies]

What would you like to be added:

Currently, we fetch the attributes of files in single directory with single batch request to Redis, that could be slow or fail, and block other requests.

We can split those into small batches, for example, 1000 per batch.

Why is this needed:

The number of files could be millions, we don't want people be bited by that.

Backlog

• Call MGET with small batches Avoid calling mget with massive number of keys in Readdir #110
• Use HSCAN instead of HGETALL List large directories as small batches #128

[suzaku]

Any more detail for this issue? In what kind of operation are the file attributes fetched?

[davies]

When you do ls, a Readdir() will be called. When plus is true, it will fetch all the attributes in one batch.

[suzaku]

Got it, I can take a try on this one.

[suzaku]

If there are millions of files, we might run into trouble when the first HGetAll call is reached. Any idea how to work around it?

[davies]

Yes. Right now, we don't have a work around, that could be the next challenge.

[davies]

The second part is done by #128