Reduce number of stat calls in storage File

[CurtizJ]

Changelog category (leave one):

• Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Improve performance of reading from storage File and table functions file in case when path has globs and matched directory contains large number of files.

[CurtizJ]

In directory with about 25000 files:
Query: clickhouse benchmark <<< "desc file('sample/*.json', JSONAsString)" --cumulative

this PR:

localhost:9000, queries 127, QPS: 11.446, RPS: 11.446, MiB/s: 0.001, result RPS: 11.446, result MiB/s: 0.001.

0.000%          0.086 sec.
10.000%         0.086 sec.
20.000%         0.086 sec.
30.000%         0.086 sec.
40.000%         0.087 sec.
50.000%         0.087 sec.
60.000%         0.087 sec.
70.000%         0.087 sec.
80.000%         0.088 sec.
90.000%         0.089 sec.
95.000%         0.090 sec.
99.000%         0.092 sec.
99.900%         0.113 sec.
99.990%         0.113 sec.

master:

localhost:9000, queries 125, QPS: 5.699, RPS: 5.699, MiB/s: 0.000, result RPS: 5.699, result MiB/s: 0.000.

0.000%          0.173 sec.
10.000%         0.173 sec.
20.000%         0.174 sec.
30.000%         0.174 sec.
40.000%         0.174 sec.
50.000%         0.174 sec.
60.000%         0.175 sec.
70.000%         0.175 sec.
80.000%         0.176 sec.
90.000%         0.179 sec.
95.000%         0.183 sec.
99.000%         0.191 sec.
99.900%         0.204 sec.
99.990%         0.204 sec.

[alexey-milovidov]

Thank you!

Further directions:

• the function for glob matching is strange; it will be better to generate paths manually (by mini-interpreter) instead of converting globs to regexp;
• maybe then we can easily implement Support ** glob for recursive directory traversal. #36316

[alexey-milovidov]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated

[alexey-milovidov]

Stress test (debug, actions) — Found signal in gdb.log

See this: #36660