Reduce the number of requests in dir_size

[robertbindar]

Extend ls_with_sizes to get is_dir/is_file metadata info within the same request and use the extra info to get rid of the
recursive vfs::is_file and vfs::file_size requests.

Note for reviewers:

• We can further optimize this function if we get rid of the initial is_dir sanity check.
• If we make the assumption that dir_size(file) = file_size(file), which is similar to what du does on unix, we can get rid of the extra request which we pay for all dir_size calls.
• This will result in a slight change of dir_size APIs behavior, so it's not obvious to me if it's worth chasing this or not.

---

TYPE: IMPROVEMENT
DESC: Reduce the number of requests in dir_size

[shortcut-integration]

This pull request has been linked to Shortcut Story #18944: Size for arrays is often N/A.

[KiterLuc]

Add the new parameter to the doc.

[anastasop]

Tested this with the arrays mentioned in the SC card. For almost all of them there is a speedup of 3

[robertbindar]

Spyros is reporting good numbers, but still, it's minutes for a dir_size op. The S3 v1 client can get the dir size in 90sec for an array, whilst VFS gets the same size in 9 mins (my implementation here), so I guess there is still room for improvement.

I've researched this a bit for S3 (with a bit of luck we might be able to generalize it to other backends) and it seems it's possible to get the dir_size with a single ls_with_sizes(prefix, delimiter="") request, which should give us all the objects under a prefix, no squashing based on \. @Shelnutt2 is this somehing worth pursuing?