Lightning: Support Custom File Iteration Algorithm when Constructing MDLoader

[dsdashun]

Enhancement

The MDLoader is the abstraction that stores the result of scanning the data source with file route rules and filters. However, in some situations, we don't need to iterate ALL the data source files to construct a complete MDLoader object. Instead, we just need to pick a few of key files in the data source to construct the information we want in MDLoader, so that the performance for generating an MDLoader can be boosted.

Currently, the file iteration logic cannot be customized, all we have is using the ExternalStorage interface, which provides a WalkDir method to iterate ALL the files in the data source. We want to make the iteration logic customizable.

[dsdashun]

/component lightning