Is it possible to resolve the schema, but not expand the paths when doing a collect_schema?

[kszlim]

Description

Path expansion for large datasets is very expensive, but I imagine schema resolution itself might be very cheap (relative to path expansion). Is it possible for schema resolution to avoid path expansion? It seems to take pretty long for me on a 10k file dataset, (i'm guessing it's due to listing all the paths).

[stinodego]

Ideally we shouldn't do path expansion for collect_schema if the reader is provided a schema, e.g. through a schema argument. But I'm not sure where we're at with the implementation. Tagging @nameexhaustion for this one.

[kszlim]

It would be interesting if you could get the schema from a single file in the inference case and then resolve the path expansion at collection time.