Dask Orc Files - Mismatched Schema Feature Request

[LongBeach805]

It would be nice to be able to select multiple orc files at once that contain different schema. For instance, if the user has three files with structs:

1. struct<id, name, city, state>
2. struct<id, city, state>
3. struct<id, name, state>

And then used the following code:

data = dd.read_orc(path, column=['id', 'name', 'state'])

Dask should be able to pull from the files, and replace any values with nulls if they don't exist in the data set instead of throwing ValueError: Incompatible schemas while parsing ORC files.

[mrocklin]

cc'ing @rjzamora for triage

From my perspective something like this sounds nice, but I suspect that it also introduces some complexity. I would not be surprised to see some resistance from people who think about this kind of thing, but in general I think that @rjzamora or others who are more familiar with this space will likely have better informed thoughts.

[LongBeach805]

My hack is to basically dd.concat([d1, d2, d3, ...]) after processing each dataframe into a similar structure. Thank you for taking a look into this.