DRILL-7055: Revise SELECT * to exclude partitions #1675
Conversation
Historically, a SELECT * (wildcard) query on a partitioned table included partition directory names as a set of "dir0", "dir1" columns. When used with files at differnt depths, this can lead to schema change exceptions as some readers create, say, "dir0" and "dir1", while others create just "dir0". The result is that either 1) things just work, 2) the client gets some batches with two partition columns, others with one, or 3) a hard schema change occurs as the project operator creates missing columns as nullable int. This change proposes to include table columns with using the wildcard and to no longer include partition columns. Partition columns will now work the way the "implicit" file columns already work, so this change improves consistency. The partition columns are still availble: they can be requested explicitly: SELECT *, dir0, dir1 FROM ... Both before and after this change, when including the partition columns explicitly, the nullable int issue described above will occur. However, this change positions us for the revised scan framework that will properly provide the partition columns as nullable VARCHAR whether a matching directory exists or not. This is a potentially breaking change: any user that uses SELECT * and expects partition columns (and manages to work around the schema change issues) will see different behavior: they will have to revise queries to include partition columns.
|
@paul-rogers, agree this a breaking change which might be considered for Drill 2.0? Or we can control it by system / session option which will retails old behavior by default and in Drill 2.0 we'll change option to default value? |
|
@arina-ielchiieva, if we want to do this for 2.0 then let's keep the current behavior. I'll need to do more fixes to the scan framework, but I'll proceed with that work. Far too difficult to add another option and have to test both paths; doing so would double the effort. My guess is that the original idea was that the partition directories have data and should be considered columns. For example, if data is partitioned by year and month, then those might want to be columns. This is particularly important if the partition columns do not appear within the data itself. Using "dir0" and "dir1" has always been a hack: they are meaningless names and require that the user map from "year" to the partitioning structure and know that, in one table "dir0" means "year", while in another it might mean "store". Moving forward, it would be better to use a Hive-like solution: a mapping from partition directories to columns so that the user sees "year" and "month" as column names, not "dir0" and "dir1". Sounds like a job for the new metadata system. If we want a generic directory solution, then allowing a "dir" array would be handy: the array can allow any number of entries, avoiding the schema change issues inherent with the current design. Agree, all these are more than we can do short term, so let's mothball this change for the Drill 1.x series. |
|
We decided to leave the current behavior for now. |
2 participants
Historically, a SELECT * (wildcard) query on a partitioned table included partition directory names as a set of "dir0", "dir1" columns. When used with files at differnt depths, this can lead to schema change exceptions as some readers create, say, "dir0" and "dir1", while others create just "dir0".
The result is that either 1) things just work, 2) the client gets some batches with two partition columns, others with one, or 3) a hard schema change occurs as the project operator creates missing columns as nullable int.
This change proposes to include table columns with using the wildcard and to no longer include partition columns. Partition columns will now work the way the "implicit" file columns already work, so this change improves consistency.
The partition columns are still available: they can be requested explicitly:
Both before and after this change, when including the partition columns explicitly, the nullable int issue described above will occur. However, this change positions us for the revised scan framework that will properly provide the partition columns as nullable VARCHAR whether a matching directory exists or not.
This is a potentially breaking change: any user that uses SELECT * and expects partition columns (and manages to work around the schema change issues) will see different behavior: they will have to revise queries to include partition columns.