How to drop hoodie.datasource.write.partitionpath.field fields from a Hudi Dataset?

[brandon-stanley]

Hi Hudi Team,

Is it possible to change the behaviour of Hudi when specifying the hoodie.datasource.write.partitionpath.field configuration for a table? I notice that the data is partitioned as expected. However, the dataset also contains the columns that were specified in the hoodie.datasource.write.partitionpath.field configuration. This behaviour is different from the native spark.write.partitionBy operation, which will partition the data based on specified columns and remove the aforementioned columns from the data set. Is there a way to match this behaviour?

Here is an example of the behaviour I am referring to: https://stackoverflow.com/questions/36164914/prevent-dataframe-partitionby-from-removing-partitioned-columns-from-schema/47104251

Cheers,

Brandon Stanley

[bvaradar]

I have added a faq entry for the reasoning in https://cwiki.apache.org/confluence/display/HUDI/FAQ#FAQ-Whypartitionfieldsarealsostoredinparquetfilesinadditiontothepartitionpath?

[brandon-stanley]

@bvaradar Thanks for the reasoning. Is it possible to turn this feature off?

[bvaradar]

@brandon-stanley : There is no such feature currently available. I have created a jira for this request : https://issues.apache.org/jira/browse/HUDI-1363 for this. If you are interested in taking this up, kindly assign it to yourself

Thanks,
Balaji.V

[krishna-paypay]

I am also facing issues in the HUDI version 0.11.1 strange thing is that is occurs in some of the tables sometime but not on all the Hudi tables I am managing.