multi-level partition

[Yogashri12]

Hi,
I have a dataset which contain many columns.
now i want to store this dataset based on year/month.

hudi_options = {
'hoodie.table.name': tableName,
'hoodie.datasource.write.recordkey.field': 'ID',
'hoodie.datasource.write.table.name': tableName,
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.precombine.field': 'ID',
'hoodie.upsert.shuffle.parallelism': 2,
'hoodie.insert.shuffle.parallelism': 2,
'hoodie.datasource.write.partitionpath.field':'year/month'
}

when i use partition path as year/month,it doesn't create any partition.

i want my partition path as (year/month) year--->month--->then parquet files.

i saw your sample example where you have used (region/country/city)

i am using pyspark to connect with apache Hudi

[bvaradar]

'hoodie.datasource.write.partitionpath.field' should be 'year,month'. Please make sure you use ComplexKeyGenerator

[bvaradar]

Please reopen if this doesnt solve the problem

[Yogashri12]

how to use ComplexKeyGenerator in pyspark.
hudi_options = {
'hoodie.table.name': tableName,
'hoodie.datasource.write.recordkey.field': 'ID',
'hoodie.datasource.write.table.name': tableName,
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.precombine.field': 'ID',
'hoodie.upsert.shuffle.parallelism': 2,
'hoodie.insert.shuffle.parallelism': 2,
'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.ComplexKeyGenerator',
'hoodie.datasource.write.partitionpath.field':'year/month'
}
throws an error
py4j.protocol.Py4JJavaError: An error occurred while calling o48.save.
: java.io.IOException: Could not load key generator class org.apache.hudi.ComplexKeyGenerator

sry i am new to apache hudi,so can you help me out without any hesitation.

thanks in advance.

[bvaradar]

@Yogashri12 :

hoodie.datasource.write.keygenerator.class
should be set to org.apache.hudi.keygen.ComplexKeyGenerator
and not org.apache.hudi.ComplexKeyGenerator

Can you let me know where you found this reference toComplexKeyGenerator. We might need to fix the document accordingly ?

Also, set hoodie.datasource.write.partitionpath.field to "year,month"

[Yogashri12]

thank you,
the partition is working totally fine.

In configuration page,

KEYGENERATOR_CLASS_OPT_KEY
Property: hoodie.datasource.write.keygenerator.class, Default: org.apache.hudi.SimpleKeyGenerator
Key generator class, that implements will extract the key out of incoming Row object

i just modified it into org.apache.hudi.ComplexKeyGenerator

[Yogashri12]

I have another issue in compression technique.

even after including snappy as compression technique my file size is not compressed.

hudi_options = {
'hoodie.table.name': tableName,
'hoodie.datasource.write.recordkey.field': 'ID',
'hoodie.datasource.write.table.name': tableName,
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.precombine.field': 'ID',
'hoodie.upsert.shuffle.parallelism': 2,
'hoodie.insert.shuffle.parallelism': 2,
'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.ComplexKeyGenerator',
'hoodie.datasource.write.partitionpath.field':'year,month',
'hoodie.parquet.compression.codec':'SNAPPY',
}

[bvaradar]

@Yogashri12 : By default Hudi uses gzipped compression which will give better compression when it comes to file size. Your configuration looks right. How do you know that the file is not compressed. Note that you will not see .snappy as suffix even if parquet file was indeed compressed by snappy. You can use "parquet-tools.jar meta" command to check the compression type employed.