[SUPPORT] General question on Hudi about GDPR, Glue

[reena-cs]

We are assessing Apache hudi for GDPR compliance purpose. In the process , i have a bunch of question. Please have a look at them and help me understand .

1. How does hudi deal with schema evolution:
   a. during type change
   b. during addition of new fields.
   c. during absence of field .
2. Currently Athena supports Apache Hudi, is it same for AWS Glue?? or is there a hack to do it?
3. To be GDPR compliant i would like to keep my file versions at 1, with this configuration will i be able to do a roll back (to the last commit)?
4. I tried doing a incremental pull and i was able to get the last set of records that got inserted or updated, is there a way to get the records which got deleted?

[bvaradar]

cc @n3nash who is rethinking schema management in Hudi.

1. Regarding schema evolution, Hudi uses avro schema compatibility rules. For example, new fields must be nullable, columns cannot be deleted...
2. Error running hudi on aws glue #1977 should give context around AWS Glue support (cc @umehrot2 )
3. Yeah, I am assuming you are referring to cleaner retentions. Commits happen before cleaning. Any failed commits will not cause any side-effects.
4. For Incremental pull to work, you need to employ soft-deletes only with tombstones general pattern. You can retain hudi key but null out other columns (nullable) or obfuscate.

[vinothchandar]

Good to see if we can update our FAQ with these questions? :)

[bvaradar]

Good point. Added FAQ sections for these questions in https://cwiki.apache.org/confluence/display/HUDI/FAQ

[reena-cs]

Thanks a lot @bvaradar and @vinothchandar for the taking your time to answer my questions, it helped a lot . Closing this issue.