ORC-1200: Extracting encryption setup logic from WriterImpl
#1156
Conversation
|
Thank you for making a PR, @liujiawinds . |
There was a problem hiding this comment.
It's unclear to me why this is requested by Apache Flink community. For example,
- Do you have some documentation or reference in Apache Flink community?
- Is this Apache ORC design issue?
- Is this request aligned with other encryption handling in Apache Flink community (like Apache Parquet Modular Encryption handling)?
In general, branch-1.8 is already in feature freeze mode. I don't think this is required at v1.8.0 (currently).
Also, cc @omalley because he is the main author of Encryption Feature.
Yes, first we must create a stream based
Yes, I think it is.
Parquet has a similar class named
It's ok to release in later versions. I will extract this part of code in flink. |
|
Thank you for the details. |
|
What should I do next for this pr? I don't quite understand the meaning of it being marked as |
|
Instead of commenting, you are supposed to revise your PR description with your previous comment's content because only the PR title and description becomes a commit message.
|
|
After rethinking about this PR, I removed the milestone. |
|
@dongjoon-hyun The PR description has been revised. |
What changes were proposed in this pull request?
Extracting the encryption setup logic as a tool class.
Why are the changes needed?
Because of flink's ORC writer is based of stream, we must create a stream based PhysicalFsWriter before WriterImpl initial. WriterImpl
Then, You can look into flink orc writer implementation from OrcBulkWriterFactory.
Given the above, now I want to pass encryption settings to PhysicalFsWriter before WriterImpl initail, but the encryption setup at WriterImpl is so tightly coupled that the encryption variant cannot be obtained externally.
How was this patch tested?
It doesn't introduce new features and passed all test cases.