NIFI-3594 Encrypted provenance repository implementation #1686
…eration. Added BC dependency to nifi-persistent-provenance-repository module.
…der w/ 2 impls, Encryptor skeleton, and exceptions/utilities). Reorganized tests to proper path.
… Pausing to re-evaluate because work may need to be done at lower level (EventWriter/EventReader -- byte/Object serialization).
…t intercepting SchemaRecordReader/Writer serialization (no updates to schema necessary).
…ity-utils to include in nifi-data-provenance-utils.
Resolved RAT and checkstyle issues. All tests pass.
…nhanced error checking and guard controls).
…ption metadata serialization.
…ved cached ciphers to allow non-repeating IVs). Added unit tests.
Added validity checks for algorithm and version in AESProvenanceEventEncryptor. Added unit tests.
Refactored encryptor composition. Added unit tests.
…nce repository. Added utility methods for validation. Added unit tests.
…cryption. Added nifi.provenance.repository.encryption.key to default sensitive keys and updated unit tests and test resources. Added method to correctly calculate protected percentage of sensitive keys (unpopulated keys are no longer counted against protection %).
Moved getBestEventIdentifier() from StandardProvenanceEventRecord to ProvenanceEventRecord interface and added delegate in all implementations to avoid ClassCastException from multiple classloaders. Initialized IV before cipher to suppress unnecessary warnings. Added utility method to read encrypted provenance keys from key provider file. Suppressed logging of event record details in LuceneEventIndex. Added logic to create EncryptedSchemaRecordReader (if supported) in RecordReaders. Cleaned up EncryptedSchemaRecordReader and EncryptedSchemaRecordWriter. Added keyProvider, recordReaderFactory, and recordWriterFactory initialization to EncryptedWriteAheadProvenanceRepository to provide complete interceptor implementation. Added logic to RepositoryConfiguration to load encryption-related properties if necessary. Refactored WriteAheadProvenanceRepository to allow subclass implementation. Registered EncryptedWAPR in ProvenanceRepository implementations. Added unit tests for EWAPR. Added new nifi.properties keys for encrypted provenance repository.
@alopresto one thought regarding the StaticKeyProvider... rather than having a .key= perhaps it would make sense to instead use a .key.<key_id>= property scheme. This would allow multiple Key ID's to be used, which would resolve the 'Potential Issue' listed above, of now allowing the key to rotate, with rather minimal changes to the code??
…ation of arrays. Added unit test demonstrating performance improvement.
…ency with Mark Payne's feedback. Cleaned up commented code.
Refactored StaticKeyProvider and FileBasedKeyProvider to reduce duplicate code. Added helper methods in NiFiProperties to read multiple key definitions for StaticKeyProvider. Fixed undetected NPE in tests (storing null value into properties). Added unit tests.
I had to make some substantial changes to enable multiple keys for
This should generate a key map of
@alopresto @YolandaMDavis just a quick update on my review... I think the code looks good and am a +1 on that front. I had no problem getting this up & running and things seem to work well. I just wanted to verify some corner cases like running out of disk space, kill -9 against NiFI, etc. and ensure that NiFi is still able to restart, as this is an issue that some have hit in the past. Otherwise all looks good to me. Thanks!
@alopresto Ran a test where I had an existing unencrypted provenance repo and switched to the encrypted provider. The issues you've documented did occur (where switching between configuration would lead to errors in the log and when querying). However I'm wondering two things:
If clearing the provenance repo when changing configurations would resolve the issue can that step be included in the documentation (if needed I would recommend including that step in the issues section)?
@YolandaMDavis Thanks Yolanda.
During most of my testing, I did clear the repositories between application starts when switching the implementation (
I will re-evaluate those scenarios as soon as I finish reviewing PR 1712 for Bryan.
I do think suppressing the stacktrace and providing a more descriptive error is a good idea and will tackle that as well. If it is determined there is a simple reason your queries were not working, great. If not, adding documentation instructing repository removal may be necessary.
I can reproduce your described issue. I believe the difference is that when I tested and did not clear the existing provenance repository, I was switching between
…en switching to encrypted implementation (see PR 1686 @ apache#1686 (comment)).
…o Admin Guide and User Guide. Added screenshot of encrypted provenance repository contents on disk. Added note about clearing existing provenance repository when switching to encrypted implementation (see PR 1686 @ #1686 (comment)). This closes #1713. Signed-off-by: Andy LoPresto <firstname.lastname@example.org>
@alopresto ran tests with the following corner cases:
Also validated that switch between EWAPR and PPR (and vice versa) along with removal of content, flowfile and provenance repos would prevent classcastexception seen previously.
will merge shortly
I rebased and resolved the conflict. There were test errors in