Skip to content

Add 'pseudonymize' pipeline operator #2719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 70 commits into from
Dec 10, 2022
Merged

Add 'pseudonymize' pipeline operator #2719

merged 70 commits into from
Dec 10, 2022

Conversation

Dakostu
Copy link

@Dakostu Dakostu commented Nov 18, 2022

Add a pseudonymize operator that pseudonymizes IP addresses in records.

Daniel Kostuj added 30 commits November 18, 2022 16:14
Add declaration for address::anonymize()
7020d35
Add the first unit tests for IP address anonymization
347eeb5
Implement an anonymization algorithm in address::anonymize()
e512bdb
Delegate address encryption to a new helper class AddressEncryptor
14b8d99
Extend IPv4 anonymization test
46c88d4
Add some IPv6 anonymization test data
eed264c
Add more IP anonymization tests with different key
d990e49
Add even more IP anonymization tests with different key
7eea2ae
Fix memory leak and some weirdness in AddressEncryptor
4209dec
Add documentation to address::anonymize()
6c72357
Add the skeleton for anonymization pipeline operator
b959d01
Move templated values() definition from arrow_table_slice.hpp to arro…
4065079 
…w_table_slice.cpp
Add a configuration for anonymization operator
83c49d5
Make address::anonymization_key_size size_t
9e23c78
Implement parsing of the anon plugin 'key' option
63515ae
Only allow hexadecimal input for anon plugin 'key' key
94c7f4b
Make anonymization operator key parsing more robust
e6eb7a9
Add basic implementation of anonymization operator add()
ad5dde5
Rename 'anonymize operator' test to 'hash operator'
abfaf88
Allow a more graceful handling of non-address columns in anonymize op…
e549e34 
…erator
Add anonyimzation pipeline operator unit tests
ce4bb4f
Remove superfluous lines and comments in anonymize.cpp
9139ae3
Add unit test for invalid anonymization key
1f462dd
Change anonymization syntax to pseudonymization syntax
4accab7
Change pseudonymization key syntax to seed syntax
0510c1e
Add pseudonymization operator to documentation
778f5f5
Insert pseudonymization to pipeline integration tests
f6fabef
Improve syntax a little bit
51fcf41
Turn pseudonymize() from a data-modifying method to a static function…
130495c 
… that returns a copy
Rename pseudonymization_seed_size to pseudonymization_seed_array_size
08e703f
Daniel Kostuj added 12 commits November 21, 2022 16:51
Change address_encryptor::encrypt() to address_encryptor::operator()()
3d542e4
Replace idempotent pseudonymization transformation with calls to die()
c3dbf73
Move pseudonymization transformation lambda outside of range-based loops
7ded7cd
Remove empty lines in check_address_pseudonymization()
0270705
Make pseudonymization transformations unique
16f610b
Fix formatting in libvast/CMakeLists.txt
e2b8595
Change syntax of changelog item
e1ba0dc
Add new changelog item w.r.t. OpenSSL
c272ae0
Rename address::pseudonymized() to address::pseudonymize()
a404388
Merge branch 'master' of https://github.com/tenzir/vast into story/sc…
d78cdc7 
…-38603
Omit creating idempotent pseudonymization transformations for real now
d3d0d4a
Replace duplicate "currently" with "as of now" in pseudonymize.md
41c832f
dominiklohmann
dominiklohmann reviewed Nov 21, 2022
View reviewed changes
Copy link
Member

@dominiklohmann dominiklohmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some additional small things I've found.

@dominiklohmann
Initialize all address_encryptor variables
ce301f5 
Co-authored-by: Dominik Lohmann <mail@dominiklohmann.de>
dominiklohmann
dominiklohmann approved these changes Nov 22, 2022
View reviewed changes
Copy link
Member

@dominiklohmann dominiklohmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's ship it! (once we're out of feature freeze)

@dominiklohmann dominiklohmann added rc blocked Blocked by an (external) issue and removed blocked Blocked by an (external) issue rc labels Dec 6, 2022
@dominiklohmann dominiklohmann mentioned this pull request Dec 9, 2022
Merged
@dominiklohmann dominiklohmann removed the blocked Blocked by an (external) issue label Dec 10, 2022
@dominiklohmann dominiklohmann merged commit 0f1e40e into master Dec 10, 2022
@dominiklohmann dominiklohmann deleted the story/sc-38603 branch December 10, 2022 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dominiklohmann dominiklohmann dominiklohmann approved these changes

Assignees
No one assigned
Labels
feature New functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Dakostu @dominiklohmann