Adding data type to encrypted column in AcraServer encryption configuration

[wdesplas]

issues:

In the documentation, this is mentioned :

Transparent encryption proxy mode allows you to configure AcraServer to encrypt records in specific database columns without altering the application code.

However, There is not possibility for my not editable application to work with this proxy as :

Encrypted data is binary data. As AcraServer doesn't know the nature of data, it returns the decrypted binary data to the web app. You’ll need to change the source code of your web app for the app to expect the decrypted data as binary, then to encode it into the original format (strings, numbers, bytes, etc.).

Expected behaviour :

As a transparent proxy user , I expect to have to define the database field type either in the acra server/connector configuration and in the database (change encrypted field into binary field) but not in the application code.

Regards,

[vixentael]

@wdesplas totally makes sense, thank you for your request.

Adding data type to the AcraServer's encryption configuration file – that's a feature in our backlog.

More about this feature and security design issues behind it.

It would be very useful to define data types of encrypted columns. However, there are certain design complexities that didn't allow us to simply add this feature to Acra.

Imagine, that data is stored encrypted for certain field, it has binary type. When authenticated application reads encrypted data from a database, AcraServer decrypts it, converts into desired data type, and sends back to the app. All smooth.

However, if application is not authenticated, or doesn't have correct keys, AcraServer won't be able to decrypt encrypted data, and won't be able to convert it to desired data type.

Depending on security requirements and business logic, "can't decrypt" situation might be considered as error, as security issue or as part of a normal application flow. Right now, Acra's users decide how to handle these situations depending on their security requirements and often put this logic into application code. For example, sending security event to SIEM if data field wasn't successfully decrypted.

If your concern is related to wording, will changing this line to "without much altering the application code" help?

[wdesplas]

@vixentael :
Thansk a lot for you reply.
Your sentence make sense however, In my case, as a telecom compagny Engineer, I'm not able to change any code line in the applications.
This is the only feature that does not allow us to make acra working in our environement to move forward to adopt this solution.

regards,

[pvleap]

I am facing the same issue trying to transparently encrypt 2 columns of a postgres table and these columns are of postgres datatype text and hold simple text values. The result returned for my select queries is the hex encoded string. Please let me know if there is any other config file where I can specify the data type of these columns and also if i can specify the dialect of my postgres db?

Also I keep seeing the error "ignoring error of non parsed sql statement" in the acra server logs though there is no one connecting to my acra server. Is it continously polling my encryptor config file and framing a select query against the postgres db with the columns configured in there?

[Lagovas]

@wdesplas , @pvleap, recently we have released a new version of Acra with new features like transparent tokenization that supports string/text types in databases. You can find out more details on our documentation site about configuration AcraServer's encryptor config and about tokenization

[9gunpi]

just for future reference - fix to this (fully transparent mode without all those type concerns) is coming in 0.92-0.93, with limited scope first (not all types will be supported, but most). I will update this issue when that happens.

[kumar1202]

Hey @9gunpi, any updates on the progress of full transparent mode with types support?

[vixentael]

@kumar1202 work in progress. That's not the easiest thing to do considering all the features of SQL protocol. We have progress with PostgreSQL, working on MySQL now. Type awareness is scheduled to the nearest releases.

[kumar1202]

Hey @vixentael, is this feature supported in Enterprise edition? As I was reading through the comparisons table, and found that format preserving encryption is supported in EE.

[vixentael]

@kumar1202 format preserving encryption is different from type awareness.

Type awareness means that it's possible to "tell" AcraServer the original data types of the columns. During encryption, AcraServer will encrypt fields to binary, the database will store fields as binary. During decryption, AcraServer will decrypt fields and cast them to the original data type, so that the application receives decrypted data fields as expected in most cases. Transparent for app, minimum code changes.

Type awareness is a big feature for the upcoming 0.93 release which is scheduled before end of May. It might happen next week, if we are lucky and all edge cases are found, or later in May, if we find more ORMs that behave weird. The feature is ready, we are hunting edge cases and updating playgrounds.

This is what this issue is about.

Type awareness will be available for both Acra CE and Acra EE.

Acra provides different security controls that suit best for different use cases, security guarantees and tech limitations. For some cases, encryption is enough, for others tokenization + encryption, or even masking + tokenization + encryption. Our goal here to give understandable tools with high security guarantees without requiring companies to actually send their sensitive data to a "magic saas".

[vixentael]

Actually, you can see how type awareness will work if you look at these PRs:

https://github.com/cossacklabs/acra-engineering-demo/pull/47/files
https://github.com/cossacklabs/acra-engineering-demo/pull/50/files

These PRs are updating Acra's engineering examples, and we are waiting for 0.93 release before merging them. As I mentioned, type awareness is working for 90% use cases right now, while we are fighting edge cases and interesting SQL protocol caveats trying to release as stable build as possible.

[kpsinghlubana]

hey, @vixentael
I have few doubts regarding tokenization and type awareness.

1. Is tokenization not available in community edition?
2. How does the de-tokenization works? How do we store token's mapping to the original data? documentation mentions the following statement: Both tokenization types require deploying an additional database, Redis by default, to store pairs: token <-> encrypted data..
   Would Acra encrypt the token <-> encrypted data mapping in redis as well?
3. With type awareness in place, the only change that we need to do on the application side is the change of data type of the column to be encrypted to bytea via db migration?

[Lagovas]

hey, @vixentael I have few doubts regarding tokenization and type awareness.

1. Is tokenization not available in community edition?
2. How does the de-tokenization works? How do we store token's mapping to the original data? documentation mentions the following statement: Both tokenization types require deploying an additional database, Redis by default, to store pairs: token <-> encrypted data..
   Would Acra encrypt the token <-> encrypted data mapping in redis as well?
3. With type awareness in place, the only change that we need to do on the application side is the change of data type of the column to be encrypted to bytea via db migration?

Let me answer:

1. It is available in community edition
2. Acra encrypts data, generates new value and store them in the token db as <hash(token)>: <encrypted_data>. So when it receive token, it can find source encrypted value, decrypt and return decrypted
3. Yes, you are right.

[kpsinghlubana]

@Lagovas Thanks for the reply.
As you mentioned, tokenization is available in community edition, does this implies that format preserving encryption feature is available in community edition as well? CMIIW, preserving the format of the data column is made possible using tokenisation feature, right?

[vixentael]

@kpsinghlubana let's step back from the definitions and discuss your use case. what exactly are you trying to do?

yes, tokenization allows to leave the format of original data column, put a non-sensitive token there, but place a real encrypted data into separate column/database.

[vixentael]

Tadam! 🍉

Acra 0.93 release brings Type Awareness!

This release brings type awareness which improves transparent encryption on AcraServer. Type awareness means that it's possible to tell AcraServer what are the original data types for fields. During decryption, AcraServer will convert decrypted fields to their original data types. No need to change client application code to work with "binary data".

It's also possible to choose a default value for each data field if its decryption failed. AcraServer can send a a default value like "<encrypted data>" instead of decryption errors, making developers' and users' life easier.
https://github.com/cossacklabs/acra/releases/tag/0.93.0

Usage examples are in the acra engineering demo repository
https://github.com/cossacklabs/acra-engineering-demo/

Here is how configuration files looks like:
https://github.com/cossacklabs/acra-engineering-demo/blob/master/django-transparent/acra-server-configs/encryptor_config.yaml#L20

Docs:
https://docs.cossacklabs.com/acra/configuring-maintaining/general-configuration/acra-server/encryptor-config/
https://docs.cossacklabs.com/acra/guides/integrating-acra-server-into-infrastructure/

It might look like a simple feature, but under the hood it required to significantly improve SQL parser and handle edge cases that modern ORMs bring to the table (when they hide SQL from end developers, making their life easier, but our life more complicated).

cc @wdesplas @pvleap @kumar1202

Kudos to Acra maintainers: @Lagovas @ZhmakaAS @G1gg1L3s @iamnotacake @shadinua @9gunpi and @vixentael :)