End-to-End Encrypted Chat & the road to HIPAA & GDPR compliance!

Ahoy Back4app community!

This is a guest post from the team at Virgil Security, Inc.: we’re the crypto tech behind Twilio’s End-to-End Encrypted Messaging. Our friends @ Back4app asked us to show you how to build an End-to-End Encrypted chat app on top of Back4app.

In this post, we’ll walk you through the steps to make Back4app’s Android Simple Messenger app End-to-End Encrypted! Are you ready? PS: If you don’t care about the details, simply skip to the end of the post and download the final product.

What is End-to-End Encryption?

First, let’s start with a quick refresher of what E2EE (End-to-End Encryption) is and how it works. E2EE is simple: when you type in a chat message, it gets encrypted on your mobile device (or in your browser) and gets decrypted only when your chat partner receives it and wants to display it in chat window.

The message remains encrypted while it travels over Wi-Fi and the Internet, through the cloud / web server, into a database, and on the way back to your chat partner. In other words, none of the networks or servers have a clue of what the two of you are chatting about.

What’s difficult in End-to-End Encryption is the task of managing the encryption keys in a way that only the users involved in the chat can access them and nobody else. And when I write “nobody else”, I really mean it: even insiders of your cloud provider or even you, the developer, are out; no accidental mistakes or legally enforced peeking are possible. Writing crypto, especially for multiple platforms is hard: generating true random numbers, picking the right algorithms, and choosing the right encryption modes are just a few examples that make most developers wave their hands in the air and end up just NOT doing it.

This blog post will show you how to ignore all these annoying details and quickly and simply End-to-End Encrypt using Virgil’s SDK.

For an intro, this is how we’ll upgrade Back4app’s messenger app to be End-to-End Encrypted:

1. During sign-up: we’ll generate the individual private & public keys for new users (remember: the recipient's public key encrypts messages and the matching recipient's private key decrypts them).
2. Before sending messages, you’ll encrypt chat messages with the recipient's public key.
3. After receiving messages, you’ll decrypt chat messages with the recipient's private key.

We’ll publish the users’ public keys to Virgil’s Cards Service so that chat users are able to look up each other and able to encrypt messages for each other. The private keys will stay on the user devices.

Keep it simple

This is the simplest possible implementation of E2EE chat and it works perfectly for simple chat apps between 2 users where conversations are short-lived and it's okay to lose the message history if a device is lost with the private key on it. For a busier, Slack-like, chat app where history is important and users are joining and leaving channels all the time, we’ll build a Part II for this post: sign up here if you’re interested and we’ll ping you once we finish it.

OK, enough talking! Let’s get down to coding.

• We’ll start by guiding you through the Android app’s setup,
• Then, we’ll make you add the E2EE code and explain what each code block does.

Prerequisites:

• Sign up for a Back4app account and create a new app;
• Sign up for a Virgil Security account (we’ll create the app later)
• You’ll need Android Studio for the coding work, we used 3.0.1.

Let’s set up the Back4app messenger app

1) Import Project in Android Studio:

• File -> New -> Project from Version Control -> Git
• Git Repository URL: https://github.com/VirgilSecurity/chat-back4app-android
• Check out the “clean-chat” branch

Important!

Select "Project" type of file tree. It will be used all-through the tutorial:

2) Set up the App with the Credentials from your new Back4App App’s Dashboard:

• Open “Dashboard” of your app -> “App Settings” -> “Security & Keys”:
• Return to your /app/src/main/res/values/strings.xml file in the project and paste your “App Id” into “back4app_app_id” and “Client Key” into “back4app_client_key”.

<string name="back4app_app_id">your_back4app_app_id</string>
<string name="back4app_client_key">your_back4app_client_key</string>

3) Enable Live Query to get live updates for messages and chat threads:

• Launch Data Management for your app and create two classes: Message and ChatThread:

  

• Go back to your Back4App account

• Press the “Server Settings” button on your Application

• Find the “Web Hosting and Live Query” block

• Open the Live Query Settings and check the “Activate Hosting” option.

• Choose a name for your subdomain to activate Live Query for the 2 classes you created: Message and ChatThread.

• Copy your new subdomain name and click the SAVE button:

Return to /app/src/main/res/values/strings.xml and paste "Subdomain name" you have entered above into the back4app_live_query_url instead of "yourSubdomainName":

<string name="back4app_live_query_url">wss://yourSubdomainName.back4app.io/</string>

Now you can build and run your app on your device or emulator:

If it all worked out, you should see the chat messenger app popping up. Register two users and send a few messages to each other: you should see new data showing up in the Message class:

Note that you can see on the server what your users are chatting about:

Next: Close your chat interface and move on to the next step – adding E2EE encryption.

Now, let’s End-to-End Encrypt those messages!

By the end of this part, this is how your chat messages will look like on the server: can you spot the difference?

And this is how we’ll get there:

• Step 1: we’ll set up a minimal Back4App server app that will approve the creation of new users at registration time: otherwise, you’ll end up with a bunch of spam cards. Later, you can introduce an email/SMS verification by customizing this app!
• Step 2: we’ll modify the messenger app by adding E2EE code; I’ll do my best to explain every step along the way, hopefully simply enough that you can continue playing with it and reuse in your own project!

But before we begin, let’s clear two important terms for you: what’s a Virgil Key and a Virgil Card?

• Virgil Key – this is what we call a user's private key. Remember, private keys can decrypt data that was encrypted using the matching public key.
• Virgil Card – Virgil Сards carry the user’s public key. Virgil cards are published to Virgil’s Cards Service (imagine this service is like a telephone book) for other users to retrieve them: Alice needs to retrieve Bob’s public key in order to encrypt a message for Bob using that key.

Step 1: Set up your App Server

You’ll need some minimal server code to make the sample work. This piece of server code will enable you to verify new users before they can start using Virgil’s crypto service. To keep this app server simple, we created one for you: it will automatically approve all new users. Later, you can add your own SMS/email verification code, so that you won’t end up with a ton of false users.

Let’s get started:

• Download this archive that contains two files: main.js and package.json;
• Extract the archive and open main.js with your favorite editor;
• Go back to your Virgil developer account and create a new application. Make sure that you save the Private Key that is generated for your application. Also, copy the new app’s base64-encoded AppKey string before you complete the app creation:
• Edit main.js, find the function resolveAppKEy() and replace:
  • YOUR_VIRGIL_APP_PRIVATE_KEY with the Base64 AppKey on your clipboard
  • YOUR_VIRGIL_APP_PRIVATE_KEY_PASSWORD with the password you’ve set for your new Virgil app:

    function resolveAppKey() {
      try {
        return virgil.crypto.importPrivateKey('MIGhMF0GCSqGSIb3DQEFDTBQMC8GCSqGSIb3DQEFDDAiBBAmU9m+EJOvLRxRaJP6d......',
          'a0KEOifsd2Ean6fzQ'
        );
      } catch (e) {
        return null;
      }
    }

• Now, go back to your Virgil dashboard and copy your new app’s App ID:
• Find the function signCardRequest and replace YOUR_VIRGIL_APP_ID with the ID on your clipboard:

function signCardRequest(cardRequest) {
  const signer = virgil.requestSigner(virgil.crypto);
  signer.authoritySign(cardRequest, 'bd7bf7e832f16e2b3f6fd343s1f90778ab0e15515aa775e7b7db3', appKey);
}

• Now, let’s get back to the Virgil dashboard and create a new token for your app:
• Choose any name, leave all settings as-is and generate the token:

- As a result, you get the Access Token:

- Copy the Token to the clipboard, find the function `signCardRequest(cardRequest, appKey)` and replace `YOUR_VIRGIL_APP_ACCESS_TOKEN` with the token on your clipboard:

signCardRequest(cardRequest);
const client = virgil.client('AT.8641c450a983a3435aebe79sad32abea997d29b3e8eed7b35beab72be3');
client.publishCard(cardRequest)
...

Now, back to the Back4app dashboard:

• Go to your App Dashboard at Back4App website:

• Open “Server Settings” and find “Cloud Code”:

  

• Open Cloud “Settings”

• Upload the main.js and package.json files in your Cloud Code settings and press “save” button:

Step 2: Update Android app with E2EE code

Let’s update your InfoHolder Class:

• Add new fields:

private VirgilApi virgilApi;
private VirgilApiContext virgilApiContext;
private KeyStorage keyStorage;

• Update constructor to initialize created fileds:

public InfoHolder(Context context) {
    keyStorage = new VirgilKeyStorage(context.getFilesDir().getAbsolutePath());

    virgilApiContext = new VirgilApiContext(context.getString(R.string.virgil_token));
    virgilApiContext.setKeyStorage(keyStorage);

    virgilApi = new VirgilApiImpl(virgilApiContext);
}

• Add getters for new fields, so you will be able to access it all-through the application:

public VirgilApi getVirgilApi() {
    return virgilApi;
}

public VirgilApiContext getVirgilApiContext() {
    return virgilApiContext;
}

public KeyStorage getKeyStorage() {
    return keyStorage;
}

Add Virgil’s Android SDK to your project:

• In the app-level gradle at /app/build.gradle:

implementation "com.virgilsecurity.sdk:crypto-android:$rootProject.ext.virgilSecurity"
implementation "com.virgilsecurity.sdk:sdk-android:$rootProject.ext.virgilSecurity"

• Add the following to the end of your project-level /build.gradle:

virgilSecurity = "4.5.0@aar"

Enter your access token:

• Open /app/src/main/res/values/strings.xml and add copy & paste your Virgil token and App ID from the Virgil app dashboard:

<string name="virgil_token">your_virgil_token</string>
<string name="virgil_app_id">your_virgil_app_id</string>

Note: for simplicity, we re-used the access token you created for the server app. Don’t do this in production: create a separate token for your mobile app with Search-only permissions!

We're ready to use Virgil SDK now!

For every chat user, the new E2EE app maintains a private & public key:

1. We generate the private & public key pair as part of signup
2. Store the private key in the key storage on the device
3. Publish the public key in Virgil’s Card Service as a “Virgil Card” for other users to download & encrypt messages with it.

1. Generate Private Key & Public Key then store Private Key

Let’s make some updates to ../virgilsecurity/virgilback4app/auth/LogInPresenter class.

• Add VirgilCard and VirgilKey fields:

private VirgilCard virgilCard;
private VirgilKey virgilKey;

• Update signUp method passing virgilCard as argument, so it will looks like:

restartableFirst(SIGN_UP, () ->
                         RxParse.signUp(identity,
                                        password,
                                        virgilCard)
                                .subscribeOn(Schedulers.io())
                                .observeOn(AndroidSchedulers.mainThread()),
                 LogInFragment::onSignUpSuccess,
                 LogInFragment::onSignUpError
);

• Add method where we will generate the private key and the public key (for decrypting incoming chat messages) then saving it ../virgilsecurity/virgilback4app/auth/LogInPresenter:

private void generateKeyPair(String identity) {
    virgilKey = infoHolder.getVirgilApi().getKeys().generate();
    try {
        virgilKey.save(identity);
    } catch (VirgilKeyIsAlreadyExistsException e) {
        e.printStackTrace();
    }
    virgilCard = infoHolder.getVirgilApi().getCards().create(identity, virgilKey);
}

• Now we have to call this method when requesting sign up and generate password using private key:

void requestSignUp(String identity) {
    this.identity = identity;

    generateKeyPair(identity);
    password = generatePassword(virgilKey.getPrivateKey().getValue());

    start(SIGN_UP);
}

When user will request log in - you have to generate password using private key loaded with user’s identity (currently - username):

void requestLogIn(String identity) {
    this.identity = identity;

    try {
        virgilKey = infoHolder.getVirgilApi().getKeys().load(identity);
    } catch (KeyEntryNotFoundException e) {
        getView().onLoginError(e);
        return;
    } catch (VirgilKeyIsNotFoundException e) {
        getView().onLoginError(e);
        return;
    } catch (CryptoException e) {
        getView().onLoginError(e);
        return;
    }
    password = generatePassword(virgilKey.getPrivateKey().getValue());

    start(LOG_IN);
}

Create and Publish Virgil Card

As you already created VirgilCard few moments ago and passed it to the signUp method - you have to handle new argument and publish Virgil Card.

We pass exported Virgil Card to the Back4App code that will intercept the create user request and publish the Virgil Card on Virgil Cards Service.

Now you need to send this Card request to the App Server where it has to be signed with your application's Private Key (AppKey).

The VirgilCard object has a convenience method called export that returns the base64-encoded string representation of the request suitable for transfer (../virgilsecurity/virgilback4app/util/RxParse.javaRxParse class):

public static Observable<VirgilCard> signUp(String username,
                                            String password,
                                            VirgilCard card) {
    return Observable.create(e -> {
        final ParseUser user = new ParseUser();
        user.setUsername(username);
        user.setPassword(password);
        user.put("csr", card.export());

        user.signUpInBackground((exception) -> {
            if (exception == null) {
                e.onNext(card);
                e.onComplete();
            } else {
                e.onError(exception);
            }
        });
    });
}

Now your project automatically sends the Virgil Card exported to base64 to the Back4App, after that Cloud Code intercepts and publishes it.

We encrypt messages with the recipient user's Virgil card.

Let's add some code to encrypt data to the ../virgilsecurity/virgilback4app/chat/thread/ChatThreadPresenter class:

• Add some new fields:

private static final int GET_CARDS = 2;

private VirgilApi virgilApi;
private String identitySender;
private String identityRecipient;
private List<VirgilCard> cards;

• Init VirgilApi in the onCreate method:

@Override protected void onCreate(Bundle savedState) {
    super.onCreate(savedState);

    virgilApi = AppVirgil.getInfoHolder().getVirgilApi();
...
}

• Add method that encrypts data:

private String encrypt(String text, List<VirgilCard> cards) {
    try {
        VirgilCards virgilCards = new VirgilCards(AppVirgil.getInfoHolder().getVirgilApiContext());
        virgilCards.addAll(cards);
        return virgilCards.encrypt(text).toString(StringEncoding.Base64);
    } catch (CryptoException e) {
        e.printStackTrace();
        return "";
    }
}

• Find method sendMessage and update it calling encrypt method on message text before sending, so it will looks like:

        restartableFirst(SEND_MESSAGE, () ->
                                 RxParse.sendMessage(encrypt(text, cards),
                                                     thread)
                                        .observeOn(AndroidSchedulers.mainThread()),
                         ChatThreadFragment::onSendMessageSuccess,
                         ChatThreadFragment::onSendMessageError);

• Update requestSendMessage method passing to it List<VirgilCard> and initializing with it field:

    void requestSendMessage(String text,
                            ChatThread thread,
                            List<VirgilCard> cards) {
        this.text = text;
        this.thread = thread;
        this.cards = cards;

        start(SEND_MESSAGE);
    }

That’s almost it! But as you might have noticed - we need sender and receiver’s Virgil Cards.

So let’s go on updating ChatThreadPresenter class to be able to get Virgil Cards from Virgil Card Service.

• Add request method:

void requestGetCards(String identitySender, String identityRecipient) {
    this.identitySender = identitySender;
    this.identityRecipient = identityRecipient;

    start(GET_CARDS);
}

• Add new restartableFirst to connect presenter with fragment to the onCreate method along with others restartableFirst's (onGetCardSuccess and onGetCardError methods will be implemented later):

restartableFirst(GET_CARDS, () ->
                           Observable.zip(findCard(identitySender).toObservable()
                                                            .subscribeOn(Schedulers.io()),
                                          findCard(identityRecipient).toObservable()
                                                            .subscribeOn(Schedulers.io()),
                                          Pair::new)

                                             .observeOn(AndroidSchedulers.mainThread()),
                   ChatThreadFragment::onGetCardSuccess,
                   ChatThreadFragment::onGetCardError);

• Add method that will find Virgil Card via its identity on Virgil Cards Service:

private Single<VirgilCard> findCard(String identity) {
    return Single.create(e -> {
        VirgilCards cards = virgilApi.getCards().find(identity);
        if (cards.size() > 0) {
            e.onSuccess(cards.get(0));
        } else {
            e.onError(new VirgilCardIsNotFoundException());
        }
    });
}

• Update isDisposed() and disposeAll() methods:

void disposeAll() {
    stop(GET_MESSAGES);
    stop(SEND_MESSAGE);
    stop(GET_CARDS);
}

boolean isDisposed() {
    return isDisposed(GET_MESSAGES)
            || isDisposed(SEND_MESSAGE)
            || isDisposed(GET_CARDS);
}

Now you have to update ChatThreadFragment class to handle get cards request.

• First you have to add fields:

private VirgilCard meCard;
private VirgilCard youCard;

• Implement initCards() method:

private void initCards() {
    showProgress(true);

    getPresenter().requestGetCards(thread.getSenderUsername(),
                                   thread.getRecipientUsername());
}

• Update getMessages() method to check whether Virgil Cards are already fetched:

private void getMessages() {
    if (meCard == null || youCard == null) {
        initCards();
    } else if (messages == null || messages.size() == 0) {
        showProgress(true);
        getPresenter().requestMessages(thread, 50, page,
                                       Const.TableNames.CREATED_AT_CRITERIA);
    }
}

• Update onInterfaceClick with creating List of VirgilCard's and passing it to the requestSendMessage method:

@OnClick({R.id.btnSend}) void onInterfaceClick(View v) {
    switch (v.getId()) {
        case R.id.btnSend:
            String message = etMessage.getText().toString().trim();
            if (!message.isEmpty()) {
                List<VirgilCard> cards = new ArrayList<>();
                cards.add(meCard);
                cards.add(youCard);
                lockSendUi(true, true);
                getPresenter().requestSendMessage(message, thread, cards);
                isLoading = true;
            }
            break;
    }
}

• Add callbacks for successful and unsuccessful Virgil Cards fetching:

public void onGetCardSuccess(Pair<VirgilCard, VirgilCard> cards) {
    if (cards.first.getIdentity().equals(ParseUser.getCurrentUser().getUsername())) {
        meCard = cards.first;
        youCard = cards.second;
    } else {
        meCard = cards.second;
        youCard = cards.first;
    }

    if (meCard != null && youCard != null) {
        if (messages == null) {
            showProgress(false);
            getPresenter().requestMessages(thread, 50, page,
                                           Const.TableNames.CREATED_AT_CRITERIA);
        } else {
            showProgress(false);
            srlRefresh.setRefreshing(false);
            lockSendUi(false, false);
        }
    }
}

public void onGetCardError(Throwable t) {
    showProgress(false);
    srlRefresh.setRefreshing(false);
    lockSendUi(false, false);

    Utils.toast(this, Utils.resolveError(t));
}

Decrypt the Encrypted Message in the HolderMessage class (nested in the ChatThreadRVAdapter class)

General logic:

1. We first load the user’s private key from Android’s secure storage
2. Then we use it to decrypt the message received

• Add VirgilKey field:

private VirgilKey virgilKey;

• Init VirgilKey field in HolderMessage constructor:

HolderMessage(View v) {
    super(v);
    ButterKnife.bind(this, v);

    try {
        virgilKey = AppVirgil.getInfoHolder()
                             .getVirgilApi()
                             .getKeys()
                             .load(ParseUser.getCurrentUser().getUsername());
    } catch (VirgilKeyIsNotFoundException e) {
        e.printStackTrace();
    } catch (CryptoException e) {
        e.printStackTrace();
    }
}

• Implement decrypt method:

String decrypt(String text) {
    try {
        return virgilKey.decrypt(text).toString();
    } catch (CryptoException e) {
        e.printStackTrace();
        return "";
    }
}

• Decrypt message before displaying:

void bind(Message message) {
    tvMessage.setText(decrypt(message.getBody()));
}

That's all for encryption and decryption. And finally - you have to:

• Update resolveError method in Utils class to handle some exceptions that can be thrown during the work with Virgil Security SDK. So add handling for VirgilKeyIsNotFoundException, VirgilKeyIsAlreadyExistsException, VirgilCardIsNotFoundException and KeyEntryNotFoundException so corresponding part of resolveError method will looks like:

...
} else if (t instanceof ParseException) {
    ParseException exception = (ParseException) t;

    switch (exception.getCode()) {
        case ParseException.USERNAME_TAKEN:
            return "Username is already registered.\nPlease, try another one. (Parse)";
        case ParseException.OBJECT_NOT_FOUND:
            return "Username is not registered yet";
        case 60042: // Custom exception in RxParse.class
            return exception.getMessage();
        default:
            return "Oops.. Something went wrong ):";
    }
} else if (t instanceof VirgilKeyIsNotFoundException) {
    return "Username is not registered yet";
} else if (t instanceof VirgilKeyIsAlreadyExistsException) {
    return "Username is already registered. Please, try another one.";
} else if (t instanceof KeyEntryNotFoundException) {
    return "Username is not found on this device. Maybe you deleted your private key";
} else {
    return "Something went wrong";
}
...

• In ChatThreadFragment update onGetCardError method inserting next code in the very beginning of it:

public void onGetCardError(Throwable t) {
    if (t instanceof VirgilCardIsNotFoundException
            || t instanceof VirgilCardServiceException) {
        Utils.toast(this, "Virgil Card is not found.\nYou can not chat with user without Virgil Card");
        activity.onBackPressed();
    }
...
}

Important!

You have to Log Out the current user and register two new users, after that you can start E2EE chat with those two new users. The reason is that your first two users have no Virgil Card's, so you can not use encrypt\decrypt for them.

HIPAA & GDPR compliance:

End-to-End Encryption is a way to meet the technical requirements for HIPAA (the United States' Health Insurance Portability and Accountability Act of 1996) & GDPR (the European Union's General Data Protection Regulation). If you need more details, sign up for a free Virgil account, join our Slack community and ping us there: we’re happy to discuss your own privacy circumstances and help you understand what’s required to meet the technical HIPAA & GDPR requirements.

Where to go from here?

Final project. If you missed pieces from the puzzle, open the E2EE project branch. You can insert your application credentials in this code (as you did during the article) and build the project.

Don’t forget to subscribe to our Youtube channel. There you will find a video series on how to do End-to-End Encryption.

Also, use Virgil Security to verify the integrity of data at any point. Data Integrity is essential to anyone who wants to guarantee that their data has not been tampered with. Our tutorial provides more details.

You can find more information about what you can build with Virgil Security here.

Follow our posts on Back4App. In the next tutorial, we will be helping two people or IoT devices to communicate with End-to-End Encryption with PFS enabled. You’ll find out how to protect previously intercepted traffic from being decrypted even if the main private key is compromised.