Publice Release of Artifacts

This release contains a Ubuntu 20.04 VirtualBox VM, that contains the client testing setup used for our paper. Currently, only Thunderbird is setup for testing inside the VM.

Uncompressing

1. Download all files in this release.
2. Concatenate the *.gz.* files, for example using cat EAST.ova.gz* > EAST.ova.gz
3. Unzip using gzip.

Note: Steps 2 and 3 can be combined using the command cat EAST.ova.gz* | gunzip > EAST.ova

Virtual Machine

You should be able to import the provided VirtualBox VM (tested on VirtualBox 6.1.24, note that some older version have a bug that prevents running the nested VM). Please make sure nested virtualization (Nested VT-X/AMD-V) is enabled. Otherwise, the embedded Thunderbird VM will not work.

Requirements:

• 2 processors
• Nested Virtualization
• 6 GB of memory (due to the nested VM)

Increasing CPU/Memory will probably make the experience better, but everything should be testable with these settings.

Credentials: artifacts:artifacts

Client Testing

The VM contains the fake_mail_server in /home/artifacts/fake_mail_server/. For details on usage, see the README for the server in the repository.

We provide a subset of our test setup for email clients (MUAs). Specifically, we provide 3 VM snapshots to reproduce our results for the Thunderbird MUA. However, you should be able to configure any email client of your choice using the initial snapshot -- even one we did not test!

Overview

In contrast to server testing, client testing is more difficult. A server testing tool can connect to an always-open port. In client testing, our test server needs to wait for an incoming client connection. Thus, we needed to set up an environment where our test server can "request" a connection from any client. To solve this problem, we used VM snapshots with pre-configured clients, which are ready to connect to our email server upon a simple event such as a button click. Concretely, Thunderbird has three snapshots: thunderbird_smtp, thunderbird_pop3, and thunderbird_imap. The POP3 and IMAP variants provide a pre-configured Thunderbird with a running script that will click on the RECEIVE button as soon as the server connects to port 12345 of the VM. In the SMTP variant, we composed an email, and the script will then press the SEND button.

Testing Thunderbird

In order to execute a test case, use the test subcommand. For example, to execute all STARTTLS-stripping tests against IMAP run

$ sudo ./fake_mail_server test thunderbird imap testcases/imap/Negotiation/O_*.ron

To see what is going on inside the tested VM, remember to open it in virt-manager (e.g., for the UI tests).

Interpreting the Server output

The server output of the O_1 IMAP test will look like this:

testcases/imap/Negotiation/O_1.ron ("thunderbird") > pzKfOLCA | S: * OK [CAPABILITY IMAP4REV1 AUTH=PLAIN AUTH=LOGIN] IMAP server ready.\r\n
testcases/imap/Negotiation/O_1.ron ("thunderbird") > pzKfOLCA | C: 1 STARTTLS\r\n
testcases/imap/Negotiation/O_1.ron ("thunderbird") > pzKfOLCA | S: 1 BAD STARTTLS not supported.\r\n
testcases/imap/Negotiation/O_1.ron ("thunderbird") > pzKfOLCA | {"message": "Connection was closed."}

Every line starts with the test case and application, followed by a unique ID (pzKfOLCA) for the test run. The actual server message starts after |. Lines starting with S: are messages sent by the server, while lines starting with C: are messages sent by the client. .. indicates that the line continues the previous line of a multiline message. Lines without a special Identifier in front of them are general (error) messages. In this example, the client reacted by closing the connection after receiving a BAD response to STARTTLS.

Setup any MUA yourself (optional)

The initial snapshot is meant to demonstrate how clients were set up to work with the testing server. You do not have to execute this step, but it shows how we configured every MUA. If you want to reproduce this step, you should execute the setup subcommand of the fake_mail_server ...

$ sudo ./fake_mail_server setup

... and try to configure a new email account in any MUA of your choice. The server mimics a real email server such that no external servers (e.g., Postfix + Dovecot) must be installed for testing.

You should be able to install any MUA in the version from the paper by executing ...

$ nix-env -f nixpkgs-20.03 -i <MUA>

... without sudo.

Ensure that either the client uses the system provided RootCAs or include the rootCA.pem on the NixOS user's desktop.

The server listens on the default STARTTLS ports (587, 110, 143) and accepts any credentials. The VM is configured to resolve example.org to the host system.

Generate snapshots for resets

(For Details also see the README.md on the Desktop of the NixOS VM)

1. Install and configure the client as described above.
2. Run watch -n1 'xdotool getmouselocation' in a terminal.
3. Switch to the mail client and move the mouse over the Send/Receive button. (For SMTP, create a new mail first).
4. Check the mouse position in the terminal (without moving the mouse, use alt-tab).
5. Modify the run_*.sh script for the current protocol on the desktop with the new coordinates.
6. Run the changed run script from the terminal and switch to the mail client.
7. Create snapshot with the name <application>_<protocol>

Additional Notes

Origin of the nested VM

The NixOS VM is based on

https://releases.nixos.org/nixos/19.09/nixos-19.09.2008.ea553d8c67c/nixos-graphical-19.09.2008.ea553d8c67c-x86_64-linux.iso

and we used the 20.03 nixpkgs tag

https://github.com/NixOS/nixpkgs/tree/20.03

to install all MUAs.

Inside NixOS, the command

nix-env -f nixpkgs-20.03 -i <MUA>

will install any MUA in the exact version as provided in the paper.

Creation of keys and certificates

We used "real" certificates for a private domain issued by Let's Encrypt during the evaluation.

For the this release, we used the mkcert tool

https://github.com/FiloSottile/mkcert

with the following commands

$ mkcert -install
$ mkcert -pkcs12 example.org

The default password used by mkcert is changeit.