Enabling "Last Leg Encryption" On Older deployments.

Last Leg Encryption Setup For Existing Deployments

Don't worry, we've made it very easy:

• First, update your local copy of the beiwe-backend codebase, make sure you are on the main branch.
• Deploy the most recent version of the code on the main branch using the eb commandline tool to the Elastic Beanstalk servers.
• Navigate to the cluster management folder and get ready to run a command.
• Run this command, it should execute within a couple seconds with some log statements indicating updated Security Group rules.
  • python launch_script.py -update-old-environment-last-leg-encryption
  • This command will update the network traffic configuration to allow the appropriate connection between the load balancer and the servers.
• Now follow the updated SSL configcuration settings in the deployment guide. (The port mapping of port 443 to port 80 should be changed to 443 to 443. Leave 80 to 80 as untouched.)

(You should run the -terminate-processing-servers and then -create-manager commands somewhere in here to make sure your processing servers are on the same branch as your web servers)

What is Last Leg Encryption

As part of some due-diligence about this issue it is important that we have a full technical explaination of a relatively minor architectural detail. This isn't obscure to experts, but it is detailed and technical enough to require a diagram and an explanation of some terms.

Load Balancer

• A load balancer is a special device, usually a big router, that sits at the "Network Edge" of the data center.
• Load balancers distribute requests coming into the system across multiple servers.
• You need a load balancer of some kind once a system scales past what one server can handle.
  • There are a lot of options for load balancers.
  • We just use the AWS-provided load balancer service, which is called Elastic Load Balancer (ELB).
  • This is what lets the Beiwe Platform scale to very high numbers of users.
• Load balancers are also used to "Terminate SSL Connections"
  • This is the secure piece, the "S", in HTTPS.
    • (there are naming and classification ambiguities I'm cutting here, I will just use "HTTPS")
  • HTTPS is a wrapper around Beiwe's network traffic.
  • It encrypts and ensures that data remains private even if intercepted. (How that works is out of scope.)
  • When the Load Balancer acts as the Termination Point of an HTTPS connection that means a layer of encryption is removed.
• Applying/Teminating encryption for HTTPS has some overhead.
  • Using the load balancer as a dedicated place for this computational load to occur is a common practice.
• But largely for legal reasons (HIPA, GDPR) data needs remain encrypted while in transit.
• "Last leg" refers to the final network transfer, fully within the local network of the data center, and between the load balancer and the Beiwe server(s).
  • We have to make sure that final connection uses its own instance of HTTPS.
  • For a while we had this, but it was a bit manual and had some weird failure modes that could block deployment or fail on server rotation.
  • Then it fully broke as part of configuration issues in a platform/python version upgrade.
  • We have now fixed it and made it more reliable.
  • But enabling it requires a configuration change be taken by the system administrator to the port mapping of your Elastic Beanstalk environment's load balancer.

Where is/was the issue?

I have put together some simple diagrams of the entire data and encryption flow for the Beiwe platform as it was before addressing the issue. It includes sending data out to the storage platform after receipt, this is included for completeness but is not really a "leg" of the same data. ("HTTPS" is a stand-in for secure TLS-encapsulated connections.)

Data Encryption During Upload

Device records encrypts data, only decryptable by server.
             [encrypted raw data]
                      |
             "Crossing The Internet"
          Data encrypted again with HTTPS
                      |
          [[double encrypted raw data]]
                      |
                      V
          Data Center Load Balancer
        <Terminates HTTPS encryption>
                      |
          Load Balancer to Web Server
          (private network connection)     <~ Here (but at least the bulk data is still encrypted)
             [encrypted raw data]
                      |
                      V
          Web Server - Decrypts Data
<Do work, create new [encrypted processed data]>
                      |
HTTPS connection to data storage system ("S3")
         HTTPS encryption applied
                      |
      [[double encrypted processed data]]
                      |
                      V
            S3 Terminates HTTPS encryption
            Stores Encrypted Data

Data Encryption at Download Time

    S3 - Stores Encrypted Data
      API to access uses HTTPS
                |
[[double encrypted processed data]]
                |
                V
            Our Server
      <Decrypts HTTPS connection>
   <Decrypts data's regular encryption>
<Do some work, create output data for user>
      <send unencrypted data to user>
                |
Unencrypted But "Local" Network Connection       <~ Here (potentially an issue)
        [unencrypted data]
                |
                V
          Load Balancer
     <Creates HTTPS connection>
<encrypts [unencrypted data] for HTTPS>
                |
     "Crossing the internet"
        [encrypted data]
                |
                V
      The user's computer
      Decrypts the HTTPS data
       User has their data

Theory vs Practice

Onnela Lab works with the Harvard University IT Department to periodically review our security posture. Any platform security issues identified are resolved, we push out updates to this wiki, and create appropriate announcement issues publicly on our issue tracker and Pin them if necessary. (Public issues tend to be created once the fix is already available.)

Addressing this item of ensuring encryption is applied on the last leg of data transmission has been on our private issue tracker for a while, but while it sounds scary in the context of research and HIPA / GDPR data security, it presented a ver low "threat surface" - the ability to exploit this was always near zero.

In order to exploint this would require:

• Physically breaking into [the correct] highly secure and massive AWS data center,
• Hacking AWS' systems generally to identify "your" specific wire/network port,
• One of:
  • fully hacking and taking over a network switch to run arbitrary code,
  • or tapping into the raw line feed of network traffic using special state-level-espionage technology.
• Identifying your traffic generally, and then particular network traffic within that,
• And all of this would only expose some data specifically when and as you download it.

In short, it requires compromising all the digital and physical security layers at a hundred billion+ dollar company.

And so we decided to address it last.