Skip to content

Design Notes

Prasad Talasila edited this page Oct 7, 2017 · 8 revisions

Execution Nodes

Secure Execution in Containers

We must strive to undertake secure execution of user code with bare minimal privileges. We initially thought of containers as a cost effective way of achieving the effect. The idea is to create a container in advance, keep the container running in anticipation of user request. Once an evaluation request comes, the user files are copied into the container via mapped volumes, evaluated inside the container, and the results are left in the mapped volume itself. At the end of the evaluation work, the container is stopped and deleted. Done this way, one users code evaluation has no way of influencing subsequent code evaluations. We can take advantage of Docker API and wrapper clients to perform the fast life cycle management of containers.

The overall code isolation should be better than achieved by SELinux-based user jail method used by CS50 check component. The backend library component of INGInious has a controller component that does dynamic spawing of containers that are customized to each course / assignment. We can reuse this library. For further details, see the architecture section of the INGInious paper.

See also the Docker remote API for automating container lifecycle management.

Misfit of P2P model

For a long time, we were thinking about leveraging P2P model to outsource the deployment of execution nodes. There are two primary concerns with this model.

  1. Security requirements are much more complex.
  2. Cognitively, this deployment model is equivalent to a teacher asking the students to self-evaluate. It may be difficult for teacher / evaluator to accept such a model.

Instead, if we put emphasis on making the execution node infrastructure auto-scalable in a trusted environment, we get more tangible benefits.

Security Model

Each component of project currently uses asymmetric PKI using SSL certificates. This setup only provides secrecy, but not the authentication. We need to use symmetric key setup from NaCl (sodium) security library to provide symmetric security.

Secure Evaluation in Untrustworthy Environments using Exam Protocols

  1. Remark!: A Secure Protocol for Remote Exams, Rosario Giustolisi
  2. Design and Analysis of Secure Exam Protocols, Rosario GIUSTOLISI at University of Luxembourg thesis
  3. Trustworthy Exams Without Trusted Parties, Giampaolo Bella et al., Elsevier Computers & Security paper, another version
  4. A Secure Exam Protocol Without Trusted Parties, Giampaolo Bella et al., 2016 paper c,ode
  5. A SECURE ELECTRONIC EXAM SYSTEM, ANDREA HUSZTI, ATTILA PETHO paper
  6. Formal Correctness of Security Protocols, Bella, Giampaolo
  7. Papers of Rosario Giustolisi

Resource Usage Statistics

In order to be truly useful to algorithms courses, the execution node must collect the resource usage statistics of the user code after completion of evaluation.

Load Balancer

Refactor Load Balancer

The load balancer can be broken down into 2 conponents(LB1 and LB2) . One would handle communications with the main server, mysql and execution nodes. The second would schedule the process and dynamically change the number of execution nodes.

Assumptions when devising a mechanism for addition/removal of nodes -

  1. The amount of time taken for execution of one submission is 5 seconds.
  2. Out of all the pending jobs, each node gets a maximum of 5 submissions to execute.
  3. Hence the maximum waiting time for each submission is 25 seconds.

The second component of the load balancer will have an array of unused ports from 8081 to 8181 from which the new nodes will be attached, one node_queue, array with all the available nodes and job_queue, array with all the pending jobs.

  1. When LB1 recieves from the main server, forwards the request to LB2 as per the following algorithm:-
    if node is available :
         send the job with the scheduled node to LB1
    else if number_of_jobs >= number_of_nodes*5 :
         create a new node, send the job with the new node to LB1
    else :
         wait for a node to complete execution.

  2. When LB1 receives a submission from one of the nodes, it forwards the score to the main server, and appends the database. The node details are sent to LB2 -
    if number_of_nodes*5 >= number_of_jobs :
         remove node
    else :
         if job is pending :
             send the job with the node_details to LB1 for execution
         else :
             add node to queue

Autolab Clients

React + MVC vs Microservices

We chose microservices architecture for its scalable deployment model. Even though its not in place, we want to reach a scenario of automatic load balancing of execution nodes using either docker-compose or ansible-tower. The overall architectural principles for the project are outlined in wiki page. I agree that the existing architecture is amenable to react + MVC, the proposed architecture is more powerful. Hence, we chose to stick with Microservices framework. However on the web browser client side, we use socket.io which provides an event-based network protocol capability. I certainly agree that we might use rxjs to improve the client-side capability. Right now, I am not able to choose between the following choices.

  1. FSM ( redux + redux-machine / machina / stately.js ) on the client-side with socket.io feeding events to the FSM
  2. rxjs and redux wrapping around the events of socket.io

For me, choice of FSM looks more modular. But I heard a lot of good things about react. I am not able to see a way to use react with network calls between two independent components that run on different hosts. If there is good programming idiom / tutorial to use for that scenario, we can learn from such a document. If you can suggest a sample structure for our web browser client using socket.io + rxjs that can also work as a good candidate to compare against FSM + socket.io alternative.

References

  1. Filename extensions of security certificates.
Clone this wiki locally
You can’t perform that action at this time.