Skip to content

v0.3.0 Deployment on Two Machines

Prasad Talasila edited this page Jan 25, 2018 · 4 revisions

Prerequisite

  1. Ubuntu server 16.04 operating system.

  2. Record the Domain names, Public and Private IP Addresses and Login Credentials of both the machines. Let's label the IP addresses as follows.

    Machine Public IP Private IP
    Front-End-and-LB Public_IP_1 Private_IP_1
    Execution-Nodes Public_IP_2 Private_IP_2

    Because of ease of use, domain names (DNS) are a preferred option for hostnames in all the configuration files. If you have domain names for the machines, you can use the domain names in the place of IP addresses in all the configuration files.
    Most likely, your machine may be on a corporate private network with a gateway / proxy server / firewall connection to Internet. In that case, please use your machine's private ip address.
    You can locate the configured IP address of your machine using the following command.

    ip addr show
    

    in the displayed output, look for IP address following the inet keyword.

  3. Change the listening port of SSH server to 2222 in /etc/ssh/sshd_config file and restart the SSH server using the following command. This should be done on both the machines.
    sudo service ssh restart

  4. For the SSH setting to be fully effective, logout of the current SSH session and relogin.

  5. From Machine 1, SSH into Machine 2 using its Private IP once to add the SSH server public key of Machine 2 to ~/.ssh/known_hosts in Machine 1.

  6. Note: This setup installs main_server, load_balancer, gitlab, mysql, and 5 execution _nodes on Machine 1. On Machine 2, it installs 5 execution_nodes. You can install any number of execution nodes by configuring the config files on either of the machines.

On Machine 1

  1. Download the code repository
    wget https://github.com/AutolabJS/AutolabJS/archive/dev.zip -O AutolabJS.zip
    unzip AutolabJS.zip
    mv AutolabJS-dev AutolabJS

  2. Change to deploy directory
    cd AutolabJS/deploy

  3. Specify the installation directory by changing the INSTALL_DIR variable in setup.conf file. If unchanged, the installation defaults to /opt/autolabjs directory.

  4. The following steps will setup five execution nodes on Machine 1. To increase/decrease this number, update the variable NUMBER_OF_EXECUTION_NODES in the files keys.conf and setup.conf. In case you intend to increase the number of nodes, the config files in the directory, configs/execution_nodes/ will have to be replicated for the same.
    For example, you need to setup 6 nodes on Machine 1. By default, the configs for 5 nodes are available. You can copy the directory execution_node_1 to execution_node_6 with the command:
    cp -r configs/execution_nodes/execution_node_1 configs/execution_nodes/execution_node_6.
    The files scores.json and conf.json in the newly created execution_node_6 directory will need to be updated. The port number and hostname will be updated here.
    This should be done for all the additional nodes.

  5. Copy AutolabJS codebase to installation base directory.
    bash copy.sh

  6. All the further steps in the installation procedure need to be executed from the installation directory. Go to installation directory.
    cd /opt/autolabjs

  7. Install third-party software dependencies required for AutolabJS.
    cd deploy bash dependencies.sh

  8. AutolabJS uses docker containers for implementing microservices. Docker containers are bulky and require significant download time. This step helps you download the docker container images.
    cd ../docker-images
    bash docker-pull.sh
    The script should download three docker containers. If the download of container images fails due to network issues, you can re-run the bash docker-pull.sh command repeatedly until all the three container images are downloaded.

  9. Build the docker containers of AutolabJS components by installing the required dependencies on the containers downloaded in the previous step.
    bash docker_build.sh
    The image building step can fail due to network errors. You can run the bash docker_build.sh as many times as you need until all the required docker images have been built. You can check the successful completion of this step by running the following command and checking the expected output.
    sudo docker images
    The expected output is as follows.

    execution_node      latest              ...
    load_balancer       latest              ...
    main_server         latest              ...
    ubuntu              16.04               ...
    gitlab/gitlab-ce    ce:10.1.4-ce.0      ...
    mysql               latest              ...
    
  10. Configure the autolab components by editing the configuration files in deploy/configs/<component>. Note that you need to use the Private IP of the machines wherever required.
    The list of configuration files to be changed are:
    deploy/configs/main_server: APIKeys.json, conf.json, courses.json, labs.json
    deploy/configs/load_balancer: nodes_data_conf.json
    deploy/configs/execution_nodes: scores.json, conf.json

  11. Update the certificate information in deploy/openssl.cnf and deploy/keys.conf.

  12. Edit the ansible inventory file. You can find the inventory in AutolabJS/deploy directory.

    1. Update the credentials as required. Please make sure that the changed passwords match in both the inventory file and the configuration files.
      The locations of credentials are:

      File MySQL GitLab Main Server
      deploy/inventory
      deploy/configs/main_server/APIKeys.json ✔ (for /admin route)
      deploy/configs/main_server/conf.json
      deploy/configs/load_balancer/nodes_data_conf.json
    2. In the gitlab section of inventory file, change the hostname of gitlab item from localhost to Public IP. The line should look like

    [gitlab]    
    
    <Public_IP_1> ansible_connection=local ansible_python_interpreter=/usr/bin/python2 gitlab_password=<gitlab_password>
    

    The only exception to the use of private IP is given in the above line. If your Gitlab installation runs on a computer with one public IP address and one private IP address, use the public_IP_1 address in the above line. Otherwise, use private_IP_1 address.

    1. In the loadbalancer section of inventory file, change gitlab_hostname variable in loadbalancer group to Private IP of Machine-1.
    [loadbalancer]
    
    localhost ansible_connection=local ansible_python_interpreter=/usr/bin/python2 
    submissions_dir=../load_balancer/submissions gitlab_hostname=<Private_IP_1> gitlab_password=<gitlab_password>
    
    1. In the executionnode1 section of inventory file, change gitlab_hostname variable to Private IP of Machine 1. The line should look like
    [executionnode1]
    
    localhost ansible_connection=local ansible_python_interpreter=/usr/bin/python2 gitlab_hostname=<Private_IP_1> 
    gitlab_password=<gitlab_password> port_1=<PORT_NUMBER>
    

    Repeat the same change for all the execution nodes.

  13. In the config file for loadbalancer at deploy/configs/load_balancer, you need to add all the details for the additional nodes to be deployed on Machine 2. The Nodes array should be updated as,

    {
        "Nodes": [
          {
              "role": "execution_node",
              "hostname": "<Public_IP_1>",
              "port": "8082"
          }, 
          ...,
          {
              "role": "execution_node",
              "hostname": "<Public_IP_2>",
              "port": "8083"
          },...
          
        ], ...
        other details 
    } 
    
  14. Execute setup script
    ./setup.sh

  15. Run the ansible-playbook
    sudo ansible-playbook -i inventory playbook-single.yml --ask-sudo-pass
    You will be asked for sudo password. Please enter the password and proceed with the installation on Machine 1.

  16. To install any additional node, for which you had created additional configuration files in Step 4, run the following command:
    Here, the port number should be a unused port, for convenience one higher than the previous execution node.
    docker run --net=host --name=execution-node-<Public_IP_1>-<Port_No> -e GITLAB_IP='<GITLAB_IP>' -e LOGGER_CONFIG='/etc/util/logger.json' -e ENCONFIG='/etc/execution_node/conf.json' -e ENSCORES='/etc/execution_node/scores.json' -v /opt/autolabjs/execution_nodes/execution_node_6:/execution_nodes -v /etc/localtime:/etc/localtime:ro -v /opt/autolabjs/deploy/configs/execution_nodes/execution_node_6:/etc/execution_node -v /opt/autolabjs/deploy/keys/gitlab/execution_nodes/execution-node-<Public_IP_1>-<Port_No>:/root/.ssh/ -v /opt/autolabjs/util:/util -v /opt/autolabjs/deploy/configs/util:/etc/util -v /opt/autolabjs/log/execution_nodes/execution_node_6:/log -dit execution_node
    Remember to change execution_node_6 to the next execution_node, like execution_node_7 after creating the necessary files in Step 4, for adding the next additional execution node.

  17. Ansible installs all the components of Autolab on Machine 1. You can now move to installation on Machine 2.

On Machine 2

  1. Repeat steps 1-7 of Machine 1 Installation.

  2. You only need the execution node image on this system. For building this image, first change the directory
    cd ../execution_nodes/execution_node_1/
    Then run the command to build the image.
    sudo docker build -t execution_node .
    This command can be repeated multiple times if any failure is encountered.

  3. Return back to the deploy directory.
    cd ../../deploy

  4. Configure the autolab components by editing the configuration files in deploy/configs/execution_nodes. Note that you need to use the Private IP of the machines wherever required.
    The list of configuration files to be changed are:
    deploy/configs/execution_nodes: scores.json, conf.json

  5. Update the certificate information in deploy/openssl.cnf and deploy/keys.conf.

  6. Edit the ansible inventory file. You can find the inventory in AutolabJS/deploy directory.

    1. Update the credentials for the execution nodes. You can ignore the credentials of the main server, load balancer and gitlab components. Please make sure that the changed passwords match in both the inventory file and the configuration files. Also the credentials used on Machine-2 must match the credentials used on Machine-1.
    2. In the executionnode1 section of inventory file, change gitlab_hostname variable to Private IP of Machine 1. The line should look like
    [executionnode1]
    
    localhost ansible_connection=local ansible_python_interpreter=/usr/bin/python2 gitlab_hostname=<Private_IP_1> 
    gitlab_password=<gitlab_password> port_1=<PORT_NUMBER>
    

    Repeat the same change for all the execution nodes.

  7. In the config file for loadbalancer at deploy/configs/load_balancer, you need to add all the details for the additional nodes to be deployed on Machine 2. The Nodes array should be updated as,

    {
        "Nodes": [
          {
              "role": "execution_node",
              "hostname": "<Public_IP_1>",
              "port": "8082"
          }, 
          ...,
          {
              "role": "execution_node",
              "hostname": "<Public_IP_2>",
              "port": "8083"
          },...
          
        ], ...
        other details 
    } 
    
  8. Now, run the setup script,
    bash ./setup.sh

  9. To install 5 execution nodes, you can comment (put a # at the start of each line) lines 18 to 187 of playbook-single.yml.
    Now run the playbook using the command,
    sudo ansible-playbook -i inventory playbook-single.yml --ask-sudo-pass

  10. If you want additional nodes, follow step 16 of Machine 1 Installation.

  11. Add cron job to restart the autolab components automatically. You need to do this on both the machines.

    1. Login as root
      sudo su
    2. Copy restart script to root account on both the machines
      cp deploy/autolab-restart.sh /root/
    3. Open crontab editor as root user
      crontab -e
    4. Enter a job in the file to run the restart script every minute.
      * * * * * bash /root/autolab-restart.sh

If the all of the above steps execute successfully, the installation is complete.

Installation creates containers for main server, load balancer, execution node, data base and Gitlab. For further information on managing these containers see container maintenance page.


In case of installation failure due to incorrect configuration, please see the FAQ page to see if you can resolve the error.

  1. If not, you can, follow the Uninstall Guide, to uninstall AutolabJS from Machine-1.

  2. For uninstalling on Machine-2, run the following steps:
    For removing all execution node containers,
    sudo docker stop execution-node-<Public_IP_2>-<Port_Number>
    sudo docker rm execution-node-<Public_IP_2>-<Port_Number>
    These two commands have to be run for all the execution node on Machine-2.
    Then, at the end, to remove the execution_node image,
    sudo docker rmi execution_node
    To remove the AutolabJS directory,
    sudo rm -rf /opt/autolabjs/

  3. You can then follow all the steps again to install AutolabJS.

Clone this wiki locally
You can’t perform that action at this time.