Skip to content

v0.4.0 Administrator FAQs

AnkshitJain edited this page Sep 29, 2018 · 1 revision

General

  1. How many execution nodes do I need to run AutolabJS?
    Ans: We suggest that you calculate the required number of execution nodes as per the following formula.
    Required number of execution nodes = (4/3)*no of requests per second * evaluation time per request.
    In the given formula, the number of requests per second is the average number of evaluation requests submitted by the students to AutolabJS.
    The default installation setup installs five execution nodes. If you need more execution nodes, please follow the instructions given under the Execution Nodes FAQ of this page for adding more execution nodes. Note: There is a limit to the number of of nodes that can be run on a system without degrading the performance. This largely depends on the system configuration.

  2. How can I know the load (number of requests per second) for my lab?
    Ans: If you do not have the concrete numbers for the load, i.e. the number of requests per second and have some past data from a previous lab / course, then we can estimate the load. Please use the following commands to perform the load estimation.

    1. Obtain the docker logs of the main server for the time period that corresponds to one of the previous labs.
      $docker logs -t --since T1 --until T2 mainserver 2>/dev/null 1>msLogs.txt 
      # If the lab was conducted from 3:30 GMT to 6:20 GMT, then the command would be as follows
      $docker logs -t --since 2018-02-03T03:30 --until 2018-02-03T06:21 mainserver  2>/dev/null 1>msLogs.txt
    2. Use the UNIX text processing pipeline to extra the load distribution
      $grep 'Socket submission event triggered' msLogs.txt | sed 's/^.*T\([:0-9]\{8\}\).*$/\1/' | uniq -c | tr -s ' \t' | cut -d ' ' -f 2 | sort -n | uniq -c
      A sample result given by the text processing pipeline is shown below.
      520 1
       31 2
        3 3
      The first column corresponds to the number of times a certain load quantity has been received and the second column corresponds to the number of requests per second. In this example, the maximum load experienced was three requests per second. You can use this maximum load number to calculate the number of execution nodes required for the lab.
  3. Is there a way to see a time-series of evaluation requests per second received by AutolabJS from the participating students?
    Ans: On the main server log files, you can apply the following text processing pipeline.

    grep 'Socket submission event triggered' msLogs.txt | sed 's/^.*T\([:0-9]\{8\}\).*$/\1/' | uniq -c | more

    A sample output from the command is shown below.

    1 05:18:27    
    2 05:18:33    
    1 05:18:49    

    The first column represents the number of requests received and the second column represents the time slot (of one second duration) during which the requests were received.

  4. How can I estimate the evaluation time for my lab?
    Ans: You can issue two serial evaluation requests and go check for the execution node that received the evaluation requests recently. The same execution node evaluates two consecutive evaluation requests. From the logs of the execution node, find out the difference between the timestamps of the lines containing the phrase, "requestRun post request recieved." The time difference is approximately equal to the evaluation time of your lab.
    A sample log of an execution node is shown below.

    2018-02-03T04:05:10.769584805Z requestRun post request recieved
    2018-02-03T04:05:10.770103841Z { id_no: '2016A7PS0004G',
    2018-02-03T04:05:10.770136282Z   Lab_No: 'Lab4_1',
    2018-02-03T04:05:10.770145270Z   time: '2018-02-03 04:05:12',
    2018-02-03T04:05:10.770152344Z   commit: '',
    2018-02-03T04:05:10.770159698Z   status: 1,
    2018-02-03T04:05:10.770166125Z   penalty: 0,
    2018-02-03T04:05:10.770172680Z   socket: 'YxtkbC6757lUdOV-AAKT',
    2018-02-03T04:05:10.770179232Z   language: 'cpp' }
    2018-02-03T04:05:10.770185699Z bash extract_run.sh 2016A7PS0004G Lab4_1 autolab.
    bits-goa.ac.in "" cpp
    2018-02-03T04:05:21.779266551Z requestRun post request recieved

    The time difference between the two "requestRun" log lines is approximately 11 seconds. This time difference is the approximate evaluation time per request.

Installation

  1. The configuration file seems to be full of IP addresses. Does the Autolab support domain names (DNS)?
    Ans: The domain names are supported and are in fact the preferred way of specifying the hostnames in all the configuration files.

  2. I tried to run the Ansible playbook twice and got the database error shown below.
    TASK [Create Autolab database] *************************************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["docker", "exec", "autolab-db", "bash", "-c", "mysql -u root -proot -e 'CREATE DATABASE Autolab'"], "delta": "0:00:01.447437", "end": "2016-11-28 02:28:17.562664", "failed": true, "rc": 1, "start": "2016-11-28 02:28:16.115227", "stderr": "mysql: [Warning] Using a password on the command line interface can be insecure.\nERROR 1007 (HY000) at line 1: Can't create database 'Autolab'; database exists", "stdout": "", "stdout_lines": [], "warnings": []}

    What should I do now?
    Ans: Comment (Please put a # symbol at the beginning of each line) lines 41 and 42 of playbook.yml / playbook-single.yml file. Now run the installation command again. That should install the Autolab setup properly.
    Another solution is to look at deleting an existing Autolab database completely. Please understand that choosing this option leads to data loss. Given that caveat, you can try the following recipe.

    $sudo docker exec -it autolab-db bash
    #mysql -u root -p          #on prompt, enter the DB root password set in "inventory" file
    mysql>DROP DATABASE Autolab;
    mysql>quit
    #exit
  3. I got the installation error with Gitlab server with the error shown below.
    TASK [Waiting for Unicorn on GitLab server to be up] ***************************
    fatal: [13.65.36.224]: FAILED! => {"changed": true, "cmd": "docker exec gitlab bash -c \"/wait-for-it.sh -t 300 127.0.0.1:8080\"", "delta": "0:05:00.112174", "end": "2016-11-28 02:17:04.006885", "failed": true, "rc": 124, "start": "2016-11-28 02:12:03.894711", "stderr": "wait-for-it.sh: waiting 300 seconds for 127.0.0.1:8080\nwait-for-it.sh: timeout occurred after waiting 300 seconds for 127.0.0.1:8080", "stdout": "", "stdout_lines": [], "warnings": []}
        to retry, use: --limit @/home/autolab/JavaAutolab/deploy/playbook.retry

    What should I do now?
    Ans: Comment (Please put a # symbol at the beginning of each line) lines 41 and 42 of playbook.yml / playbook-single.yml file. Now run the installation command again. That should install the Autolab setup properly.
    If this still fails, look at the Gitlab container status by using the command
    sudo docker ps -a
    If the Gitlab container gitlab has unhealthy in the STATUS column, restart the container using
    sudo docker restart gitlab
    and restart the installation.
    If the installation still fails with the same message, please verify the Gitlab configuration in all the config file as given in the Installation guide. Then remove the container using the commands
    sudo docker stop gitlab
    sudo docker rm gitlab
    and then restart the installation.

  4. I made a configuration mistake in one of the containers (i.e., mainserver, loadbalancer or execution nodes) and that container is restarting. What should I do now?
    Ans: First stop the container using the following command
    sudo docker stop <container_name>
    Correct the configuration in the configuration files available in the following locations.
    mainserver --> /opt/autolabjs/deploy/configs/main_server/.json
    loadbalancer --> /opt/autolabjs/deploy/configs/load_balancer/
    .json
    execution_nodes... --> /opt/autolabjs/deploy/configs/execution_nodes/execution_node_1/*.json
    AND then restart the respective container.

  5. I got an error during installation. As per the suggestions on the installation page, I ran uninstall.yml and tried to install again. I get an error saying "Can't create database 'Autolab'". What should I do?
    Ans: Please run the following commands and try the installation procedure again.

    sudo rm -rf /opt/autolabjs

    If you changed the default installation directory, please force remove that directory and try the installation procedure again.

  6. I am getting installation error during installation of one component, but the uninstall script removes all the successfully installed components as well. Is there any way to run the installation playbook from the point of failure instead of from the very beginning?
    Ans: You can open the playbook file and comment the lines corresponding to the components that have been successfully installed; you can comment a line in the yaml file by putting # symbol at the beginning of the line. You can clearly identify the beginning of a component by looking for hosts: <component_name>. You can comment out lines till the end of the component, i.e., one line before the next hosts: <component_name> line. When you run the playbook again, only the remaining components get installed.

  7. What kind of pathnames can I use in volume mappings of containers?
    Ans: You can use absolute or relative pathnames. You are advised not to use the ~ symbol as a shortcut for home directory of a user.

Load Balancer

  1. My evaluation requests get stuck forever. Upon checking the load balancer logs, I see socket connection refused message. Why?
    Ans: The scores.json file of the execution nodes are not configured properly. Please correctly set the hostname and port parameters in scores.json for each of the execution nodes. In addition, view the status of all the execution nodes and restart any stopped nodes.

Execution Nodes

  1. How do I add more execution nodes?
    Ans: We have not automated the process for this step. In general the outline for adding extra execution nodes is as follows. Note that if you changed the default installation directory, you will have to make the necessary changes in place of /opt/autolabjs in all the paths.

    1. Login to the machine running the execution node. In a two machine setup, this would be Machine 2. In a single machine setup, the steps would be followed on the same machine.

    2. Change to root user.
      sudo su

    3. Make a copy of execution node files.
      cp /opt/autolabjs/execution_nodes/execution_node_1 /opt/autolabjs/execution_nodes/execution_node_6

    4. Change to configs directory.
      cd /opt/autolabjs/deploy/configs/execution_nodes (for two machine case, this path would be on machine-2)

    5. Copy the existing settings of an execution node into new directory, say execution_node_6.
      cp -r execution_node_1 execution_node_6

    6. Change the host port in execution_node_6/conf.json to one higher, non-used port, say 8087.

    7. Set the hostname and port parameters in execution_node_6/scores.json appropriately. The hostname field is the IP address of the machine on which the execution node container is to be run. The port field needs to be one higher, non-used port, say 8087.

    8. Change to deploy directory.

        cd ../../
      
    9. Update the file single_machine in deploy directory to reflect the new node. Instructions for the same can be found in the Step 4 of installation.

    10. Run the command to install the execution node.

      ansible-playbook -l <new_en_name> -i single_machine playbook.yml
      
    11. Exit the super user account.
      exit

    12. Update the load balancer configuration to add new execution node.

      "Nodes": [
          {
            "hostname": "10.0.0.5",
            "port": "8082"
          },
          {
            "hostname": "10.0.0.5",
            "port": "8087"
          }
      .......rest of configuration file continues
      
    13. Restart the load balancer.
      sudo ansible -i inventory loadbalancer --become --ask-become-pass -m docker_container -a "name=loadbalancer state=started restart=yes"

  2. What is the role of scores.json configuration file?
    Ans: Execution node uses json object format specified in scores.json to configure itself. Please set the hostname and port of each execution node properly.

gitlab

  1. How do I change my gitlab server IP address?
    Ans: First, make sure that your gitlab container is up and running. Then follow these steps:
    $sudo docker stop gitlab
    $sudo docker rm gitlab
    $sudo docker run --detach --hostname gitlab.example.com -e GITLAB_ROOT_PASSWORD="<pwd>" -e GITLAB_OMNIBUS_CONFIG="external_url 'https://<new_ip>'; nginx['redirect_http_to_https'] = true;" --publish 443:443 --publish 80:80 --publish 22:22 --name gitlab --restart always --volume /etc/localtime:/etc/localtime:ro --volume /opt/autolabjs/gitlab/config:/etc/gitlab --volume /opt/autolabjs/gitlab/logs:/var/log/gitlab --volume /opt/autolabjs/gitlab/data:/var/opt/gitlab --volume /opt/autolabjs/keys/gitlab/ssl/localhost.key:/etc/gitlab/ssl/<inventory_hostname>.key --volume /opt/autolabjs/keys/gitlab/ssl/localhost.crt:/etc/gitlab/ssl/<inventory_hostname>.crt gitlab/gitlab-ce:latest
    $sudo docker exec -it gitlab bash
    #gitlab-ctl reconfigure
    #gitlab-ctl restart

    Change the <pwd> and <new_ip> variables to suit your scenario.
    
  2. I am not able to send / receive forgotten password emails from gitlab. What should I do?
    Ans: You need to enable SMTP settings in Gitlab. Modify the /etc/gitlab/gitlab.rb file in gitlab container as per the instructions available on the gitlab smtp settings page.
    Reconfigure gitlab using the command
    gitlab-ctl reconfigure
    send a test mail to yourself by using the following commands
    gitlab-rails console
    From gitlab ruby terminal, type
    irb(main):003:0> Notify.test_email('<your_mail_id>', 'Message Subject', 'Message Body').deliver_now
    Most mail providers like Gmail and Yahoo need modifications to settings in their mail accounts from more secure to less secure in order for the mail account to be used for sending gitlab email.

  3. I forgot the root password of gitlab. Is there a way to recover the same?
    Ans: You can recover the password using instructions from GitLab.

  4. How do I disable groups feature for regular users?
    Ans: Please see the groups documentation available on gitlab.

Clone this wiki locally
You can’t perform that action at this time.