Skip to content

v0.2.1 Creating Labs for IO Tests with Source Code

Prasad Talasila edited this page Sep 8, 2017 · 2 revisions


  1. Here you are setting up the evaluation for HackerRank kind of I/O equivalence testing.

  2. Please have your testcases, ie, inputs and outputs ready before you begin the solution setup on AutolabJS.
    Name your testcases as per the following convention.

    Test Number Input file name Output file name
    Test1 input1.txt output1.txt
    Test2 input2.txt output2.txt
    Test3 input3.txt output3.txt

Setup Solution

Setup of a matching gitlab repository is mandatory before a lab evaluation is setup on AutolabJS. Please follow the steps given below to create and setup a lab solution on gitlab.

  1. Gitlab Login
    Login to gitlab as lab_author. In case you have not created an account with username of lab_author before, please do so before going to the next step.

  2. Create repository
    Create a gitlab repository for the lab. You can create a new gitlab repository by clicking on New Project button. After successful creation of a project repository, Gitlab creates a URL for the project.
    For example, lab1 of lab_author hosted at shall have a URL of

  3. Clone repository
    Clone the gitlab repository onto your local machine.

    $ export GIT_SSL_NO_VERIFY=1
    $ git clone

    This step clones the gitlab repository into lab1 directory of the present working directory.

  4. Copy evaluation skeleton
    You can copy the contents of sample directory structure in io_tests into your repository. Remove the student_solution sub-directory and the READMEs. A brief explanation of the contents of the directory follows.

     test_cases/                 ---> contains testcases to be run against submitted code
         /checks                 ---> any sample input / output text files to be used for evaluation
         /c                      ---> test cases for C language (DON'T CHANGE THIS)
         /cpp                    ---> test cases for C++ language (DON'T CHANGE THIS)
         /cpp14                  ---> test cases for C++ 2014 language (DON'T CHANGE THIS)
         /python2                ---> test cases for Python2 language (DON'T CHANGE THIS)
         /python3                ---> test cases for Python3 language (DON'T CHANGE THIS)
         /java/                  ---> test cases for Java language (DON'T CHANGE THIS)
             /setup              ---> compile, execute and test-setup bash shell scripts (DON'T CHANGE THIS)
             ---> script to compile the code for each test (DON'T CHANGE THIS)
         ---> script to execute the code for each test (DON'T CHANGE THIS)
               ---> script to complete the tests (calls, (DON'T CHANGE THIS)
     (one language-specific sub-directory exists under test_cases)
                  ---> orchestrating script for one submission (DON'T CHANGE THIS)
     test_info.txt               ---> text file containing an enumeration of all tests to be run

    You can evaluate C, C++ 2011, C++ 2014, Python2, and Python3 languages.

  5. Replace I/O Files
    Delete the contents of test_cases/checks directory and place your input and output text files there. These files must strictly follow the naming conventions suggested at the top of the page.

  6. Evaluation Setup
    Apart from the above directory structure, when the evaluation happens three more directories are created by AutolabJS. They are:

    student_solution        ---> the submitted code gets copied here
    working_dir             ---> a temporary directory used during evaluation
    results                 ---> results of evaluation are stored in this directory
        /log.txt            ---> evaluation log is in this file
        /scores.txt         ---> scores received for all the test cases
        /comments.txt       ---> comments received for all the test cases
  7. Create List of Tests
    Copy test_info.txt from io_tests and modify to suit your needs. Each line of test_info.txt has two fields separated by a tab space. A sample test_info.txt with two tests is shown below.

    Test1	1
    Test2	1

    The first field corresponds to name of the test and the second field corresponds to time limit to run the test. The last line ends without any new line character.

  8. Additional Tests
    You can run as many tests as you want by placing the matching input, output files in test_cases/checks and adding the corresponding line in test_info.txt.

  9. save and export the solutions to gitlab
    Make a commit in the local repository and push the changes to the gitlab.

    $ git add .
    $ git commit -m 'instructor's solutions'
    $ git push origin master
  10. Follow the details provided on Manage Course page to add the lab to website.

Clone this wiki locally
You can’t perform that action at this time.