Skip to content

Functional Tests

AnkshitJain edited this page Mar 21, 2018 · 9 revisions

Functional Tests

Functional testing is primarily used to verify that a piece of software is providing the same output as required by the end-user.

The main file for running functional tests is located at tests/functional_tests/test.sh.
The extract_run_test.sh is rigged to emulate the behavior of Gitlab clone and evaluates the submission.

The structure for functional tests is shown below:

  tests/

    functional_tests/
      data/
      helper_scripts/
      shell/
      bats/
      env.conf
      test.sh
      ...other files

    test_modules/
      data/
      helper_scripts/
      bats/ 
      ...test files

The files in test_modules are common to both deployment and functional tests. The same conventions and directory structure is to be used in both the directories.

Conventions for Writing Functional Tests

  1. The functional tests use BATS testing module and shell scripts.
  2. Every test category has its own .bats file. For example: scoreboard.bats.
  3. All the tests are run in the /tmp directory.
  4. A subdirectory is created for each subsection of tests in the /tmp directory.
    For example: for running functional tests for scoreboard, we create a directory named scoreboard.
    Hence, the temporary directory for scoreboard tests is /tmp/scoreboard.
  5. The setup of a test copies the necessary files to/from the required location, and creates the necessary temporary directory for the tests. For example: scoreboard tests setup
  6. The teardown of a test reverses any change made during the test and deletes the temporary directory for the tests. For example: scoreboard tests teardown
  7. The evaluation requests are sent to AutolabJS server using the submit.js script.
  8. The submission requests, which include the student solutions and the lab author solution is copied to the temporary test directory from the directory docs/examples. Here, either of unit_tests orio_tests are copied during the setup phase of a test.
  9. All additional data required for the tests are located in a subdirectory named data. For example, data for scoreboard tests is located in tests/functional_tests/data/scoreboard directory.
  10. Similarly, if any helper scripts are needed for a test, they are located in a directory called helper_scripts.
    For example, helper scripts for scoreboard tests is located in tests/functional_tests/helper_scripts/scoreboard directory.

Writing a New Functional Test

  1. Firstly, follow all the conventions above.
  2. Have a look at writing tests for BATS
  3. If you have a bats test file, follow this step. In case you only have a shell script as a test file go to step 6. To start with, say you have a file named abc.bats, which has the tests for verifying abc functionality.
  4. We put this file in the directory, tests/functional_tests/bats/ directory. Hence we now have a file at tests/functional_tests/bats/abc.bats.
  5. Write a shell script that calls this bats file. All the setup before running the bats tests should be done in this shell script before running the bats test. Remember, if the setup changes anything, the same must be reversed back in the teardown section, i.e. after calling the bats script.
  6. To call the bats script, use the following command:
    > $BATS abc.bats
    
    Here, $BATS is already declared as an environment variable.
  7. Place this shell script in the shell directory, with the name abc.sh. Remember, the filename for bats file and shell script file, along with the directory names for the data and helper_scripts should be same. Hence we now have a file at tests/functional_tests/shell/abc.sh.
  8. The test_runner test.sh at tests/functional_tests/test.sh will automatically discover this file and run the
    test.

The following diagram summarizes the above explanation, regarding the placement of files and also gives an existing example for the same.

functional_tests
|
|--------> bats
|          |--------> abc.bats
|--------> shell
|          |--------> abc.sh
|--------> data
|          |--------> abc
|          |           |--------> {data files for tests for abc module}
|--------> helper_scripts
|          |--------> abc
|          |           |--------> abc_setup.sh
|          |           |--------> abc_teardown.sh
|          |           |--------> {other helper_scripts for tests for abc module}

An example of this structure is: scoreboard tests.

Shell File -> tests/functional_tests/shell/scoreboard.sh

Bats File -> tests/test_modules/bats/scoreboard.bats

Setup -> tests/test_modules/helper_scripts/scoreboard/scoreboard_test_setup.sh

Teardown -> tests/test_modules/helper_scripts/scoreboard/scoreboard_test_teardown.sh

Helper Scripts Directory -> tests/test_modules/helper_scripts/scoreboard

Data Directory -> tests/test_modules/data/scoreboard

Note: Since we have scoreboard tests to be used both for functional and deployment tests, we have used files from test_modules directory.

Running Functional Tests

Functional tests can be run by a single script located at tests/functional_tests/test.sh. The functional tests can also be run using the below command from the AutolabJS home directory.

> TEST_TYPE="functional_tests" bash tests/test_runner.sh     

Call Graph of Functional Tests

The detailed call graph for functional tests is shown below. All filenames and paths are with respect to the tests/ directory. All the necessary directory setup and dependencies are installed by install.sh.

functional_tests/test.sh (runs all the functional tests)
  | creates 
  |-------> process_pid.txt (stores the process pids of MS, LB and EN components)
  | uses
  |-------> functional_tests/env.conf
  | runs
  |-------> functional_tests/helper_scripts/setup.sh
  |
  |-------> {launches MS, LB and EN components and preserves their process IDs in process_pid.txt}
  | runs
  |-------> {discovers tests in functional_tests/shell/ directory and runs them}
  | runs
  |-------> functional_tests/helper_scripts/teardown.sh
  |         | deletes 
  |         |-------> process_pid.txt 

The tests discovered by functional_tests/test.sh shell script are as follows.

functional_tests/test.sh (runs all the functional tests)
  | runs
  |-------> functional_tests/shell/website_load.sh
  |         | runs
  |         |-------> test_modules/bats/website_load.bats
  |         |         | uses
  |         |         |-------> test_modules/data/website_load/index.html
  |         |         | uses
  |         |         |-------> test_modules/data/website_load/admin.html
  |         |         | uses
  |         |         |-------> test_modules/data/website_load/status.txt
  |         |         | uses
  |         |         |-------> test_modules/data/website_load/js/userLogic.js
  |         |         | uses
  |         |         |-------> test_modules/data/website_load/js/adminLogic.js
  |         |         | uses
  |         |         |-------> test_modules/data/website_load/js/package.json
  |         |         | uses
  |         |         |-------> test_modules/data/website_load/css/icon.css
  |         |         | uses
  |         |         |-------> test_modules/data/website_load/css/style.css
  |         |         | uses
  |         |         |-------> test_modules/helper_scripts/website_load/status_check.sh
  |
  | runs
  |-------> functional_tests/shell/unit_tests.sh
  |         | runs
  |         |-------> test_modules/bats/unit_tests.bats
  |         |         | uses
  |         |         |-------> test_modules/data/unit_tests/c_unsupported.txt
  |         |         | uses
  |         |         |-------> test_modules/data/unit_tests/test_result.txt
  |
  | runs
  |-------> functional_tests/shell/io_tests.sh
  |         | runs
  |         |-------> test_modules/bats/io_tests.bats
  |         |         | uses
  |         |         |-------> test_modules/data/io_tests/test_result.txt
  |
  | runs
  |-------> functional_tests/shell/scoreboard.sh
  |         | runs
  |         |-------> helper_scripts/scoreboard/scoreboard_test_setup.sh
  |         | runs
  |         |-------> test_modules/bats/scoreboard.bats
  |         |         | uses
  |         |         |-------> test_modules/data/scoreboard/15charID.json
  |         |         | uses
  |         |         |-------> test_modules/data/scoreboard/20charID.json
  |         |         | uses
  |         |         |-------> test_modules/data/scoreboard/30charID.json
  |         |         | uses
  |         |         |-------> test_modules/data/scoreboard/BuyerMistake.cpp
  |         |         | uses
  |         |         |-------> test_modules/data/scoreboard/empty.json
  |         |         | uses
  |         |         |-------> test_modules/data/scoreboard/first_eval.json
  |         |         | uses
  |         |         |-------> test_modules/data/scoreboard/second_eval.json
  |         |         | uses
  |         |         |-------> test_modules/data/scoreboard/labs.json
  |         |         | uses
  |         |         |-------> test_modules/data/scoreboard/savecode.sh
  |         | runs
  |         |-------> helper_scripts/scoreboard/scoreboard_test_teardown.sh
  |
  | runs
  |-------> functional_tests/shell/missing_files.sh
  |         | runs
  |         |-------> functional_tests/bats/missing_files.bats
  |         |         | uses
  |         |         |-------> functional_tests/data/missing_files/author_student_repository.txt
  |         |         | uses
  |         |         |-------> functional_tests/data/missing_files/no_result_found.txt
  |         |         | uses
  |         |         |-------> functional_tests/data/scoreboard/no_test_info_found.txt
  |
  | runs
  |-------> functional_tests/shell/execution_node.sh
  |         | runs
  |         |-------> test_modules/bats/execution_node.bats
  |         |         | uses
  |         |         |-------> test_modules/helper_scripts/execution_node/concurrent_eval.sh
  |
  | runs
  |-------> functional_tests/shell/evaluation_logs.sh
  |         | runs
  |         |-------> functional_tests/bats/evaluation_logs.bats
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/compilation_error.cpp
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/input1.txt
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/output1.txt
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/Solution_Compilation.java
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/Solution_Exception.java
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/test_info.txt
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/no_log.txt
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/log_greater_than_25.txt
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/log_greater_than_50.txt
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/shellOut_greater_than_25.txt
  |         |         | uses
  |         |         |-------> functional_tests/data/evaluation_logs/Test.sh
  |
  | runs
  |-------> functional_tests/shell/socket_events.sh
  |         | runs
  |         |-------> functional_tests/bats/socket_events.bats
Clone this wiki locally
You can’t perform that action at this time.