hulqBASH

hulqBASH: Hosted Universal Language Qualifier in BASH

Description

A source code evaluation REST service that downloads two files from the S3 bucket used for AES, parses out the command line argument sets and test configuration from the solution file (in the format specified below), then executes them against each other through bash scripts.

Evaluation

We are hosting an independent service (on a separate EC2 instance) to compile, execute, and evaluate source code.

Languages Currently Supported: Java(8)[.java], C[.c], C++[.cpp], C#[.cs]

Uploads

An S3 bucket in Amazon uses a key instead of a filename to reference a specific file.

Code snippet template should be named with the key = template_templateId in the database.

• This file is an incomplete source file (to be provided to the individual taking the assessment).

Code snippet solution should be named with the key = solution_templateId in the database.

• This file is a complete source file (to be used for evaluation, containing command line argument sets).

Config

This should be done within the same comment block as the arguments (see below)

keyfilename AND testfilename MUST be defined

Use the following schema to define configuration for evaluation:

  @Config
  mathmode:false
  trimwhitespace:false
  extractwhitespace:false
  caseignore:false
  grosserrormargin:0.0
  caseerrormargin:0.0
  keyfilename:key.cpp
  testfilename:test

mathmode(boolean): indicates that outputs should be numeric, grading will be relative difference of the key output, candidate will recieve 100% if key output is non numeric, 0% if candidate output is non numeric

values: true, false

default: false

trimwhitespace(boolean): indicates that whitespace should be trimmed from both ends of output (for both candidate and key)

values: true, false

default: false

extractwhitespace(boolean): indicates that whitespace should be removed from interior of output (for both candidate and key)

values: true, false

default: false

caseignore(boolean): indicates that case should be ignored for output (for both candidate and key)

values: true, false default: false

grosserrormargin(double): indicates the threshold of error of all executions for which a candidate can still receive a non zero grade (applies same logic as caseerrormargin to overall grade)

values: 0.0 - 1.0

default: 0.0

caseerrormargin(double): indicates the threshold of error for a single execution for which a candidate can still receive a non zero grade (ex: key result = 4, candidate result = 5, caseerrormargin:0.90, grade = 0%)

values: 0.0 - 1.0

default: 0.0

keyfilename(string): the name that the key file needs to have in order to be compiled and executed MUST include file extension

testfilename(string): the name that the candidates file should be named in order to be compiled and executed MUST NOT include file extension

Solutions

Use the following schema to define command line arguments (note: individual command line arguments cannot contain white space, single quotes, double quotes, or back-ticks). The comment block should be in the solution file.

Also, for the purposes of grading, only the OUTPUT of a file will be compared to the output of the solution for grading. Thus, the solution file MUST include a system.out.println(), << cout, printf(), etc. And the prompt should indicate that the submitted file must include the same for grading purposes.

/*
  @ArgSet(1)
  argument1 % Comment about argument1
  argument2 % Comment about argument2
  argument3 % Comment about argument3
  argument4 % Comment about argument4

  @ArgSet(2)
  argument1 % Comment about argument1
  argument2 % Comment about argument2

  @ArgSet(3)
  argument1 % Comment about argument1
  argument2 % Comment about argument2
  argument3 % Comment about argument3
*/

Each @ArgSet(X) defines a test case. For example, if we want to test a Palindrome class, we would use

java Palindrome argument1 argument2 argument3 argument4

for ArgSet(1),

java Palindrome argument1 argument2

for ArgSet(2), and

java Palindrome argument1 argument2 argument3

for ArgSet(3).

---

Full Example

public class Palindrome
{
    public static boolean isPalindrome(String s)
    {
        if(s.length()==0)
            return false;
        else if(s.length()==1)
            return true;
        else if(s.length()==2)
        {
            if(s.charAt(0)==s.charAt(1))
                return true;
            else
                return false;
        }
        else if(s.charAt(0)==s.charAt(s.length()-1))
            return palindrome(s.substring(1,s.length()-1));
        else
            return false;
    }

    public static void main(String[] args)
    {
        System.out.println(isPalindrome(args[0]));
    }
}

/*
  @Config
  mathmode:false
  trimwhitespace:false
  extractwhitespace:false
  caseignore:false
  grosserrormargin:0.0
  caseerrormargin:0.0
  keyfilename:key.cpp
  testfilename:test

  @ArgSet(1)
  racecar % true

  @ArgSet(2)
  thisisnotapalindrome % false
  bob % this second argument is ignored

  @ArgSet(3)
  anna % true

  @ArgSet(4)
  raceCar % false (capitalization)
*/

Execution:

> java Palindrome racecar
true
> java Palindrome thisisnotapalindrome bob
false
> java Palindrome anna
true
> java Palindrome raceCar
false

Implementing Additional Languages:

hulqBASH v2 includes several major improvements, the most important of which is the modularity of language implementation. The BASH source implements config files for each language, and includes a readme file that enumerates the specifications for adding new configurations to the system.