Score Calculators

Internal term score calculator denotes algorithm, which is responsible for computing overall score (correctness) of a submitted solution from the results of individual solution test. Each exercise is given a calculator name and associated calculator configuration (stored in DB text column as Yaml), which is used to initialize/prepare the calculator before it is used for computing the score.

We have implemented three calculators so far:

Uniform

Trivial algorithm, which computes the overall score as arithmetic average of individual test scores. This calculator has no configuration.

Weighted

Computes overall score as weighted average of individual test scores. The calculator expects that the weights are stored in the configuration in the following format:

testWeights:
    "Test 01": 200
    "Test 02": 300
    "Test 03": 100

The keys are test names (must correspond with the name column of the exercise_test) and the values has to be integers. When all weights are equal, the weighted calculator behaves exactly as uniform calculator.

Universal

The most complex calculator which was added last. It holds an AST (Abstract Syntax Tree) of an expression used to compute the score in the configuration.

The AST is encoded directly into Yaml structure -- each node is represented as a collection (object). A node must always have property type, which holds a string identifier of the node type (types of nodes are below). Internal nodes also have children property, which holds a list (array) of child nodes. Leaf nodes do not have children property.

Some nodes may have additional properties. Furthermore, a node is allowed to have arbitrary number of extension properties prefixed with x-. These properties are not recognized by API/backend, but they may be use by frontend (web app). The only reason they are recognized in the API is to ensure their correct (de)serialization. Other unknown properties are removed in the normalization process.

Leaf nodes (without children):

• value - a literal value, which holds one float (the value itself is stored in value property)
• test-result - reference to a test score, which is also a float between 0 and 1 (name of the test is in the test property)

Internal nodes (functions):

• sum - computes sum of all children values
• mul - computes product of all children values
• sub - binary node that computes a - b
• div - binary node that computes a / b (it yields 0 if b == 0)
• neg - unary node that negates its operand
• min - computes minimum of all children values
• max - computes maximum of all children values
• avg - computes arithmetic average of all children values
• clamp - unary node that clamps its operand value into [0,1] range

For practical reasons, we have decided to implement one optimization. Value literals may be also encoded as simple float numbers (instead of objects) in the child list. The value-typed objects and float literals are treated as equal; however, root node must be always an object (even when it is a literal).

Example

The following example should yield the same score as the weighted score config presented above.

type: div
children:
  - type: avg
    children:
      - type: mul
        children:
          - 2.0
          - type: test-result
            test: "Test 01"
      - type: mul
        children:
          - 3.0
          - type: test-result
            test: "Test 02"
      - type: test-result
        test: "Test 03"
  - 6.0