Clean up the details of static validators

[evouga]

Addresses #329

I've also cleaned up the scoring semantics of the static validator to align with the output validator: the default score is computed via maximum score inference from the parent test group, and this score can be overridden using score.txt or score_multiplier.txt as normal.

[niemela]

The "Static Validators" section really should not be after "Verdict/Score Aggregation". The logical places would be before "Input Validators" or after "Output Visualizer". I think I prefer the former.

@evouga Does it makes sense to make that change in this PR (I think so)?

[niemela]

The order here seems strange:

input_validators
input_visualizer
output_validator
static_validator
output_visualizer

I would suggest:

static_validator
input_validators
input_visualizer
output_validator
output_visualizer

or possibly:

input_validators
input_visualizer
output_validator
output_visualizer
static_validator

(But I think the former is better)

[evouga]

The static validator is much closer in function to the output validator (which judges contestant submissions) than to the input validator (which checks the judge data only).

[evouga]

I can see an argument for putting the static validator right before the output validator, but I prefer the current ordering as it lets us say that the static validator shares the judgement semantics of the output validator.

[thorehusfeldt]

Here’s a categorisation by problem life cycle:

Output validation (statically or not) belongs to submission-time. It is performed when a solver submission is validated.

Input validation belongs to installation-time of a problem. It is performed when a problem is installed on a judge system.

Visualisation belongs to problem’s construction-time. It is performed when the problem is designed. (Like input generation.)

[niemela]

I can see an argument for putting the static validator right before the output validator, but I prefer the current ordering as it lets us say that the static validator shares the judgement semantics of the output validator.

It feels strange to me to group input_validators and input_visualizer together, but then put static_validator between output_validator and output_visualizer.

The most logical order of the validators would be input_validators (mainly install time), static_validator (run-time, but in some sense "early" run-time), an then output_validator ("later" run-time). If we want to group the visualizers with "their" validators we would get:

input_validators
input_visualizer
static_validator
output_validator
output_visualizer

But actually... I think the logical grouping would be to have all the validators together and all the visualizers together. visualizers belong to judge-time, which is after run-time, so:

input_validators
static_validator
output_validator
input_visualizer
output_visualizer

I think this would be the best order?

[evouga]

Let us discuss this in a separate issue?

The remaining issue has been broken out into #383.

[niemela]

@evouga I think this is ready to merge, but it needs to be rebased.