Finals competition flow

This section gives a detailed explanation of the finals competition flow.

Contents

1. Timeline
2. Overview
3. Scoring details
4. How to submit
5. FAQs
   1. Have any of the evals changed?

Timeline

Prep @ Home (Jun 1 - 9): remote hardware testing, additional model training
Day 1 @ MBS (Jun 10): in-person testing, Surprise Challenge
Day 2 @ MBS (Jun 11): Semi-Finals and Finals matches, revealing the winner of the Surprise Challenge

Overview

An individual match of the competition takes place over 3 distinct phases:

1. NLP corpus load — each team is given the NLP corpus to load.
2. Noise — each team noises a chunk of the CV data, which is later given to other participants as missions. No time penalty is applied, but there is a time limit for responses.
3. AE — iteratively running AE and missions concurrently until all 200 steps of the AE game are complete.

There are individual time limits for each step within each phase; you can see the details of this by examining the code in test_competition_server.

Phase	Task	Time limit	Score limit
1	NLP corpus load	60s	-
2	Noise	10s per batch of 4	-
3	ASR, CV, NLP	10s per batch of 4	5s
3	AE	2s	-

For each match, all of your team's containers (and thus, models) will be loaded simultaneously on one desktop machine with the hardware specifications listed here, along with a container for your orchestration server (we call this the "participant server") that will handle routing the calls to individual models on one machine using HTTP request. Your participant server will also be responsible for responding to the competition server, which runs the AE environment and handles task evaluation. We provide a boilerplate participant server and docker compose in til-26-finals; should this setup not work for you, feel free to change it. You are expected to submit your participant server as a Docker image, see til-26-finals for submission instructions; you are not required to submit a custom docker compose.

Scoring details

Scores for individual missions are averaged across each task to get an overall % value, and applied as a multiplier to your AE score. As in the qualifiers, they are weighted 75% accuracy 25% speed. For each individual batch of 4, the timeout is 10s, and the maximum time for scoring purposes is 5s; that is, if you take 5s or more to return a batch of 4, your speed score will be 0.

As an example, if you scored 400 in the AE challenge and achieved a mean mission multiplier of 75%, your final score for the purposes of the challenge would be 300.

How to submit

Pull an updated version of til-26-finals. We provide a utility function for til that should work automatically on your instance if you run til pull-finals. Otherwise, or if the utility function doesn't work, do the following:

git submodule update --init
cd til-26-finals
git checkout main
git pull

Then reference the README.md there for further instructions.

Ensure to run testing both on your local machine and remotely on the competition hardware. Note that we only have one machine and it has very mediocre Wi-Fi, so the testing queue will likely be rather slow. It will also frequently go down as we use it for other things, and as we discover the newfangled ways in which experimental infrastructure will inevitably break. The real testing will be done on Day 1 at MBS.

FAQs

This section will be updated as other questions come in. If your question isn't answered, ping @Tech on Discord in the #semifinalist channel (or your team's channel if you're unsure if it's a private concern).

Have any of the evals changed?

Slightly. The full changes are documented below:

• ASR: no change.
• CV: no change.
• Noise: no change.
• NLP: one line change in the tokenizer truncation behaviour. See updated logic in the test_nlp.py script.
• AE: to reduce variance, AE advanced is now seeded and some sources of non-determinism were removed. We may end up changing the models deployed to the AE eval as well, so teams can get a better sense of how their AE system performs against other models rather than the existing BenBots™.

Remember that these evals are no longer relevant to your scoring at all, as the leaderboard is frozen. They are just potential indications of how your model may fare on non-training data. You will need other ways of evaluating your model performance.