How does ForecastBench work?

ForecastBench nightly automated workflow

Below we describe how the Question Bank is updated, Question Sets are created, and Forecasts processed. For more info, see Section 3 of the ForecastBench paper.

The code in src/nightly_update_workflow orchestrates the nightly updates. src/nightly_update_workflow/manager manages the code that can run in parallel and blocks when it needs to wait for certain jobs to finish before proceeding. The code in src/nightly_update_workflow/worker takes care of actually running the Cloud Run jobs in parallel when possible.

Nightly at 0:00 UTC

At midnight UTC we pull new questions from our data sources and store them in the question bank along with the associated resolution values. for those questions. Once this process has completed, we update the relevant metadata

Update questions and resolutions

We pull data from our sources every night, beginning at 12AM UTC.

For every source that we get data from, we create a question file named <<source>>_questions.jsonl. See the data dictionary below for more details.

We also store data that is pertinent to the resolution of the questions contained in the aforementioned file. The way that data is stored is source-dependent. The basic resolution file data dictionary is below.

All files are stored as JSON Lines files (saved with the .jsonl extension).

⚠️ These files are not released publicly as this would be a violation of some of the data sources' licenses.

The code that produces these files is located in src/questions/<<source>> (see the questions folder here). These directories have both a fetch directory and an update_questions directory.

Question file data dictionary

{
    "id": {
        "description": "A unique identifier string for the source. Required.",
        "type": "string"
    },
    "question": {
        "description": "The question we're asking. It should be a Python f-string that contains the fields `forecast_due_date` and `forecast_evaluation_date`, which will be inserted during question curation. Required.",
        "type": "string"
    },
    "background": {
        "description": "The question background. Default: 'N/A'.",
        "type": "string"
    },
    "url": {
        "description": "The URL where the resolution value can be found. Default: 'N/A'.",
        "type": "string"
    },
    "resolved": {
        "description": "Whether or not the question was resolved. Default: 'N/A'.",
        "type": "boolean"
    },
    "forecast_horizons": {
        "description": "A list of integers representing the forecast horizons (in days) over which we want forecasts for this question. Default: 'N/A'.",
        "type": "array<number>"
    },
    "freeze_datetime_value": {
        "description": "The value of the question at the question set freeze datetime. Default: 'N/A'.",
        "type": "string"
    },
    "freeze_datetime_value_explanation": {
        "description": "Text explanation of what the freeze_datetime_value represents. Default: 'N/A'.",
        "type": "string"
    },
    "market_info_resolution_criteria": {
        "description": "The resolution criteria provided by the source. Default: 'N/A'.",
        "type": "string"
    },
    "market_info_open_datetime": {
        "description": "The ISO 8601 datetime in UTC the market began. e.g. 2022-05-02T05:00:00+00:00. Default: 'N/A'.",
        "type": "string"
    },
    "market_info_close_datetime": {
        "description": "The ISO 8601 datetime in UTC the market closes. e.g. 2022-05-02T05:00:00+00:00. Default: 'N/A'.",
        "type": "string"
    },
    "market_info_resolution_datetime": {
        "description": "The ISO 8601 datetime in UTC the market will resolve or was resolved. e.g. 2022-05-02T05:00:00+00:00. Default: 'N/A'.",
        "type": "string"
    }
}

Resolution file data dictionary

{
    "id": {
        "description": "A unique identifier string for the source. Required.",
        "type": "string"
    },
    "date": {
        "description": "The date for which the value is valid, formatted as YYYY-MM-DD. Required.",
        "type": "string"
    },
    "value": {
        "description": "The value for the given date, accounting for the entire day. Required.",
        "type": "string"
    }
}

Update Metadata

After we've updated the questions in the question bank, we run some checks before we include them in the set of questions to sample from to create the bi-weekly question set.

The output of this step is a file called question_metadata.jsonl. See the data dictionary below for details.

⚠️ This file is not released publicly as as it's meaningless without the corresponding <<source>>_questions.jsonl files which are not released publicly.

The code that produces these files is located in src/metadata. The code in src/metadata/tag_questions uses an LLM to categorize the question into one of the categories in the variable QUESTION_CATEGORIES in src/helpers/constants.py. The code in src/metadata/validate_questions uses an LLM to invalidate inappropriate questions (poorly phrased or dealing with topics we don't want to cover).

Processed Questions File Fields

{
    "source": {
        "description": "Source of data, e.g. manifold. Required.",
        "type": "string"
    },
    "id": {
        "description": "A unique identifier string for the source. Required.",
        "type": "string"
    },
    "category": {
        "description": "Question category assigned by LLM. Required.",
        "type": "string"
    },
    "valid_question": {
        "description": "Whether or not a question is valid as determined by an LLM. Required.",
        "type": "boolean"
    }
}

Resolve & Update Leaderboard

After we've updated the questions and resolution values in the question bank, we resolve all forecast sets and update the leaderboard.

The outputs of this step are html and csv files of the leaderboard, displayed on the website and saved in the forecastbench-datasets git repository.

The code that produces these files is located in src/resolve_forecasts and src/leaderboard.

Starting on 2024-07-21, every 2 weeks (Sunday) - 10 days at 0:00 UTC

Sampling questions

We select questions from the question bank to create question sets that LLMs and Humans will forecast from.

The question set will be generated on day $D-10$, where $D$ is the day all forecasts are due.

This step will create two JSON output files.

1. ISO_DATE-llm.json will contain $500$ forecast questions for LLMs
2. ISO_DATE-human.json will contain $200$ forecast questions for Humans

ISO_DATE is the date of day $D$ above.

On day $D-10$, the human survey is created from ISO_DATE-human.json and humans begin forecasting. On day $D$, the questions in ISO_DATE-llm.json is distributed to LLMs at 0:00 UTC via our GitHub dataset repository.

The soft link latest-llm.json points to the most recently-created question set. NB we only upload the human dataset when we run a human survey round, and only after all forecasts have been submitted.

Human and LLM forecasts are due by 23:59:59 UTC on day $D$.

We sample $500$ questions for LLMs and $200$ for humans. The questions for humans are a subset of the LLM standard questions (we don't select combination questions for humans).

We allocate questions as evenly as possible across the sources we have. From each source, we allocate evenly across categories available within that source. We select the same number of regular questions as we do combo questions, given the source's overall allocation.