Introduction

This repository contains python scripts for verifications of Metaculus and Community Predictions by using Tezos blockchain and TzStamp. See https://www.metaculus.com/tezos/ for mor information.

How to use this

Tested with python 3.8

• (optional) create a virtual environment and activate it via python -m venv .venv && source .venv/bin/active
• Install requirements.txt pip install -r requirements.txt
• Run python app.py

For example:

python app.py --question-id 1 --for-date 2022-01-20

How it works

Everyday we take all standing MP/CP and create Merkle Tree. Resulting root of the tree is timestamped on Tezos by TzStamp service. We publish Merkle Root on our website and tweet it using dedicated account.

Our simple verification interface allows you to generate audit trail and perform verification. You can go through the verification process yourself by following the tutorial below or using our example implementation.

Note that we are hashing only questions that are part of main Metaculus project https://www.metaculus.com/questions/?project=144. This excludess some tournament or private questions.

Verification

To verify existence of the prediction, here is what you need to do:

• Verify TzStamp proof
• Verify uniqueness of hash for a day
• Get prediction hash for a given day
• Perform Merkle Tree audit proof

1. Verify TzStamp proof

• download the proof from our website https://www.metaculus.com/tezos/ for the given day
• Go to "verify and stamp" section https://tzstamp.io/
• Enter Merkle Root into SHA-256 Hash and select Proof
• Click verify

Example response:

Verified!
Hash existed at 24/11/2021, 14:10:32
Block hash: BM4wrLDDanvhHkJynUkGGd2hEovYEHhvC32Rp48UVRyDG6HQwiW

2. Verify uniqueness of hash for a day

We tweet unique hash every day at https://twitter.com/tezos_forecasts

3. Get prediction hash for a given day

Find the question of the interest from https://www.metaculus.com/questions/?project=144 and click on the question you want to validate. Extract the question_id from the URL, it's the number following https://www.metaculus.com/questions/, so e.g. for https://www.metaculus.com/questions/5906/number-confirmed-global-covid-cases-by-2022/ the question_id is 5906. Then call api endpoint (date_str is in format YYYY-MM-DD)

https://www.metaculus.com/api2/questions/{question_id}/prediction-for-date/?date={date_str}

Then create hash (using python):

from hashlib import sha256
import json

prediction = {"question_id:CP": <forecast_values>}
data = json.dumps(prediction)
data = data.encode('utf-8')
hashed_prediction = sha256(data).hexdigest()

where the <forecast_values> is a dict (hashmap) with key-values ordered in the following order "y", "q1", "q2", "q3" followed by optional "low", "high" for some question types. So e.g.:

{"y": ..., "q1": ..., , "q2": ..., "q3": ..., "low": ..., "high": ...}

The JSON format is with a space after :, e.g. {"x": 3}. All numbers should use at maximum 5 decimal places.

4. Perform Merkle Tree audit proof

Once you your sha256 have hash, you need to perform Merkle Tree audit proof. You might start verification process by clicking "Start" in the above table. We will return you audit trail, to be used for verification.

Example python code to perform verification:

def compute_hash(data):
    return sha256(data.encode('utf-8')).hexdigest()

def verify_audit_trail(chunk_hash, audit_trail):
    """
    Performs the audit-proof from the audit_trail received
    from the trusted server.
    """
    proof_till_now = chunk_hash
    for node in audit_trail[:-1]: # remove last item (Merkle Root)
        node_hash = node[0]
        is_left = node[1]
        if is_left:
            # the order of hash concatenation depends on whether the
            # the node is a left child or right child of its parent
            proof_till_now = compute_hash(node_hash + proof_till_now)
        else:
            proof_till_now = compute_hash(proof_till_now + node_hash)
        print(proof_till_now)

    # verifying the computed root hash against the actual root hash
    return proof_till_now == audit_trail[-1]

You can use any other MerkleTree library for audit proof verification, as long as it can handle the following format:

# pseudo example
[
    ('leafHash', isLeft=True), 
    ('leafHash', isLeft=False), 
    'merkleRoot'
]

# example with data
[
    ('5ad918ad1558058c08c009801ef35649072114d181aac0b3c852b5b10eb6cc83', False), 
    ('81984e745bfb445153dc097fdcff7d3e8b110f6df054b94d834e64d989f92b45', True), 
    '8b288237a5f6c12d92253bdd3f36f42d5c0b067debeabde28185cb819bdd5da2'
] 

Example

First question on Metaculus is Will advanced LIGO announce discovery of gravitational waves by Jan. 31 2016?. It's already closed, so it's easy to verify because closing prediction will be part of every Merkle Tree.

You can see that Community Prediction was 65% at resolution. But since our data are more complex - we store distributions and histogram of individual forecasts - resulting data looks like this.

{'y': [...numbers...], 'q1': 0.6, 'q2': 0.65, 'q3': 0.7}

To access this data for a given day you can use our api endpoint as described above.

By sha256 hashing the following string (encoded as utf-8):

{"1:CP": {"y": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.23925, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.85021, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.31277, 0.0, 0.0, 0.71616, 0.0, 1.39605, 0.0, 0.0, 0.0, 0.0, 1.08686, 0.0, 0.0, 0.0, 0.0, 0.28917, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "q1": 0.6, "q2": 0.65, "q3": 0.7}}

we get af04f9fe178f6d5bfcc85b1166154d4cbbb684cb9666711b913be02447a8f44c.

Now let's plug in into our interface

Our prediction is part of Merkle Root. We did this verification on the backend, but everyone can verify it themselves using standard Merkle Tree audit proof approach described above. Required audit trial if public.

Current limitations & Improvement ideas

To make this better we could

• Make it work with any single prediction update on the platform. Currently we are stamping only 1 CP and MP per day, which seems to be sufficient for practical purposes.
• Allow users to stamp their predictions.
• Stamp all past predictions. Currently we stamped all last standing at time of this release.

If you are interested these improvements or you have ideas for future improvements, please let us know by using contact form or tweeting at us.