-
Miscellany
-
+
+
-
+
+
Basics
-
+
Metaculus is an online forecasting platform and aggregation engine
that brings together a global reasoning community and keeps score for
thousands of forecasters, delivering machine learning-optimized
@@ -318,12 +329,12 @@ export default function FAQ() {
organizations, university researchers and companies to increase the
positive impact of its forecasts.
-
+
Metaculus therefore poses questions about the occurrence of a variety
of future events, on many timescales, to a community of participating
forecasters — you!
-
+
The name "Metaculus" comes from the{" "}
Metaculus genus
@@ -333,25 +344,25 @@ export default function FAQ() {
What is forecasting?
-
+
Forecasting is a systematic practice of attempting to answer questions
about future events. On Metaculus, we follow a few principles to
elevate forecasting above simple guesswork:
-
+
First, questions are carefully specified so that everyone understands
beforehand and afterward which kinds of outcomes are included in the
resolution, and which are not. Forecasters then give precise
probabilities that measure their uncertainty about the outcome.
-
+
Second, Metaculus aggregates the forecasts into a simple{" "}
median (community)
prediction, and an advanced Metaculus Prediction. The Community
@@ -371,7 +382,7 @@ export default function FAQ() {
(provided the whole group is not biased in the same way).
-
+
Third, we measure the relative skill of each forecaster, using their
quantified forecasts. When we know the outcome of the question, the
question is "resolved", and forecasters receive their
@@ -385,17 +396,17 @@ export default function FAQ() {
When is forecasting valuable?
-
+
Forecasting is uniquely valuable primarily in complex, multi-variable
problems or in situations where a lack of data makes it difficult to
predict using explicit or exact models.
-
+
In these and other scenarios, aggregated predictions of strong
forecasters offer one of the best ways of predicting future events. In
fact, work by the political scientist Philip Tetlock demonstrated that
@@ -404,10 +415,10 @@ export default function FAQ() {
forecasting various geopolitical outcomes.
-
+
Why should I be a forecaster?
-
+
Research has shown that great forecasters come from various
backgrounds—and oftentimes from fields that have nothing to do with
predicting the future. Like many mental capabilities, prediction is a
@@ -415,7 +426,7 @@ export default function FAQ() {
Steady quantitative feedback and regular practice can greatly improve
a forecaster's accuracy.
-
+
Some events — such as eclipse timing and well-polled elections, can
often be predicted with high resolution, e.g. 99.9% likely or 3%
likely. Others — such as the flip of a coin or a close horse-race —
@@ -425,19 +436,16 @@ export default function FAQ() {
groups, corporations, governments, and humanity as a whole will make
better decisions.
-
+
As well as being worthwhile, Metaculus aims to be interesting and fun,
while allowing participants to hone their prediction prowess and amass
a track-record to prove it.
-
+
Who created Metaculus?
-
+
{" "}
Metaculus originated with two researcher scientists, Anthony Aguirre
and Greg Laughlin. Aguirre, a physicist, is a co-founder of{" "}
@@ -452,14 +460,14 @@ export default function FAQ() {
What Are Metaculus Tournaments and Question Series?
- Tournaments
-
+
Tournaments
+
Metaculus tournaments are organized around a central topic or theme.
Tournaments are often collaborations between Metaculus and a
nonprofit, government agency, or other organization seeking to use
@@ -467,7 +475,7 @@ export default function FAQ() {
and archived tournaments in our{" "}
Tournaments page.
-
+
Tournaments are the perfect place to prove your forecasting skills,
while helping to improve our collective decision making ability. Cash
prizes and{" "}
@@ -476,7 +484,7 @@ export default function FAQ() {
valuable contributions (like comments). Follow a Tournament (with the
Follow button) to never miss new questions.
-
+
After at least one question has resolved, a Leaderboard will appear on
the tournament page displaying current scores and rankings. A personal
score board ("My Score") will also appear, detailing your
@@ -486,17 +494,17 @@ export default function FAQ() {
.
-
+
At the end of a tournament, the prize pool is divided among
forecasters according to their forecasting performance. The more you
forecasted and the more accurate your forecasts were, the greater
proportion of the prize pool you receive.
-
+
Can I donate my tournament winnings?
-
+
If you have outstanding tournament winnings, Metaculus is happy to
facilitate donations to various non-profits, regranting organizations,
and funds. You can find the list of organizations we facilitate
@@ -507,27 +515,24 @@ export default function FAQ() {
.
- Question Series
-
+
Question Series
+
Like Tournaments, Question Series are organized around a central topic
or theme. Unlike tournaments, they do not have a prize pool.
-
+
Question Series still show leaderboards, for interest and fun. However
they do **not** award medals.
-
+
You can find all Question Series in a special section of the{" "}
Tournaments page.
-
+
Is Metaculus a prediction market?
-
+
Sort of. Like a prediction market, Metaculus aims to aggregate many
people's information, expertise, and predictive power into
high-quality forecasts. However, prediction markets generally operate
@@ -541,10 +546,10 @@ export default function FAQ() {
prediction market.
-
+
Advantages of Metaculus over prediction markets
-
+
Metaculus has several advantages over prediction markets. One is that
Metaculus forecasts are scored solely based on accuracy, while
prediction markets may be used for other reasons, such as hedging.
@@ -553,13 +558,10 @@ export default function FAQ() {
if an event occurs.
-
+
Are Metaculus Questions Polls?
-
+
No. Opinion polling can be a useful way to gauge the sentiment and
changes in a group or culture, but there is often no single
"right answer", as in a{" "}
@@ -569,7 +571,7 @@ export default function FAQ() {
"How worried are you about the environment?"
-
+
In contrast, Metaculus questions are designed to be objectively
resolvable (like in{" "}
@@ -587,19 +589,16 @@ export default function FAQ() {
-
+
What sorts of questions are allowed, and what makes a good question?
-
+
Questions should focus on tangible, objective facts about the world
which are well-defined and not a matter of opinion. "When will
the United States collapse?" is a poor, ambiguous question;{" "}
@@ -614,12 +613,12 @@ export default function FAQ() {
When will (event) X occur?
or{" "}
What will the value or quantity of X be by (date) Y?
-
+
A good question will be unambiguously resolvable. A community reading
the question terms should be able to agree, before and after the event
has occurred, whether the outcome satisfies the question's terms.
- Questions should also follow some obvious rules:
+ Questions should also follow some obvious rules:
-
@@ -649,13 +648,10 @@ export default function FAQ() {
-
+
Who creates the questions, and who decides which get posted?
-
+
Many questions are launched by Metaculus staff, but any logged-in user
can propose a question. Proposed questions will be reviewed by a group
of moderators appointed by Metaculus. Moderators will select the best
@@ -664,7 +660,7 @@ export default function FAQ() {
aligned with our writing style.
-
+
Metaculus hosts questions on{" "}
many topics, but our primary
focus areas are Science,{" "}
@@ -686,10 +682,7 @@ export default function FAQ() {
, and Geopolitics.
-
+
Who can edit questions?
@@ -710,10 +703,7 @@ export default function FAQ() {
or Pending.
-
+
How do I invite co-authors to my question?
@@ -737,7 +727,7 @@ export default function FAQ() {
How can I get my own question posted?
@@ -795,13 +785,13 @@ export default function FAQ() {
What can I do if a question I submitted has been pending for a long
time?
-
+
We currently receive a large volume of question submissions, many of
which are interesting and well-written. That said, we try to approve
just enough questions that they each can get the attention they
@@ -829,12 +819,12 @@ export default function FAQ() {
What can I do if a question should be resolved but isn't?
-
+
If a question is still waiting for resolution, check to make sure
there hasn't been a comment from staff explaining the reason for
the delay. If there hasn't, you can tag @admins to alert the
@@ -843,12 +833,12 @@ export default function FAQ() {
What is a private question?
-
+
Private questions are questions that are not visible to the broader
community. They aren't subject to the normal review process, so
you can create one and predict on it right away. You can resolve your
@@ -856,7 +846,7 @@ export default function FAQ() {
won't be added to your overall Metaculus score and they
won't affect your ranking on the leaderboard.
-
+
You can use private questions for anything you want. Use them as
practice to calibrate your predictions before playing for points,
create a question series on a niche topic, or pose personal questions
@@ -864,19 +854,16 @@ export default function FAQ() {
You can even invite up to 19 other users to view and
predict on your own questions!
-
+
To invite other forecasters to your private question, click the
'...' more options menu and select 'Share Private
Question'.
-
+
We have a full set of{" "}
community etiquette guidelines but in
summary:
@@ -914,14 +901,11 @@ export default function FAQ() {
-
+
What do "credible source" and "before [date X]"
and such phrases mean exactly?
-
+
To reduce ambiguity in an efficient way, here are some definitions
that can be used in questions, with a meaning set by this FAQ:
@@ -954,14 +938,11 @@ export default function FAQ() {
-
+
What types of questions are there?
- Binary Questions
-
+
Binary Questions
+
Binary questions can resolve as either Yes or{" "}
No (unless the resolution criteria were
underspecified or otherwise circumvented, in which case they can
@@ -975,8 +956,8 @@ export default function FAQ() {
dropped below 5% before the specified time.
-
Range Questions
-
+
Range Questions
+
Range questions resolve to a certain value, and forecasters can
specify a probability distribution to estimate the likelihood of each
value occurring. Range questions can have open or closed bounds. If
@@ -987,19 +968,19 @@ export default function FAQ() {
See here for more details
about boundaries on range questions.
-
+
The range interface allows you to input multiple probability
distributions with different weights.{" "}
See here for more details on using the
interface.
-
+
There are two types of range questions, numeric range questions and
date range questions.
-
Numeric Range
-
+
Numeric Range
+
Numeric range questions can resolve as a numeric value. For example,
the question "
@@ -1009,7 +990,7 @@ export default function FAQ() {
" resolved as 395, because the underlying source
reported 395 thousand initial jobless claims for July 2021.
-
+
Questions can also resolve outside the numeric range. For example, the
question "
@@ -1021,8 +1002,8 @@ export default function FAQ() {
and 6.5 was the upper bound.
-
Date Range
-
+
Date Range
+
Date range questions can resolve as a certain date. For example, the
question "
@@ -1032,7 +1013,7 @@ export default function FAQ() {
" resolved as July 23, 2022, because a Public
Health Emergency of International Concern was declared on that date.
-
+
Questions can also resolve outside the date range. For example, the
question "
@@ -1044,23 +1025,23 @@ export default function FAQ() {
What are question groups?
-
+
Question groups are sets of closely related questions or question
outcomes all collected on a single page. Forecasters can predict
quickly and efficiently on these interconnected outcomes, confident
that they are keeping all of their predictions internally consistent.
-
+
How do question groups facilitate more efficient, more accurate
forecasting?
-
+
With question groups, it's easy to forecast progressively wider
distributions the further into the future you predict to reflect
increasing uncertainty. A question group collecting multiple binary
@@ -1069,11 +1050,11 @@ export default function FAQ() {
each other.
-
+
What happens to the existing question pages when they are combined in
a question group?
-
+
When regular forecast questions are converted into
"subquestions" of a question group, the original pages are
replaced by a single question group page. Comments that previously
@@ -1082,28 +1063,28 @@ export default function FAQ() {
move.
-
+
Do I need to forecast on every outcome / subquestion of a question
group?
-
+
No. Question groups comprise multiple independent subquestions.
For that reason, there is no requirement that you forecast on every
outcome within a group.
-
+
How are question groups scored?
-
+
Each outcome or subquestion is scored in the same manner as a normal
independent question.
-
+
Why don't question group outcome probabilities sum to 100%?
-
+
Even if there can only be one outcome for a particular question group,
the Community and Metaculus Predictions function as they would for
normal independent questions. The Community and Metaculus Predictions
@@ -1111,7 +1092,7 @@ export default function FAQ() {
on each subquestion, respectively. These medians and weighted
aggregates are not constrained to sum to 100%
-
+
Feedback for question groups can be provided on the{" "}
question group discussion post
@@ -1119,13 +1100,10 @@ export default function FAQ() {
.
-
+
What are Conditional Pairs?
-
+
A Conditional Pair is a special type of{" "}
Question Group
@@ -1142,7 +1120,7 @@ export default function FAQ() {
.
-
+
Conditional Pairs ask two Conditional Questions (or
"Conditionals" for short), each corresponding to a possible
outcome of the Parent:
@@ -1153,13 +1131,13 @@ export default function FAQ() {
If the Parent resolves No, how will the Child resolve?
-
+
The first Conditional assumes that "The Parent resolves Yes"
(or "if Yes" for short). The second conditional does the
same for No.
-
+
Conditional probabilities are probabilities, so forecasting is very
similar to Binary Questions. The main difference is that we present
both conditionals next to each other for convenience:
@@ -1168,13 +1146,12 @@ export default function FAQ() {
-
+
Conditional questions are automatically resolved when their Parent and
Child resolve:
@@ -1193,23 +1170,21 @@ export default function FAQ() {
-
Let's work through an example:
+
Let's work through an example:
- The Parent is "Will it rain today?".
- The Child is "Will it rain tomorrow?".
-
- So the two Conditionals in the Conditional Pair will be:
-
+
So the two Conditionals in the Conditional Pair will be:
- "If it rains today, will it rain tomorrow?"
- "If it does not rain today, will it rain tomorrow?"
-
+
For simplicity, Metaculus presents conditional questions graphically.
In the forecasting interface they are in a table:
@@ -1217,13 +1192,12 @@ export default function FAQ() {
-
+
And in the feeds, each possible outcome of the Parent is an arrow, and
each conditional probability is a bar:
@@ -1231,29 +1205,28 @@ export default function FAQ() {
-
Back to the example:
+
Back to the example:
-
+
It rains today. The parent resolves Yes. This triggers the second
conditional ("if No") to be annulled. It is not scored.
-
+
You wait a day. This time it doesn't rain. The Child resolves No.
This triggers the remaining Conditional ("if Yes") to
resolve No. It is scored like a normal Binary Question.
-
+
How do I create conditional pairs?
-
+
You can create and submit conditional pairs like any other question
type. On the '
@@ -1263,25 +1236,25 @@ export default function FAQ() {
select Parent and Child questions.
-
+
Note: You can use question group subquestions as the Parent or Child
by clicking the Parent or Child button and then either searching for
the subquestion in the field or pasting the URL for the subquestion.
-
+
To copy the URL for a subquestion, simply visit a question group page
and click the '...' more options menu to reveal the Copy
Link option.
How do I find certain questions on Metaculus?
-
+
Questions on Metaculus are sorted by activity by default. Newer
questions, questions with new comments, recently upvoted questions,
and questions with many new predictions will appear at the top of the{" "}
@@ -1290,24 +1263,18 @@ export default function FAQ() {
interest and customize the way you interact with Metaculus.
-
+
Search Bar
-
+
The search bar can be used to find questions using keywords and
semantic matches. At this time it cannot search comments or users.
-
+
Filters
-
+
Questions can be sorted and filtered in a different manner from the
default using the filters menu. Questions can be filtered by type,
status and participation. Questions can also be ordered, for example
@@ -1318,20 +1285,17 @@ export default function FAQ() {
Question Resolution
-
+
What are the "open date", "close date" and
"resolve date?"
-
+
When submitting a question, you are asked to specify the closing date
(when the question is no longer available for predicting) and
resolution date (when the resolution is expected to occur). The date
@@ -1361,7 +1325,7 @@ export default function FAQ() {
not binding in any way.
-
+
In some cases, questions must resolve at the resolution date according
to the best available information. In such cases, it becomes important
to choose the resolution date carefully. Try to set resolution dates
@@ -1372,7 +1336,7 @@ export default function FAQ() {
"this question resolves as Yes if X happens
before January 1, 2040)".
-
+
The close date must be at least one hour prior to the
resolution date, but can be much earlier, depending upon the context.
Here are some guidelines for specifying the close date:
@@ -1403,7 +1367,7 @@ export default function FAQ() {
discretion of the period of interest.
-
+
Note: Previous guidance suggested that a question
should close between 1/2 to 2/3 of the way between the open time and
resolution time. This was necessary due to the scoring system at the
@@ -1414,13 +1378,10 @@ export default function FAQ() {
.
-
+
What timezone is used for questions?
-
+
For dates and times written in the question, such as "will event
X happen before January 1, 2030?", if the timezone is not
specified{" "}
@@ -1431,7 +1392,7 @@ export default function FAQ() {
timezone in the resolution criteria, and any timezone specified in the
text will be used.
-
+
For{" "}
date range{" "}
questions, the dates on the interface are in UTC. Typically the time
@@ -1447,13 +1408,10 @@ export default function FAQ() {
but not what time of day, it will resolve as noon UTC on that day.
-
+
Who decides the resolution to a question?
-
+
Only Metaculus Administrators can resolve questions. Binary questions
can resolve Yes, No,{" "}
@@ -1468,21 +1426,21 @@ export default function FAQ() {
What are "Ambiguous" and "Annulled" resolutions?
-
+
Sometimes a question cannot be resolved because the state of the
world, the truth of the matter
, is too uncertain. In these
cases, the question is resolved as Ambiguous.
-
+
Other times, the state of the world is clear, but a key assumption of
the question was overturned. In these cases, the question is Annulled.
-
+
In the same way, when a Conditional turns out to be based on an
outcome that did not occur, it is Annulled. For example, when a{" "}
@@ -1490,29 +1448,29 @@ export default function FAQ() {
's parent resolves Yes, the if No
Conditional is Annulled.
-
+
When questions are Annulled or resolved as Ambiguous, they are no
longer open for forecasting, and they are not scored.
-
+
If you'd like to read more about why Ambiguous and Annulled
resolutions are necessary you can expand the section below.
-
+
Reasons for Ambiguous and Annulled resolutions
Why was this question Annulled or resolved as Ambiguous?
-
+
An Ambiguous or Annulled resolution generally implies that there
was some inherent ambiguity in the question, that real-world
events subverted one of the assumptions of the question, or that
@@ -1523,7 +1481,7 @@ export default function FAQ() {
consider factors such as fairness to all participating forecasters
and the underlying incentives toward accurate forecasting.
-
+
To avoid this unfairness and provide the most accurate
information, we resolve all questions in accordance with the
actual written text of the resolution criteria whenever possible.
@@ -1538,12 +1496,12 @@ export default function FAQ() {
Types of Ambiguous or Annulled Resolutions
-
+
A question's resolution criteria can be thought of as akin to
a legal contract. The resolution criteria create a shared
understanding of what forecasters are aiming to predict, and
@@ -1558,7 +1516,7 @@ export default function FAQ() {
Additionally, the information provided by the forecasts on the
question will be poor due to the differing interpretations.
-
+
The following sections provide more detail about common reasons we
resolve questions as Ambiguous or Annul them and some examples.
Some of these examples could fit into multiple categories, but
@@ -1620,7 +1578,7 @@ export default function FAQ() {
-
+
Note: Previously Metaculus only had one
resolution type — Ambiguous — for cases where a
question could not otherwise be resolved. We've since
@@ -1631,12 +1589,12 @@ export default function FAQ() {
Ambiguous Resolution
-
+
Ambiguous resolution is reserved for questions where reality is
not clear. Either because reporting about an event is conflicted
or unclear about what actually happened, or available material is
@@ -1646,12 +1604,12 @@ export default function FAQ() {
No Clear Consensus
-
+
Questions can also resolve Ambiguous when there is not enough
information available to arrive at an appropriate resolution. This
can be because of conflicting or unclear media reports, or because
@@ -1724,12 +1682,12 @@ export default function FAQ() {
No Clear Consensus
-
+
Questions can also resolve Ambiguous when there is not enough
information available to arrive at an appropriate resolution. This can
be because of conflicting or unclear media reports, or because a data
@@ -1800,29 +1758,29 @@ export default function FAQ() {
Annulment
-
+
Annulling a question is reserved for situations where reality is clear
but the question is not. In other words, the question failed to
adequately capture a method for clear resolution.
-
+
Note: Annulment was introduced in April of 2023, so
while the following examples describe Annulment the questions in
actuality were resolved as Ambiguous.
The Question Was Underspecified
-
+
Writing good forecasting questions is hard, and it only gets harder
the farther the question looks into the future. To fully eliminate the
potential for a question to be Annulled the resolution criteria must
@@ -1836,7 +1794,7 @@ export default function FAQ() {
find an interpretation that is clearly an appropriate fit for the
resolution criteria, but this is not always possible.
-
+
Here are some examples of Annulment due to underspecified questions:
@@ -1910,12 +1868,12 @@ export default function FAQ() {
The Assumptions of the Question Were Subverted
-
+
Questions often contain assumptions in their resolution criteria, many
of which are unstated. For example, assuming that the underlying
methodology of a data source will remain the same, assuming that an
@@ -1926,9 +1884,7 @@ export default function FAQ() {
situations) but due to the difficulty in anticipating these outcomes
this isn't always done.
-
- Here are some examples of Annulment due to subverted assumptions:
-
+
Here are some examples of Annulment due to subverted assumptions:
Imbalanced Outcomes and Consistent Incentives
-
+
Sometimes questions imply imbalanced outcomes, for example where the
burden of proof for an event to be considered to have occurred is high
and tips the scales toward a binary question resolving No, or where
@@ -2045,7 +2001,7 @@ export default function FAQ() {
resolution of Yes or Annulled. This creates a bias in the question and
also produces bad incentives if the question isn't Annulled.
-
+
The case of imbalanced outcomes and consistent incentives is best
explained with examples, such as the following:
@@ -2448,6 +2404,7 @@ export default function FAQ() {
question, Metaculus may choose a suitable replacement.
+
track record page.
+
-
+
- A direct quotation from the prediction news source
- The name of the news source
- A link to the news source
@@ -2984,6 +2942,7 @@ export default function FAQ() {
you had a forecast, but not the others.
+
+
-
+
Miscellany
@@ -3120,7 +3080,7 @@ export default function FAQ() {
Metaculus selects individuals according to the following criteria:
-
+
- Have scores in the top 2% of all Metaculus forecasters.
-
Have forecasted on a minimum of 75+ questions that have been
diff --git a/front_end/src/app/(main)/help/scores-faq/components/baseline_math.tsx b/front_end/src/app/(main)/help/scores-faq/components/baseline_math.tsx
new file mode 100644
index 0000000000..12f69a1812
--- /dev/null
+++ b/front_end/src/app/(main)/help/scores-faq/components/baseline_math.tsx
@@ -0,0 +1,94 @@
+"use client";
+
+import React from "react";
+
+import StyledDisclosure from "../../../components/styled_disclosure";
+import MathJaxContent from "@/components/math_jax_content";
+
+const BaselineMath = () => {
+ return (
+
+
+ The Baseline scores are rescaled{" "}
+ log scores, with the general
+ form:
+
+
+
+ For binary and multiple choice questions, the{" "}
+ is chosen so that a perfect
+ prediction ()
+ gives a score of +100. The formula for a binary question is:
+
+
+
+ Note that you can rearrange this formula into:{" "}
+
+ .
+
+ The formula for a multiple choice question with N options is:
+
+
+ For continuous questions, the {" "}
+ was chosen empirically so that continuous scores have roughly the same
+ average as binary scores. The formula for a continuous question is:
+
+
+
+ Where is the natural
+ logarithm, is the
+ probability predicted for the outcome that actually happened, and{" "}
+ is the
+ value of the predicted probability density function at the outcome.
+
+
+ The continuous depends on
+ whether the question has open or closed bounds:
+
+
+ -
+ If both bounds are closed, the{" "}
+ is 1, corresponding to a
+ uniform distribution in range.
+
+ -
+ If one bound is open, the{" "}
+ is 0.95, corresponding
+ to a uniform distribution in range + 5% probability out of the open
+ bound.
+
+ -
+ If both bounds are open, the{" "}
+ is 0.9, corresponding to
+ a uniform distribution in range + 5% probability out of each open
+ bound.
+
+
+
+ );
+};
+
+export default BaselineMath;
diff --git a/front_end/src/app/(main)/help/scores-faq/components/further_math.tsx b/front_end/src/app/(main)/help/scores-faq/components/further_math.tsx
new file mode 100644
index 0000000000..961000b768
--- /dev/null
+++ b/front_end/src/app/(main)/help/scores-faq/components/further_math.tsx
@@ -0,0 +1,29 @@
+"use client";
+
+import React from "react";
+
+import StyledDisclosure from "../../../components/styled_disclosure";
+
+const FurtherMath = () => {
+ return (
+
+
+ If you have an intuition that something should be 0 and not positive,
+ you are correct! The average Peer score across all users is guaranteed
+ to be 0. This does not imply that the score of the average (or median)
+ forecast is 0: the score of the mean is not the mean of the scores.
+
+
+
+ There is another reason why the Peer score of the Community Prediction
+ is positive: you can rearrange the Peer score formula to show that it is
+ the difference between the forecaster log score and the log score of the
+ geometric mean of all other forecasters. Since the median will be higher
+ than the geometric mean in most cases, it follows that the score of the
+ Community Prediction will be positive in most cases.
+
+
+ );
+};
+
+export default FurtherMath;
diff --git a/front_end/src/app/(main)/help/scores-faq/components/peer_math.tsx b/front_end/src/app/(main)/help/scores-faq/components/peer_math.tsx
new file mode 100644
index 0000000000..9588c14c1d
--- /dev/null
+++ b/front_end/src/app/(main)/help/scores-faq/components/peer_math.tsx
@@ -0,0 +1,53 @@
+"use client";
+
+import React from "react";
+
+import StyledDisclosure from "../../../components/styled_disclosure";
+import MathJaxContent from "@/components/math_jax_content";
+
+const PeerMath = () => {
+ return (
+
+
+ The Peer scores are built on{" "}
+ log scores, with the general
+ form:
+
+
+
+ Where is the scored prediction,{" "}
+ is the number of other
+ predictions, and is the i-th
+ other prediction.
+
+ Note that this can be rearranged into:
+
+
+ Where{" "}
+
+
+ {" "}
+ is the geometric mean of all other predictions.
+
+
+ As before, for binary questions {" "}
+ is the probability given to the correct outcome (Yes or No), for
+ multiple choice questions it is the probability given to the option
+ outcome that resolved Yes, and for continuous questions it is the value
+ of the predicted pdf at the outcome.
+
+
+ );
+};
+
+export default PeerMath;
diff --git a/front_end/src/app/(main)/help/scores-faq/components/points_math.tsx b/front_end/src/app/(main)/help/scores-faq/components/points_math.tsx
new file mode 100644
index 0000000000..9b6d68db6b
--- /dev/null
+++ b/front_end/src/app/(main)/help/scores-faq/components/points_math.tsx
@@ -0,0 +1,110 @@
+"use client";
+
+import React from "react";
+
+import StyledDisclosure from "../../../components/styled_disclosure";
+import MathJaxContent from "@/components/math_jax_content";
+
+const PointsMath = () => {
+ return (
+
+
+ Your score at any given time{" "}
+ is the sum of an "absolute"
+ component and a "relative" component:
+
+
+ where:
+
+ -
+ is the outcome of the question: 1 if
+ the question resolves positive, 0 if it resolves negative.
+
+ -
+ is the number of forecasters on the
+ question.
+
+ -
+ is the log score relative to a
+ 50% prior, defined as:
+
+
+
+
+ -
+ is the betting score and
+ represents a bet placed against every other forecaster. It is
+ described under "constant pool scoring" on the Metaculus
+ scoring demo (but with a modification that for computational
+ efficiency, the full distribution of other forecaster predictions is
+ replaced by a fitted{" "}
+
+ beta distribution
+
+ ).
+
+ -
+ and{" "}
+ depend on{" "}
+ only and define how the points scale
+ with the number of forecasters.
+
+
+
+
+ Note that ,{" "}
+ , and {" "}
+ can all depend on and contribute to the
+ time-dependence of .
+
+
+ Your final score is given by the integral of{" "}
+ over{" "}
+ :
+
+
+
+ where and{" "}
+ are the opening and closing times.
+ (Note that between the opening
+ time and your first prediction, and is also zero after question
+ resolution but before question close, in the case when a question
+ resolves early.)
+
+
+ Before May 2022, there was also a 50% point bonus given at the time the
+ question closes, but it was discontinued and the points multiplied by
+ 1.5 henceforth.
+
+
+ );
+};
+
+export default PointsMath;
diff --git a/front_end/src/app/(main)/help/scores-faq/components/truncation_example.tsx b/front_end/src/app/(main)/help/scores-faq/components/truncation_example.tsx
new file mode 100644
index 0000000000..adc3de9ecb
--- /dev/null
+++ b/front_end/src/app/(main)/help/scores-faq/components/truncation_example.tsx
@@ -0,0 +1,172 @@
+"use client";
+
+import React from "react";
+
+import StyledDisclosure from "../../../components/styled_disclosure";
+import MathJaxContent from "@/components/math_jax_content";
+
+const TruncationExample = () => {
+ return (
+
+
+ This example uses the Baseline score, which will be noted{" "}
+ , but results would be equivalent
+ with any proper score.
+
+
+ Alex wants to predict if they will be fired this year. They have a
+ performance review scheduled this week. They estimate there is a{" "}
+ chance they fail it, and if
+ so they will be fired on the spot. If they don’t fail this week, there
+ is still a chance they will be
+ fired at the end of the year. A proper scoring rule ensures that the
+ best strategy on this question is to predict{" "}
+ {" "}
+ this week, and then for the
+ other 51 weeks (if they weren’t fired).
+
+
+ Without truncation
+
+ Without truncation, this honest strategy gives Baseline scores of:
+
+ -
+ in the{" "}
+ of cases they are fired
+ this week.
+
+ -
+ {" "}
+ in the {" "}
+ of cases they are fired at the end of the year.
+
+ -
+ {" "}
+ in the{" "}
+ of
+ cases they are not fired.
+
+
+
+ For an average score of{" "}
+ {" "}
+ in expectation.
+
+
+ But the strategy of “predicting close to 100% in the beginning and lower
+ later”, let's say 99% today, then 5% the other 6 days, without
+ truncation gives Baseline scores of:
+
+
+ -
+ in the{" "}
+ of cases they are fired
+ this week.
+
+ -
+ {" "}
+ in the {" "}
+ of cases they are fired at the end of the year.
+
+ -
+ {" "}
+ in the{" "}
+ of
+ cases they are not fired.
+
+
+
+ For an average score of{" "}
+ {" "}
+ in expectation.
+
+
+ Notice that +36\\)`} />, so without
+ truncation, the gaming strategy gives you a score almost twice as high
+ in expectation! It is really not proper.
+
+
+ With truncation
+
+ With truncation, the honest strategy gives Baseline scores of:
+
+ -
+ {" "}
+ in the of cases they are
+ fired this week.
+
+ -
+ {" "}
+ in the {" "}
+ of cases they are fired at the end of the year.
+
+ -
+ {" "}
+ in the{" "}
+ of
+ cases they are not fired.
+
+
+
+ For an average score of{" "}
+ {" "}
+ in expectation.
+
+ While the gaming strategy gives:
+
+ -
+ {" "}
+ in the of cases they are
+ fired this week.
+
+ -
+ {" "}
+ in the {" "}
+ of cases they are fired at the end of the year.
+
+ -
+ {" "}
+ in the{" "}
+ of
+ cases they are not fired.
+
+
+
+ For an average score of{" "}
+ {" "}
+ in expectation.
+
+
+ This time, +48\\)`} />, so with
+ truncation, the gaming strategy gives you a worse score than the honest
+ strategy! Which is proper.
+
+
+ );
+};
+
+export default TruncationExample;
diff --git a/front_end/src/app/(main)/help/scores-faq/page.tsx b/front_end/src/app/(main)/help/scores-faq/page.tsx
index a172cc1ee3..482a708157 100644
--- a/front_end/src/app/(main)/help/scores-faq/page.tsx
+++ b/front_end/src/app/(main)/help/scores-faq/page.tsx
@@ -1,8 +1,15 @@
import PageWrapper from "../../components/pagewrapper";
+import MathJaxContent from "@/components/math_jax_content";
+import BaselineMath from "./components/baseline_math";
+import PeerMath from "./components/peer_math";
+import FurtherMath from "./components/further_math";
+import TruncationExample from "./components/truncation_example";
+import PointsMath from "./components/points_math";
export const metadata = {
title: "Scores FAQ | Metaculus",
- description: "tbd",
+ description:
+ "Learn how Metaculus scores work, including Peer scores, Relative scores, and legacy scoring methods. Understand tournament rankings, coverage, and prize calculations.",
};
export default function ScoresFAQ() {
@@ -11,17 +18,16 @@ export default function ScoresFAQ() {
Scores FAQ
Below are Frequently Asked Questions (and answers!) about scores. The
- general FAQ is here, and the medals FAQ is{" "}
+ general FAQ is here, and the medals FAQ is{" "}
here.
- Contents:
-
+
-
+
Scores
+
What is a scoring rule?
@@ -128,6 +135,7 @@ export default function ScoresFAQ() {
see in the next section.
+
What is a proper scoring rule?
@@ -149,7 +157,7 @@ export default function ScoresFAQ() {
-
+
| outcome die roll |
naive score of p=5% |
naive score of p=17% |
@@ -157,43 +165,43 @@ export default function ScoresFAQ() {
-
+
| 1 |
0.95 |
0.83 |
0.7 |
-
+
| 2 |
0.95 |
0.83 |
0.7 |
-
+
| 3 |
0.95 |
0.83 |
0.7 |
-
+
| 4 |
0.95 |
0.83 |
0.7 |
-
+
| 5 |
0.95 |
0.83 |
0.7 |
-
+
| 6 |
0.05 |
0.17 |
0.3 |
-
+
| average |
0.8 |
0.72 |
@@ -219,7 +227,7 @@ export default function ScoresFAQ() {
-
+
| outcome die roll |
log score of p=5% |
log score of p=17% |
@@ -227,43 +235,43 @@ export default function ScoresFAQ() {
-
+
| 1 |
-0.05 |
-0.19 |
-0.37 |
-
+
| 2 |
-0.05 |
-0.19 |
-0.37 |
-
+
| 3 |
-0.05 |
-0.19 |
-0.37 |
-
+
| 4 |
-0.05 |
-0.19 |
-0.37 |
-
+
| 5 |
-0.05 |
-0.19 |
-0.37 |
-
+
| 6 |
-3 |
-1.77 |
-1.2 |
-
+
| average |
-0.54 |
-0.45 |
@@ -277,59 +285,880 @@ export default function ScoresFAQ() {
With the log score, you do get a higher (better) score if you predict
the true probability of 17%.
- {/*
- What is the log score?
- The logarithmic scoring rule, or "log score" for short, is defined as:
-
- {`\\[
+
+
+ What is the log score?
+
+
+ The logarithmic scoring rule, or "log score" for short, is
+ defined as:
+
+
- Where {`\\(\\ln\\)`} is the natural logarithm and {`\\(P(outcome)\\)`} is the probability predicted for the outcome that actually happened. This log score applies to categorical predictions, where one of a (usually) small set of outcomes can happen. On Metaculus those are Binary and Multiple Choice questions. See the next section for the log scores of continuous questions.
- Higher scores are better:
-
- - If you predicted 0% on the correct outcome, your score will be {`\\(-\\infty\\)`} (minus infinity).
- - If you predict 100% on the correct outcome, your score will be 0.
-
- This means that the log score is always negative (for Binary and Multiple Choice questions). This has proved unintuitive, which is one reason why Metaculus uses the Baseline and Peer scores, which are based on the log score but can be positive.
- The log score is proper (see What is a proper scoring rule?). This means that to maximize your score you should predict your true beliefs (see Can I get better scores by predicting extreme values?).
- One interesting property of the log score: it is much more punitive of extreme wrong predictions than it is rewarding of extreme right predictions. Consider the scores you get for predicting 99% or 99.9%:
+ block
+ />
+
+ Where is the natural logarithm
+ and is the probability
+ predicted for the outcome that actually happened. This log score applies
+ to categorical predictions, where one of a (usually) small set of
+ outcomes can happen. On Metaculus those are Binary and Multiple Choice
+ questions. See the next section for the log scores of continuous
+ questions.
+
+ Higher scores are better:
+
+ -
+ If you predicted 0% on the correct outcome, your score will be{" "}
+ (minus infinity).
+
+ -
+ If you predict 100% on the correct outcome, your score will be 0.
+
+
+
+ This means that the log score is always negative (for Binary and
+ Multiple Choice questions). This has proved unintuitive, which is one
+ reason why Metaculus uses the{" "}
+ Baseline and{" "}
+ Peer scores, which are based
+ on the log score but can be positive.
+
+
+ The log score is proper (see{" "}
+
+ What is a proper scoring rule?
+
+ ). This means that to maximize your score{" "}
+ you should predict your true beliefs (see{" "}
+
+ Can I get better scores by predicting extreme values?
+
+ ).
+
+
+ One interesting property of the log score: it is much more punitive of
+ extreme wrong predictions than it is rewarding of extreme right
+ predictions. Consider the scores you get for predicting 99% or 99.9%:
+
-
-
-
-
- |
- 99% Yes, 1% No |
- 99.9% Yes, 0.1% No |
-
-
-
-
- | Score if outcome = Yes |
- -0.01 |
- -0.001 |
-
-
- | Score if outcome = No |
- -4.6 |
- -6.9 |
-
-
-
-
+
+
+
+
+ |
+ 99% Yes, 1% No |
+ 99.9% Yes, 0.1% No |
+
+
+
+
+ | Score if outcome = Yes |
+ -0.01 |
+ -0.001 |
+
+
+ | Score if outcome = No |
+ -4.6 |
+ -6.9 |
+
+
+
+
- Going from 99% to 99.9% only gives you a tiny advantage if you are correct (+0.009), but a huge penalty if you are wrong (-2.3). So be careful, and only use extreme probabilities when you're sure they're appropriate!
+
+ Going from 99% to 99.9% only gives you a tiny advantage if you are
+ correct (+0.009), but a huge penalty if you are wrong (-2.3). So be
+ careful, and only use extreme probabilities when you're sure
+ they're appropriate!
+
- What is the log score for continuous questions?
- Since the domain of possible outcomes for continuous questions is (drum roll) continuous, any outcome has mathematically 0 chance of happening. Thankfully we can adapt the log score in the form:
-
- {`\\[
+
+
+ What is the log score for continuous questions?
+
+
+ Since the domain of possible outcomes for continuous questions is (drum
+ roll) continuous, any outcome has mathematically 0 chance of happening.
+ Thankfully we can adapt the log score in the form:
+
+
- Where {`\\(\\ln\\)`} is the natural logarithm and {`\\(\\operatorname{pdf}(outcome)\\)`} is the value of the predicted probability density function at the outcome. Note that on Metaculus, all pdfs have a uniform distribution of height 0.01 added to them. This prevents extreme log scores.
- This is also a proper scoring rule, and behaves in somewhat similar ways to the log score described above. One difference is that, contrary to probabilities that are always between 0 and 1, {`\\(\\operatorname{pdf}\\)`} values can be greater than 1. This means that the continuous log score can be greater than 0: in theory it has no maximum value, but in practice Metaculus restricts how sharp pdfs can get (see the maximum scores tabulated below).
*/}
+ block
+ />
+
+ Where is the natural logarithm
+ and is
+ the value of the predicted{" "}
+
+ probability density function
+ {" "}
+ at the outcome. Note that on Metaculus, all pdfs have a{" "}
+
+ uniform distribution
+ {" "}
+ of height 0.01 added to them. This prevents extreme log scores.
+
+
+ This is also a proper scoring rule, and behaves in somewhat similar ways
+ to the log score described above. One difference is that, contrary to
+ probabilities that are always between 0 and 1,{" "}
+ values can be
+ greater than 1. This means that the continuous log score can be greater
+ than 0: in theory it has no maximum value, but in practice Metaculus
+ restricts how sharp pdfs can get (see the maximum scores tabulated
+ below).
+
+
+
+ What is the Baseline score?
+
+
+ The Baseline score compares a prediction to a fixed "chance"
+ baseline. If it is positive, the prediction was better than chance. If
+ it is negative, it was worse than chance.
+
+
+ That "chance" baseline gives the same probability to all
+ outcomes. For binary questions, this is a prediction of 50%. For an
+ N-option multiple choice question it is a prediction of 1/N for every
+ option. For continuous questions this is a uniform (flat) distribution.
+
+
+ The Baseline score is derived from the{" "}
+ log score, rescaled so that:
+
+
+ -
+ Predicting the same probability on all outcomes gives a score of 0.
+
+ -
+ Predicting perfectly on a binary or multiple choice question gives a
+ score of +100.
+
+ -
+ The average scores of binary and continuous questions roughly match.
+
+
+ Here are some notable values for the Baseline score:
+
+
+
+
+ |
+
+ Binary questions
+ |
+
+ Multiple Choice questions
+
+ (8 options)
+ |
+
+ Continuous questions
+ |
+
+
+
+
+ |
+ Best possible Baseline score on Metaculus
+ |
+ +99.9 |
+ +99.9 |
+ +183 |
+
+
+ |
+ Worst possible Baseline score on Metaculus
+ |
+ -897 |
+ -232 |
+ -230 |
+
+
+ |
+ Median Baseline empirical score
+ |
+ +17 |
+ no data yet |
+ +14 |
+
+
+ |
+ Average Baseline empirical score
+ |
+ +13 |
+ no data yet |
+ +13 |
+
+
+
+
+
+ Theoretically, binary scores can be infinitely negative, and continuous
+ scores can be both infinitely positive and infinitely negative. In
+ practice, Metaculus restricts binary predictions to be between 0.1% and
+ 99.9%, and continuous pdfs to be between 0.01 and ~35, leading to the
+ scores above. The empirical scores are based on all scores observed on
+ all resolved Metaculus questions, as of November 2023.
+
+
+ Note that the above describes the Baseline score at a single point in
+ time. Metaculus scores are time-averaged over the lifetime of the
+ question, see{" "}
+
+ Do all my predictions on a question count toward my score?
+
+ .
+
+ You can expand the section below for more details and maths.
+
+
+
+ What is the Peer score?
+
+
+ The Peer score compares a prediction to all the other predictions made
+ on the same question. If it is positive, the prediction was (on average)
+ better than others. If it is negative it was worse than others.
+
+
+ The Peer score is derived from the{" "}
+ log score: it is the average
+ difference between a prediction's log score, and the log scores of
+ all other predictions on that question. Like the Baseline score, the
+ Peer score is multiplied by 100.
+
+
+ One interesting property of the Peer score is that, on any given
+ question, the sum of all participants' Peer scores is always 0.
+ This is because each forecaster's score is their average difference
+ with every other: when you add all the scores, all the differences
+ cancel out and the result is 0. Here's a quick example: imagine a{" "}
+ continuous question
+ , with three forecasters having predicted:
+
+
+
+
+
+ |
+ Forecaster
+ |
+
+ log score
+ |
+
+ Peer score
+ |
+
+
+
+
+ | Alex |
+
+
+ |
+
+
+ |
+
+
+ |
+ Bailey
+ |
+
+
+ |
+
+
+ |
+
+
+ | Cory |
+
+
+ |
+
+
+ |
+
+
+ |
+
+ sum
+ |
+
+
+ |
+
+
+
+
+
+ Here are some notable values for the Peer score:
+
+
+
+ |
+
+ Binary and
+
+ Multiple Choice
+
+ questions
+ |
+
+ Continuous
+
+ questions
+ |
+
+
+
+
+ |
+ Best possible Peer score on Metaculus
+ |
+
+ +996
+ |
+
+ +408
+ |
+
+
+ |
+ Worst possible Peer score on Metaculus
+ |
+
+ -996
+ |
+
+ -408
+ |
+
+
+ |
+ Median Peer empirical score
+ |
+
+ +2
+ |
+
+ +3
+ |
+
+
+ |
+ Average Peer empirical score
+ |
+
+ 0*
+ |
+
+ 0*
+ |
+
+
+
+ *The average Peer score is 0 by definition.
+
+ Theoretically, binary scores can be infinitely negative, and continuous
+ scores can be both infinitely positive and infinitely negative. In
+ practice, Metaculus restricts binary predictions to be between 0.1% and
+ 99.9%, and continuous pdfs to be between 0.01 and ~35, leading to the
+ scores above.
+
+
+ The "empirical scores" are based on all scores observed on all
+ resolved Metaculus questions, as of November 2023.
+
+
+ Note that the above describes the Peer score at a single point in time.
+ Metaculus scores are time-averaged over the lifetime of the question,
+ see{" "}
+
+ Do all my predictions on a question count toward my score?
+
+ .
+
+ You can expand the section below for more details and maths.
+
+
+
+
+ Why is the Peer score of the Community Prediction positive?
+
+
+ The Peer score measures
+ whether a forecaster was on average better than other forecasters. It is
+ the difference between the forecaster's{" "}
+ log score and the average of
+ all other forecasters' log scores. If you have a positive Peer
+ score, it means your log score was better than the average of all other
+ forecasters' log scores.
+
+
+ The Community Prediction is a
+ time-weighted median of all forecasters on the question. Like most
+ aggregates, it is better than most of the forecasters it feeds on: it is
+ less noisy, less biased, and updates more often.
+
+
+ Since the Community Prediction is better than most forecasters, it
+ follows that its score should be higher than the average score of all
+ forecasters. And so its Peer score is positive.
+
+
+
+
+
+ Do all my predictions on a question count toward my score?
+
+
+ Yes. Metaculus uses time-averaged scores, so all your predictions count,
+ proportional to how long they were standing. An example goes a long way
+ (we will use the Baseline score for simplicity, but the same logic
+ applies to any score):
+
+
+ A binary question is open 5 days, then closes and resolves Yes. You
+ start predicting on the second day, make these predictions, and get
+ those scores:
+
+
+
+
+
+ |
+ Day 1 |
+ Day 2 |
+ Day 3 |
+ Day 4 |
+ Day 5 |
+
+ Average
+ |
+
+
+
+
+ |
+ Prediction
+ |
+ |
+ 40% |
+ 70% |
+ |
+ 80% |
+ N/A |
+
+
+ |
+ Baseline score
+ |
+ 0 |
+ -32 |
+ +49 |
+ +49 |
+ +68 |
+ +27 |
+
+
+
+
+ Some things to note:
+
+ -
+ Before you predict, your score is considered to be 0 (this is true for
+ all scores based on the log score). This means that if you believe you
+ can do better than 0, you should predict as early as possible.
+
+ -
+ You have a score for Day 4, despite not having predicted that day.
+ This is because your predictions stay standing until you update them,
+ so on Day 4 you were scored on your Day 3 prediction. On Day 5 you
+ updated to 80%, so you were scored on that.
+
+ -
+ This example uses days, but your Metaculus scores are based on exact
+ timestamped predictions, so a prediction left standing for 1 hour will
+ count for 1/24th of a prediction left standing for a day, etc.
+
+
+
+
+ Lastly, note that scores are always averaged for every instant between
+ the Open date and (scheduled) Close date of the question. If a question
+ resolves early (i.e. before the scheduled close date), then scores are
+ set to 0 between the resolution date and scheduled close date, and still
+ count in the average. This ensures alignment of incentives, as explained
+ in the section{" "}
+
+ Why did I get a small score when I was right?
+ {" "}
+ below.
+
+
+
+
+ Can I get better scores by predicting extreme values?
+
+
+ Metaculus uses proper scores (see{" "}
+
+ What is a proper scoring rule?
+
+ ), so you cannot get a better score (on average) by making predictions
+ more extreme than your beliefs. On any question, if you want to maximize
+ your expected score, you should predict exactly what you believe.
+
+
+ Let's walk through a simple example using the Baseline score.
+ Suppose you are considering predicting a binary question. After some
+ thought, you conclude that the question has 80% chance to resolve Yes.
+
+
+ If you predict 80%, you will get a score of +68 if the question resolves
+ Yes, and -132 if it resolves No. Since you think there is an 80% chance
+ it resolves Yes, you expect on average a score of
+
+
+
+
+
+ If you predict 90%, you will get a score of +85 if the question resolves
+ Yes, and -232 if it resolves No. Since you think there is an 80% chance
+ it resolves Yes, you expect on average a score of
+
+
+
+
+
+ So by predicting a more extreme value, you actually lower the score you
+ expect to get (on average!).
+
+ Here are some more values from the same example, tabulated:
+
+
+
+
+ |
+ Prediction
+ |
+
+ Score if Yes
+ |
+
+ Score if No
+ |
+
+ Expected score
+ |
+
+
+
+
+ | 70% |
+ +48 |
+ -74 |
+ +24 |
+
+
+ | 80% |
+ +68 |
+ -132 |
+ +28 |
+
+
+ | 90% |
+ +85 |
+ -232 |
+ +21 |
+
+
+ | 99% |
+ +99 |
+ -564 |
+ -34 |
+
+
+
+
+
+ The 99% prediction gets the highest score when the question resolves
+ Yes, but it also gets the lowest score when it resolves No. This is why,
+ on average, the strategy that maximizes your score is to predict what
+ you believe. This is one of the reasons why looking at scores on
+ individual questions is not very informative; only aggregate over many
+ questions are interesting!
+
+
+
+
+ Why did I get a small score when I was right?
+
+
+ To make sure incentives are aligned, Metaculus needs to ensure that our
+ scores are proper. We also time-average scores.
+
+
+ This has a counter-intuitive consequence: when a question resolves
+ before its intended close date, the times between resolution and close
+ date need to count in the time-average, with scores of 0. We call this
+ "score truncation".
+
+
+ An example is best: imagine the question "Will a new human land on
+ the Moon before 2030?". It can either resolve Yes before 2030
+ (because someone landed on the Moon), or it can resolve No in 2030. If
+ we did not truncate scores, you could game this question by predicting
+ close to 100% in the beginning (since it can only resolve positive
+ early), and lower later (since it can only resolve negative at the end).
+
+
+ Another way to think about this is that if a question lasts a year, then
+ each day (or in fact each second) is scored as a separate question. To
+ preserve properness, it is imperative that each day is weighted the same
+ in the final average (or at least that the weights be decided in
+ advance). From this perspective, not doing truncation is equivalent to
+ retroactively giving much more weight to days before the question
+ resolves, which is not proper.
+
+
+ You can read a worked example with maths by expanding the section below.
+
+
+
+
+
+
+ What are the legacy scores?
+
+
+
+ What is the Relative score?
+
+
+ The Relative score compares a prediction to the median of all other
+ predictions on the same question. If it is positive, the prediction was
+ (on average) better than the median. If it is negative it was worse than
+ the median.
+
+ It is based on the log score, with the formula:
+
+
+ Where is the prediction being scored
+ and is the median of all other
+ predictions on that question.
+
+
+ As of late 2023, the Relative score is in the process of being replaced
+ by the Peer score, but it is
+ still used for many open tournaments.
+
+
+
+ What is the coverage?
+
+
+ The Coverage measures for what proportion of a question's lifetime
+ you had a prediction standing.
+
+
+ If you make your first prediction right when the question opens, your
+ coverage will be 100%. If you make your first prediction one second
+ before the question closes, your coverage will be very close to 0%.
+
+
+ The Coverage is used in tournaments, to incentivize early predictions.
+
+
+
+ What are Metaculus points?
+
+
+ Metaculus points were used as the main score on Metaculus until late
+ 2023.
+
+
+ You can still find the rankings based on points{" "}
+ here.
+
+
+ They are a proper score, based on the log score. They are a mixture of a
+ Baseline-like score and a Peer-like score, so they reward both beating
+ an impartial baseline and beating other forecasters.
+
+ For full mathematical details, expand the section below.
+
+
+
+
+ Tournaments
+
+
+
+
+ How are my tournament Score, Take, Prize, and Rank calculated?
+
+
+ This scoring method was introduced in March 2024. It is based on the{" "}
+ Peer scores described above.
+
+
+ Your rank in the tournament is determined by the sum of your Peer scores
+ over all questions in the tournament (you get 0 for any question you
+ didn’t forecast).
+
+
+ The share of the prize pool you get is proportional to that same sum of
+ Peer scores, squared. If the sum of your Peer scores is negative, you
+ don’t get any prize.
+
+
+
+
+
+
+
+ For a tournament with a sufficiently large number of independent
+ questions, this scoring method is effectively{" "}
+
+ proper
+ {" "}
+ for the top quartile of forecasters. While there are small imperfections
+ for forecasters near a 0 Peer score for which they might win a tiny bit
+ of money by extremizing their forecasts, we believe this is an edge case
+ that you can safely ignore. In short, you should predict your true
+ belief on any question.
+
+
+ Taking the square of your Peer scores incentivizes forecasting every
+ question and forecasting early. Don’t forget to Follow a
+ tournament to be notified of new questions.
+
+
+
+
+ How are my (legacy) tournament Score, Coverage, Take, Prize, and Rank
+ calculated?
+
+
+
+ This scoring method was superseded in March 2024 by the New Tournament
+ Score described above. It is still in use for tournaments that
+ concluded before March 2024 for some tournaments that were in flight
+ then.
+
+
+
+ Your tournament Score is the sum of your Relative scores over all
+ questions in the tournament. If, on average, you were better than the
+ Community Prediction, then it will be positive; otherwise, it will be
+ negative.
+
+
+ Your tournament Coverage is the average of your coverage on each
+ question. If you predicted all questions when they opened, your Coverage
+ will be 100%. If you predicted all questions halfway through, or if you
+ predicted half the questions when they opened, your Coverage will be
+ 50%.
+
+
+ Your tournament Take is the exponential of your Score, times your
+ Coverage:
+
+
+
+ Your Prize is how much money you earned on that tournament. It is
+ proportional to your take and is equal to your Take divided by the sum
+ of all competing forecasters' Takes.
+
+
+ Your Rank is simply how high you were in the leaderboard, sorted by
+ Prize.
+
+
+ The higher your Score and Coverage, the higher your Take will be. The
+ higher your Take, the more Prize you'll receive, and the higher
+ your Rank will be.
+
+
+
+
+ What are the Hidden Period and Hidden Coverage Weights?
+
+
+ The Community Prediction is on average much better than most
+ forecasters. This means that you could get decent scores by just copying
+ the Community Prediction at all times. To prevent this, many tournament
+ questions have a significant period of time at the beginning when the
+ Community Prediction is hidden. We call this time the Hidden Period.
+
+
+ To incentivize forecasting during the hidden period, questions sometimes
+ are also set up so that the coverage you accrue during the Hidden Period
+ counts more. For example, the Hidden Period could count for 50% of the
+ question coverage, or even 100%. We call this percentage the Hidden
+ Period Coverage Weight.
+
+
+ If the Hidden Period Coverage Weight is 50%, then if you don't
+ forecast during the hidden period your coverage will be at most 50%,
+ regardless of how long the hidden period lasted.
+
);
}
diff --git a/front_end/src/app/(main)/press/components/DisclosureSection.tsx b/front_end/src/app/(main)/press/components/DisclosureSection.tsx
index ec3f9bbf98..2b56661b6a 100644
--- a/front_end/src/app/(main)/press/components/DisclosureSection.tsx
+++ b/front_end/src/app/(main)/press/components/DisclosureSection.tsx
@@ -2,295 +2,274 @@
import React from "react";
-import DisclosureItem from "./DisclosureItem"; // Assuming Disclosure is in the same directory
+import StyledDisclosure from "../../components/styled_disclosure";
const DisclosureSection = () => {
return (
-
-
- Forecasting is the practice of putting explicit probabilities,
- dates, and numbers on future events—calculating odds via both
- models and human judgment.
-
-
- Although such estimates are subjective, a substantial body of
- scientific research demonstrates two points: that people who
- forecast can improve, with some becoming expertly calibrated; and
- that aggregating across many diverse opinions produces more
- accurate forecasts than even the best individuals forecasting
- alone. The resulting predictions give us a clearer sense of what
- tomorrow will look like, allowing us to make better decisions
- today, just as a good meteorologist can help us decide whether to
- carry an umbrella.
+
+
+ Forecasting is the practice of putting explicit probabilities, dates,
+ and numbers on future events—calculating odds via both models and
+ human judgment.
+
+
+ Although such estimates are subjective, a substantial body of
+ scientific research demonstrates two points: that people who forecast
+ can improve, with some becoming expertly calibrated; and that
+ aggregating across many diverse opinions produces more accurate
+ forecasts than even the best individuals forecasting alone. The
+ resulting predictions give us a clearer sense of what tomorrow will
+ look like, allowing us to make better decisions today, just as a good
+ meteorologist can help us decide whether to carry an umbrella.
+
+
+ Of course, subjects like geopolitics are not like meteorology. Yet,
+ the scientific validity of human forecasting holds true even when it
+ comes to subjects with a high degree of uncertainty, like the war
+ between Russia and Ukraine. In fact,{" "}
+
+ research
+ {" "}
+ by University of Pennsylvania psychologists Philip Tetlock and Barbara
+ Mellers found that the aggregated geopolitical predictions of top
+ forecasters were{" "}
+
+ more accurate
+ {" "}
+ than those of CIA analysts with access to classified information.
+
+
+ Because forecasts are expressed probabilistically, we can rarely say
+ that a particular forecast was “right” or “wrong,”. Rather, we score
+ forecasts mathematically by comparing forecasts to outcomes over a
+ large body of questions. This enables us to determine how
+ “well-calibrated” any given forecaster is—i.e., do things that they
+ believe are 70% likely actually happen 70% of the time-as well as
+ track the record of the whole Metaculus community over thousands of
+ questions.
+
+
+ Metaculus forecasts are well-calibrated. They provide greater
+ visibility into the future.
+
+
+
+
+ Metaculus is an online forecasting platform and aggregation engine
+ working to improve human reasoning and coordination on topics of
+ global importance. As a Public Benefit Corporation, Metaculus provides
+ decision support based on these forecasts to a variety of institutions
+ (learn more).
+
+
+ Metaculus features questions on a wide range of topics, with a
+ particular focus on{" "}
+ artificial intelligence,{" "}
+ biosecurity,{" "}
+
+ climate change
+
+ , and nuclear risk.
+
+
+
+
+ No, Metaculus is a forecasting platform and aggregation engine. Like
+ prediction markets, we collect people's forecasts and reward them
+ for accuracy. But in prediction markets, participants place bets
+ against each other for financial rewards, can only win insofar as
+ someone else loses. Metaculus forecasters are incentivized only to
+ make the most accurate forecasts, and they often collaborate to do so.
+
+
+ Prediction markets produce forecasts via where the betting market
+ settles. Metaculus explicitly aggregates everyone's forecasts
+ together using algorithms we refine over time. We produce a
+ time-weighted median, the "Community Prediction," as well as
+ the more sophisticated "Metaculus Prediction".
+
+
+ Prediction market bettors can produce accurate forecasts because they
+ have “skin in the game.” But{" "}
+
+ research
+ {" "}
+ shows that forecasting platforms like Metaculus often outperform
+ prediction markets, while avoiding many of the downsides of market
+ incentives that lead to regulators{" "}
+
+ restricting their activity.
+ {" "}
+ And critically, the research, methods, and reasoning that Metaculus
+ forecasters produce are themselves valuable, as seen both in question
+ comments as well as in the{" "}
+ Metaculus Journal.
+
+
+
+
+
+
+ 1. Complement expert analysis.
-
- Of course, subjects like geopolitics are not like meteorology.
- Yet, the scientific validity of human forecasting holds true even
- when it comes to subjects with a high degree of uncertainty, like
- the war between Russia and Ukraine. In fact,{" "}
+
+ News stories often rely on expert analysis to put developments in
+ context and to anticipate the future course of events.
+ Unfortunately, experts frequently offer imprecisely worded
+ forecasts—e.g., “If the United States provides it with F-16s,
+ there is a real possibility that Ukraine will regain control of
+ its airspace”—and{" "}
research
{" "}
- by University of Pennsylvania psychologists Philip Tetlock and
- Barbara Mellers found that the aggregated geopolitical predictions
- of top forecasters were{" "}
+ shows that people interpret “real possibility” to mean anything
+ between 20% and 80%, confusing both journalists and their
+ audiences. Probabilistic forecasts eliminate this problem.
+
+
+ What’s more, to the extent that experts do put precise
+ probabilities on future events, their track record is poor. One of
+ Tetlock’s{" "}
- more accurate
+ early findings
{" "}
- than those of CIA analysts with access to classified information.
-
-
- Because forecasts are expressed probabilistically, we can rarely
- say that a particular forecast was “right” or “wrong,”. Rather, we
- score forecasts mathematically by comparing forecasts to outcomes
- over a large body of questions. This enables us to determine how
- “well-calibrated” any given forecaster is—i.e., do things that
- they believe are 70% likely actually happen 70% of the time-as
- well as track the record of the whole Metaculus community over
- thousands of questions.
-
-
- Metaculus forecasts are well-calibrated. They provide greater
- visibility into the future.
-
- >
- }
- />
-
-
- Metaculus is an online forecasting platform and aggregation engine
- working to improve human reasoning and coordination on topics of
- global importance. As a Public Benefit Corporation, Metaculus
- provides decision support based on these forecasts to a variety of
- institutions (learn more).
-
-
- Metaculus features questions on a wide range of topics, with a
- particular focus on{" "}
- artificial intelligence,{" "}
- biosecurity,{" "}
-
- climate change
-
- , and nuclear risk.
-
- >
- }
- />
-
-
- No, Metaculus is a forecasting platform and aggregation engine.
- Like prediction markets, we collect people's forecasts and
- reward them for accuracy. But in prediction markets, participants
- place bets against each other for financial rewards, can only win
- insofar as someone else loses. Metaculus forecasters are
- incentivized only to make the most accurate forecasts, and they
- often collaborate to do so.
+ was that political experts are highly overconfident in their
+ predictions. In fact, although there is significant variance,
+ their aggregate forecasts perform little better than chance. By
+ contrast, Metaculus predictions perform significantly better than
+ chance.
- Prediction markets produce forecasts via where the betting market
- settles. Metaculus explicitly aggregates everyone's forecasts
- together using algorithms we refine over time. We produce a
- time-weighted median, the "Community Prediction," as
- well as the more sophisticated "Metaculus Prediction".
+ 2. Serve as a check on the conventional wisdom.
-
- Prediction market bettors can produce accurate forecasts because
- they have “skin in the game.” But{" "}
-
- research
- {" "}
- shows that forecasting platforms like Metaculus often outperform
- prediction markets, while avoiding many of the downsides of market
- incentives that lead to regulators{" "}
+
+ Forecasts can suggest that the conventional wisdom may be wrong
+ and that strongly held beliefs are worth questioning strongly. For
+ example, the conventional wisdom within the American military is
+ that China will invade Taiwan in the next few years. One four-star
+ general went so far as to{" "}
- restricting their activity.
+ suggest
{" "}
- And critically, the research, methods, and reasoning that
- Metaculus forecasters produce are themselves valuable, as seen
- both in question comments as well as in the{" "}
- Metaculus Journal.
-
- >
- }
- />
-
-
-
- 1. Complement expert analysis.
-
-
- News stories often rely on expert analysis to put developments
- in context and to anticipate the future course of events.
- Unfortunately, experts frequently offer imprecisely worded
- forecasts—e.g., “If the United States provides it with F-16s,
- there is a real possibility that Ukraine will regain control of
- its airspace”—and{" "}
-
- research
- {" "}
- shows that people interpret “real possibility” to mean anything
- between 20% and 80%, confusing both journalists and their
- audiences. Probabilistic forecasts eliminate this problem.
-
-
- What’s more, to the extent that experts do put precise
- probabilities on future events, their track record is poor. One
- of Tetlock’s{" "}
-
- early findings
- {" "}
- was that political experts are highly overconfident in their
- predictions. In fact, although there is significant variance,
- their aggregate forecasts perform little better than chance. By
- contrast, Metaculus predictions perform significantly better
- than chance.
-
-
- 2. Serve as a check on the conventional wisdom.
-
-
- Forecasts can suggest that the conventional wisdom may be wrong
- and that strongly held beliefs are worth questioning strongly.
- For example, the conventional wisdom within the American
- military is that China will invade Taiwan in the next few years.
- One four-star general went so far as to{" "}
-
- suggest
- {" "}
- there was a 100% chance of the PRC attempting to seize the
- island in 2025, leading to war with the United States. By
- contrast, the Metaculus forecast for the same time period is{" "}
- {/* */}
-
- 9%
-
- , not least because war between great powers is rare and because
- war between nuclear-armed great powers is unprecedented.
- Forecasts can provide an outside perspective on highly charged
- issues and serve as a check on inside thinking, adding nuance to
- stories.
-
-
- 3. Make sense of the big questions.
-
-
- Metaculus forecasts can help both journalists and their readers
- make sense of developments where there is tremendous uncertainty
- by breaking large, difficult-to-answer questions into smaller,
- more tractable ones.
-
-
- The future of artificial intelligence falls into this category,
- where the questions people are most interested in (e.g., “Will
- AI lead to a more utopian or a more dystopian future?”) are
- impossible to answer at this point. We can, however, provide
- forecasts on{" "}
- more targeted questions on AI
- safety, the regulation of AI, technical progress on AI, and the
- business of AI—all of which can help us better understand which
- direction we are headed in and how fast. Forecasting questions
- serve as clues as to what developments we should be paying
- particular attention to. And a{" "}
-
- thorough analysis
- {" "}
- of our track record on AI-related questions showed that
- Metaculus predictions offer clear and useful insights into the
- future of the field and its impacts.
-
-
- Metaculus forecasts can also identify where there have been
- significant changes in our anticipations of the future. For
- example, the drastic reduction in the forecast arrival date of
- transformative AI—from the early 2040s to the current
- distribution, centered around{" "}
- {/*
+ 9%
+
+ , not least because war between great powers is rare and because
+ war between nuclear-armed great powers is unprecedented. Forecasts
+ can provide an outside perspective on highly charged issues and
+ serve as a check on inside thinking, adding nuance to stories.
+
+
+ 3. Make sense of the big questions.
+
+
+ Metaculus forecasts can help both journalists and their readers
+ make sense of developments where there is tremendous uncertainty
+ by breaking large, difficult-to-answer questions into smaller,
+ more tractable ones.
+
+
+ The future of artificial intelligence falls into this category,
+ where the questions people are most interested in (e.g., “Will AI
+ lead to a more utopian or a more dystopian future?”) are
+ impossible to answer at this point. We can, however, provide
+ forecasts on{" "}
+ more targeted questions on AI
+ safety, the regulation of AI, technical progress on AI, and the
+ business of AI—all of which can help us better understand which
+ direction we are headed in and how fast. Forecasting questions
+ serve as clues as to what developments we should be paying
+ particular attention to. And a{" "}
+
+ thorough analysis
+ {" "}
+ of our track record on AI-related questions showed that Metaculus
+ predictions offer clear and useful insights into the future of the
+ field and its impacts.
+
+
+ Metaculus forecasts can also identify where there have been
+ significant changes in our anticipations of the future. For
+ example, the drastic reduction in the forecast arrival date of
+ transformative AI—from the early 2040s to the current
+ distribution, centered around{" "}
+ {/* */}
-
- Apr 29, 2033
-
- —was used by{" "}
-
- The Economist
- {" "}
- as a tangible example of how society’s expectations of AI are
- changing rapidly.
-
-
-
-
-
+
+ Apr 29, 2033
+
+ —was used by{" "}
+
+ The Economist
+ {" "}
+ as a tangible example of how society’s expectations of AI are
+ changing rapidly.
+
- }
- />
+
+
+
+
+
);
};
diff --git a/front_end/src/app/(main)/press/components/ReferenceSection.tsx b/front_end/src/app/(main)/press/components/ReferenceSection.tsx
index b303d763ab..2fdae3148b 100644
--- a/front_end/src/app/(main)/press/components/ReferenceSection.tsx
+++ b/front_end/src/app/(main)/press/components/ReferenceSection.tsx
@@ -2,101 +2,79 @@
import React from "react";
-import DisclosureItem from "./DisclosureItem"; // Assuming Disclosure is in the same directory
+import StyledDisclosure from "../../components/styled_disclosure";
const DisclosureSection = () => {
return (
-
-
- On Substack, Twitter, or most other social media:
-
-
- Simply paste the link to the Metaculus question URL, like{" "}
-
- www.metaculus.com/questions/17096/us-tracks-training-runs-by-2026
-
- , and the preview image with the graph will show up automatically.
-
-
- On Other Sites:
-
-
- On the question page, click “embed” at the top. Choose
- your theme, width, and height, then copy the iframe onto your
- site.
-
-
- If you'd prefer to have a static image rather than an embed
- that users can interact with, navigate to the URL of the embed,
- e.g.{" "}
-
- www.metaculus.com/questions/embed/17096/
-
- . Then save the image, generally via right click + “save
- image as”, and upload it to your preferred site.
-
-
- >
- }
- />
-
-
- Via the API:
-
-
- See the full API documentation. You can also
- see the raw question data in your browser, like{" "}
-
- www.metaculus.com/api2/questions/17096/
-
- .
-
-
- Download Question Data:
-
-
- Select the ‘...’ menu on the question page, and click
- “Download CSV.” (Only available on questions with a
- critical mass of predictions.) If you have more expansive data
- needs, please reach out to{" "}
-
- christian@metaculus.com
- {" "}
- and we can construct a custom dataset for you.
-
- >
- }
- />
-
-
- There are thousands of Metaculus questions. You can search on{" "}
- the main feed by topic or keyword. Our AI
- questions can be found here.
-
-
- If you can't find what you're looking for, or want to
- suggest a question to forecast, reach out to{" "}
-
- christian@metaculus.com
- {" "}
- .
-
- >
- }
- />
+
+
+ On Substack, Twitter, or most other social media:
+
+
+ Simply paste the link to the Metaculus question URL, like{" "}
+
+ www.metaculus.com/questions/17096/us-tracks-training-runs-by-2026
+
+ , and the preview image with the graph will show up automatically.
+
+
+ On Other Sites:
+
+
+ On the question page, click “embed” at the top. Choose
+ your theme, width, and height, then copy the iframe onto your site.
+
+
+ If you'd prefer to have a static image rather than an embed that
+ users can interact with, navigate to the URL of the embed, e.g.{" "}
+
+ www.metaculus.com/questions/embed/17096/
+
+ . Then save the image, generally via right click + “save image
+ as”, and upload it to your preferred site.
+
+
+
+
+
+ Via the API:
+
+
+ See the full API documentation. You can also see
+ the raw question data in your browser, like{" "}
+
+ www.metaculus.com/api2/questions/17096/
+
+ .
+
+
+ Download Question Data:
+
+
+ Select the ‘...’ menu on the question page, and click
+ “Download CSV.” (Only available on questions with a
+ critical mass of predictions.) If you have more expansive data needs,
+ please reach out to{" "}
+ christian@metaculus.com{" "}
+ and we can construct a custom dataset for you.
+
+
+
+
+ There are thousands of Metaculus questions. You can search on{" "}
+ the main feed by topic or keyword. Our AI
+ questions can be found here.
+
+
+ If you can't find what you're looking for, or want to
+ suggest a question to forecast, reach out to{" "}
+ christian@metaculus.com .
+
+
);
};
diff --git a/front_end/src/components/math_jax_content.tsx b/front_end/src/components/math_jax_content.tsx
index 7b7daf5ecc..8d606798c7 100644
--- a/front_end/src/components/math_jax_content.tsx
+++ b/front_end/src/components/math_jax_content.tsx
@@ -1,17 +1,28 @@
"use client";
import { MathJax, MathJaxContext } from "better-react-mathjax";
import React, { FC } from "react";
+import dynamic from "next/dynamic";
type Props = {
content: string;
+ block?: boolean;
};
-const MathJaxContent: FC = ({ content }) => {
+const MathJaxContent: FC = ({ content, block = false }) => {
return (
- {content}
+
+ {content}
+
);
};
-export default MathJaxContent;
+// Exporting the component using dynamic import with ssr: false
+export default dynamic(() => Promise.resolve(MathJaxContent), {
+ ssr: false,
+});
From 0b98936739e6afc832c59a61c58a53989676e564 Mon Sep 17 00:00:00 2001
From: aseckin
Date: Wed, 28 Aug 2024 23:29:01 +0200
Subject: [PATCH 5/8] Added Medals FAQ page
---
.../src/app/(main)/help/medals-faq/page.tsx | 441 ++++++++++++++++++
1 file changed, 441 insertions(+)
create mode 100644 front_end/src/app/(main)/help/medals-faq/page.tsx
diff --git a/front_end/src/app/(main)/help/medals-faq/page.tsx b/front_end/src/app/(main)/help/medals-faq/page.tsx
new file mode 100644
index 0000000000..32f0526aa8
--- /dev/null
+++ b/front_end/src/app/(main)/help/medals-faq/page.tsx
@@ -0,0 +1,441 @@
+import PageWrapper from "../../components/pagewrapper";
+import MathJaxContent from "@/components/math_jax_content";
+
+export const metadata = {
+ title: "Medals FAQ | Metaculus",
+ description:
+ "Learn about Metaculus medals, how they are awarded for forecasting accuracy, comment quality, and question writing. Understand the h-index, medal tiers, and eligibility criteria.",
+};
+
+export default function MedalsFAQ() {
+ return (
+
+ Medals FAQ
+
+ Below are Frequently Asked Questions (and answers!) about medals. The
+ general FAQ is here, and the medals FAQ is{" "}
+ here.
+
+
+
+
+
+
+
+
+ Medals reward Metaculus users for excellence in forecasting accuracy,
+ insightful comment writing, and engaging question writing.
+
+
+
+ Medals are awarded based on a user's placement in the{" "}
+ Leaderboards. There are separate
+ leaderboards for each medal category (
+ Peer Accuracy,{" "}
+ Baseline Accuracy,{" "}
+ Comments, and{" "}
+ Question Writing), and each
+ leaderboard is further separated into time periods. Medals are also
+ awarded for placement in each{" "}
+ Tournament's
+ leaderboard.
+
+
+
+ A medal's tier (gold, silver or bronze) is based on a user's
+ rank compared to other users, with only the top 1% earning Gold medals.
+
+
+
+ So no one gets an unfair advantage, only public content (questions,
+ comments, tournaments) counts for medals. If you are invited to a
+ private tournament, your activity there will not count toward any medal.
+ We have also decided the three Beginner Tournaments (
+ 1,{" "}
+ 2,{" "}
+ 3) would not award
+ medals, since that would be unfair to veteran forecasters who were
+ actively discouraged from participating.
+
+
+
+ Medals appear in the Leaderboards and in
+ user profiles.
+
+
+
+
+ What are Baseline Accuracy medals?
+
+
+ The Baseline Accuracy medals reward accurate predictions on many
+ questions.
+
+
+ Users are ranked by the sum of their{" "}
+ Baseline scores over all
+ questions in the Time Period.
+
+
+
+
+ What are Peer Accuracy medals?
+
+
+ The Peer Accuracy medals reward accurate predictions compared to others,
+ and do not require forecasting a large number of questions.
+
+
+ Forecasters are ranked by the sum of their{" "}
+ Peer scores, divided by the
+ sum of their Coverages over all
+ questions in the Time Period. This
+ creates a weighted average, where each prediction is counted
+ proportionally to how long it was standing.
+
+
+ If the forecaster has a total coverage below 30 in a particular time
+ period (e.g. they predicted 20 questions with 100% coverage, or 50
+ questions with 50% coverage), then their average score on the
+ leaderboard will include (30 - total coverage) questions with a 0 score.
+ This makes it unlikely that a user wins a medal by getting lucky on a
+ single question.
+
+
+ Before 2024, the Peer accuracy was slightly different. The forecaster
+ score was the average of their Peer scores, not taking Coverage into
+ account. This caused some incentives problems, see{" "}
+
+ here
+ {" "}
+ for details. The initial handicap was also 40 instead of the current 30.
+
+
+
+
+ What are tournament medals?
+
+
+ Tournament medals are awarded based on a user's rank on a
+ tournament leaderboard. The top 1% get gold, top 2% silver, and top 5%
+ bronze.
+
+
+ The three Beginner Tournaments (
+ 1,{" "}
+ 2,{" "}
+ 3) will not award medals,
+ since that would be unfair to veteran forecasters who were actively
+ discouraged from participating.
+
+
+
+
+
+ A Comments medal is awarded for writing valuable comments, with a
+ balance between quantity and quality.
+
+
+
+ Users are ranked by the{" "}
+ h-index of upvotes on their
+ comments made during the{" "}
+ Time Period.
+
+
+
+
+ What are Question Writing medals?
+
+
+ The Question Writing medals reward writing engaging questions, with a
+ balance between quantity and quality.
+
+
+
+ Users are ranked by the{" "}
+ h-index of the number of
+ forecasters who predicted on their authored questions in the{" "}
+ Time Period.
+ Because there are few questions but many forecasters, the number of
+ forecasters is divided by 10 before being used in the h-index.
+
+
+
+ All co-authors on a question receive full credit, i.e. they are treated
+ the same as if they had authored the question alone.
+
+
+
+ Additionally, a single question may contribute to medals over many
+ years, not just the year it was written. If a question receives
+ predictions from 200 unique forecasters every year, then the author
+ receives credit for those 200 forecasters every year.
+
+
+
+
+ What are the Times Periods for Comments & Question writing medals?
+
+
+ Comments and Question writing medals are awarded annually, based on the
+ # of upvotes on your comments and # of forecasters on your questions in
+ a given calendar year.
+
+
+
+ For example, if you wrote 20 long-term questions in 2016 that each
+ attracted 200 forecasters in every calendar year then your score for
+ Question Writing would be 20 in every year after 2016. Even though you
+ didn't write any questions in 2017, the engagement that your
+ questions attracted in 2017 makes you eligible for a 2017 medal. Said
+ another way, a great long-term question can contribute to many medals.
+
+
+
+
+ What are the Time Periods for Baseline & Peer medals?
+
+
+ Time Periods for Accuracy medals serve two main purposes. They ensure a
+ periodic fair starting line on January 1, at which point long-time and
+ new forecasters are on equal grounds. They also group questions with
+ similar durations together, so it is easier to separate long-term and
+ short-term forecasting skill.
+
+
+
+ A Time Period for the Baseline and Peer medals consists of a Duration
+ (1, 2, 5, 10… years), a start year and an end year. The end date for a
+ time period is December 31 of the end year. The start date for a time
+ period is January 1 of the start year. So, a 5 year medal covering
+ 2016–2020 has a start date of Jan 1, 2016 and an end date of Dec 31,
+ 2020.
+
+
+
+ The Time Period determines which questions are included in a medal
+ calculation:
+
+
+ -
+ A question only belongs to exactly 1 Time Period, so it only
+ contributes to 1 Baseline medal and to 1 Peer medal.
+
+ -
+ A question is assigned to the shortest time period that satisfies the
+ following:
+
+
+ - The question opened after the Time Period start date.
+ -
+ The question is scheduled to close before the Time
+ Period's end date.{" "}
+
+ (There is a 3 day buffer here: some questions were written to
+ close on Jan 1 and naturally belong in the prior year. Also,
+ sometimes time zones make it unclear which date to use.)
+
+
+ -
+ The question resolves before the Time Period's end date + a
+ buffer of 100 days.{" "}
+
+ (This allows time for data sources to become available. For
+ instance, a question about 2022 GDP naturally should count toward
+ the 2022 medal, but the final economic report is often published
+ ~90 days after the year end.)
+
+
+
+
+
+ Following the rules above, almost all questions are assigned to their
+ Time Period before forecasting begins. On rare occasions, a question
+ will fail to be resolved before the end of the 100-day buffer: by the
+ rules above it is automatically assigned to the next higher Duration in
+ which it fits.
+
+
+ Note: If a question closes early it remains in its originally assigned
+ time period. This is important to ensure that an optimistic forecaster
+ does not gain an advantage. For example, imagine ten 5-year questions
+ that can either resolve Yes this week (with 50% probability), or resolve
+ No after the full 5 years. Starting next week, an optimist looks
+ misleadingly good: they predicted 99% on all of the 5 questions that
+ resolved, and they all resolved Yes. After the full 5 years they
+ correctly look very bad: of the 10 questions they predicted 99% on, only
+ 5 resolved Yes. Keeping questions in their initial time period ensures
+ that optimists don't get undue early credit.
+
+
+
+ What are h-indexes?
+
+
+ An h-index is a
+ metric commonly used in academia to measure the quantity and quality of
+ researcher publications. If a researcher has an h-index of N it means
+ that they have published at least N papers that each individually have
+ at least N citations.
+
+
+ We use h-indexes for the Comments medals (number of upvotes per comment)
+ and for the Question Writing medals (tens of forecasters per question).
+
+
+ Traditional h-indexes are integers. To break ties, we use a fractional
+ h-index, described below.
+
+
+
+
+ What is the fractional h-index?
+
+
+ The fractional h-index is like the standard h-index, but with an added
+ fractional part that measures progress toward the next higher h-index
+ value.
+
+
+ Imagine that you have exactly 2 comments with exactly 2 upvotes. Your
+ h-index is therefore 2. To reach an h-index of 3, you need to receive 1
+ more upvote on each of your 2 existing comments (a total of 2 more
+ upvotes) and you need to write a new comment that receives 3 upvotes. In
+ total you need 5 more upvotes to reach an h-index of 3.
+
+
+ Imagine one of your comments receives 1 new upvote, and you write a
+ comment that receives 2 new upvotes. Your fractional h-index is then:
+
+ 2 + (1 + 2) / 5 = 2.6
+ In general, the formula is:
+
+
+ Where is your integer h-index,
+ and is the number of upvotes
+ on your i-th most upvoted comment.
+
+
+
+
+ How are medal tiers determined?
+
+
+ Medals are awarded based on a user's rank within a Category for a
+ Time Period, or in a Tournament:
+
+
+ - Top 1% = Gold
+ - Top 2% = Silver
+ - Top 5% = Bronze
+
+
+ The denominators for the percentages are the number of users who have
+ made any contribution toward that medal. Specifically, the denominators
+ are:
+
+
+ -
+ Baseline & Peer Accuracy: the number of users who made a forecast
+ on any public question in the time period.
+
+ -
+ Tournament: the number of users who made a forecast on any question in
+ the tournament.
+
+ -
+ Comments: the number of users who made a public comment in the time
+ period.
+
+ -
+ Question writing: the number of users who authored (or co-authored) a
+ public question in the time period.
+
+
+
+ To make the leaderboards more interesting and fair, we also enforce the
+ following rules:
+
+
+ -
+ The first, second, and third place finishers always receive (at least)
+ a gold, silver, and bronze medals, in that order.
+
+ - If two users are tied, they always get the same medal.
+ -
+ Metaculus staff are ineligible for medals for contributions they made
+ during the time they were on staff.
+
+
+
+
+ Are medals forever?
+
+
+ In general, yes! We designed the medal system so that once a medal is
+ awarded, it never goes away.
+
+
+ However, when we discover an error - an incorrectly resolved question or
+ a bug in the code - we plan to correct the error and medals could shift,
+ hopefully only very slightly. We believe this will be a rare occurrence,
+ but it may happen. The spirit of Metaculus is to be accurate.
+
+
+ );
+}
From d45545ff7841ad593d751f10b9b99442e1c9d93e Mon Sep 17 00:00:00 2001
From: aseckin
Date: Thu, 29 Aug 2024 00:05:01 +0200
Subject: [PATCH 6/8] Added Markdown Help page
---
.../src/app/(main)/components/pagewrapper.tsx | 2 +-
.../src/app/(main)/help/markdown/page.tsx | 413 ++++++++++++++++++
2 files changed, 414 insertions(+), 1 deletion(-)
create mode 100644 front_end/src/app/(main)/help/markdown/page.tsx
diff --git a/front_end/src/app/(main)/components/pagewrapper.tsx b/front_end/src/app/(main)/components/pagewrapper.tsx
index 4f2fc9e3b5..dd13621cb0 100644
--- a/front_end/src/app/(main)/components/pagewrapper.tsx
+++ b/front_end/src/app/(main)/components/pagewrapper.tsx
@@ -6,7 +6,7 @@ interface PageWrapper {
const PageWrapper: React.FC = ({ children }) => {
return (
-
+
{children}
);
diff --git a/front_end/src/app/(main)/help/markdown/page.tsx b/front_end/src/app/(main)/help/markdown/page.tsx
new file mode 100644
index 0000000000..d13a5a5114
--- /dev/null
+++ b/front_end/src/app/(main)/help/markdown/page.tsx
@@ -0,0 +1,413 @@
+import PageWrapper from "../../components/pagewrapper";
+import MathJaxContent from "@/components/math_jax_content";
+
+export const metadata = {
+ title: "Markdown Syntax | Metaculus",
+ description:
+ "Learn how to use Markdown and MathJax on Metaculus. Discover syntax for links, headers, lists, tables, code, and equations to enhance your comments and questions.",
+};
+
+export default function MedalsFAQ() {
+ return (
+
+ Markdown Syntax
+
+
+ When adding comments or suggesting questions, you can take advantage of{" "}
+ Markdown{" "}
+ syntax to add links, emphasis, and headers. Additionally, you can add
+ mathematical equations via MathJax
+ , which will convert{" "}
+
+ LaTeX syntax
+ {" "}
+ into nicely typeset equations. We closely follow the{" "}
+
+ official Markdown syntax
+
+ , so that's the best place to look for a thorough explanation of
+ how the system works. We provide a brief overview of the most common
+ uses here.
+
+
+
+
+ Inline elements
+
+
+ Links can be produced using a{" "}
+ [link title](http://and-link-address.com) or by surrounding
+ a link with < and >, like{" "}
+ <http://www.example.com>. There are a number of
+ shortcuts to make your life easier if you keep repeating the same link
+ (see the{" "}
+ docs),
+ but these will cover 90% of the use cases.
+
+
+ Asterisks (*) and underscores (_) will both _italicize_ text,
+ and two asterisks will make the text **bold**.
+ Back-ticks denote fixed-width text. If you want small text,
+ you can wrap it in a literal{" "}
+ <small>html tag</small>. Special characters (
+ *_{}#+-.!\) can be escaped using a backslash, like{" "}
+ \*, if they would otherwise be converted into a markdown
+ element.
+
+
+ We also allow a limited subset of HTML tags, which you can mix with
+ markdown syntax if you want. These include: <a>,{" "}
+ <p>, <em>,{" "}
+ <strong>, <small>,{" "}
+ <ol>, <ul>,{" "}
+ <li>, <br>,{" "}
+ <code>, <pre>,{" "}
+ <blockquote>, <aside>,{" "}
+ <div>, <h1>,{" "}
+ <h2>, <h3>,{" "}
+ <h4>, <h5>,{" "}
+ <h6>, <math-inline>,{" "}
+ <math-display>, <hr>,{" "}
+ <table>, <thead>,{" "}
+ <tbody>, <tr>,{" "}
+ <th>, <td>,{" "}
+ <del>, <sup>,{" "}
+ <sub>.
+
+
+
+
+ Math
+
+
+ We supplement Markdown with{" "}
+ MathJax equation processing.
+ Mathematical formatting works by placing your equation between{" "}
+ \( and \) (for inline equations) or{" "}
+ \[ and \] (for displayed equations). More
+ complicated equations can be put in an align environment,
+ like so
+
+
+ {`\\begin{align}
+ \\log_2 \\left ( \\frac{p}{0.5} \\right ) &= \\log_2 \\left ( p \\right ) + 1 \\\\
+ \\log_2 \\left ( \\frac{p}{0.5} \\right ) &= \\frac{\\log(p) - \\log(0.5)}{\\log(1) - \\log(0.5)}
+\\end{align}`}
+
+ producing
+
+
+
+
+ Headers are easiest to add using hash marks, for example
+
+ {`# Primary header
+## Secondary header
+##### Fifth-level header`}
+
+
+ Please use headers in comments sparingly!
+
+
+ Code
+
+ Big chunks of code can be wrapped in three back-ticks. For example:
+
+ ```
+ {`
+def hello_world():
+ print('hello!')`}
+
+ ```
+
+
+
+
+ Quotes
+
+
+ If you want to quote someone, precede each line with a >
+ :
+
+
+ {`> This is a blockquote with two paragraphs. Lorem ipsum dolor sit amet,
+> consectetuer adipiscing elit. Aliquam hendrerit mi posuere lectus.
+> Vestibulum enim wisi, viverra nec, fringilla in, laoreet vitae, risus.
+
+> Donec sit amet nisl. Aliquam semper ipsum sit amet velit. Suspendisse
+> id sem consectetuer libero luctus adipiscing.`}
+
+ which would produce:
+
+
+ This is a blockquote with two paragraphs. Lorem ipsum dolor sit amet,
+ consectetuer adipiscing elit. Aliquam hendrerit mi posuere lectus.
+ Vestibulum enim wisi, viverra nec, fringilla in, laoreet vitae, risus.
+
+
+ Donec sit amet nisl. Aliquam semper ipsum sit amet velit. Suspendisse
+ id sem consectetuer libero luctus adipiscing.
+
+
+
+
+
+ Lists
+
+ Markdown can handle both ordered and unordered lists. For example,
+
+ {`1. First item
+2. Second item
+
+ Another paragraph in the second item. (Note the 4-spaces indentation.)
+
+ - Sublist item 1. (Note the 4-spaces indentation.)
+ - Sublist item 2.
+
+3. Third item.`}
+
+ produces:
+
+ - First item
+ -
+ Second item
+
+ Another paragraph in the second item. (Note the 4-spaces
+ indentation.)
+
+
+ - Sublist item 1. (Note the 4-spaces indentation.)
+ - Sublist item 2.
+
+
+ - Third item.
+
+
+ Unordered lists behave similarly, but use * or{" "}
+ + or - to denote new items.
+
+
+
+
+ Tables
+
+ We support simple tables of the form:
+
+ {`| Header 1 | Header 2 | ← headers
+|----------|----------| ← mandatory header separator
+| Cell 1 | Cell 2 | ← line 1
+| Cell 3 | Cell 4 | ← line 2`}
+
+
+ Columns are separated by the pipe character |, and each
+ line is a row. For example, this:
+
+
+ {`|Year | Predictions | Total |
+|-----|-------------|--------|
+|2015 | 500 | 500 |
+|2016 | 25500 | 26000 |
+|2017 | 21000 | 47000 |
+|2018 | 63000 | 110000 |
+|2019 | 50000 | 160000 |
+|2020 | 220000 | 380000 |`}
+
+ Will render as:
+
+
+
+
+ |
+ Year
+ |
+
+ Predictions
+ |
+
+ Total
+ |
+
+
+
+
+ |
+ 2015
+ |
+
+ 500
+ |
+
+ 500
+ |
+
+
+ |
+ 2016
+ |
+
+ 25500
+ |
+
+ 26000
+ |
+
+
+ |
+ 2017
+ |
+
+ 21000
+ |
+
+ 47000
+ |
+
+
+ |
+ 2018
+ |
+
+ 63000
+ |
+
+ 110000
+ |
+
+
+ |
+ 2019
+ |
+
+ 50000
+ |
+
+ 160000
+ |
+
+
+ |
+ 2020
+ |
+
+ 220000
+ |
+
+ 380000
+ |
+
+
+
+
+
+
+
+ Embeds
+
+
+ We allow <iframe> embeds from a limited list of
+ trusted sites, currently including:
+
+
+ - afdc.energy.gov
+ - data.worldbank.org
+ - fred.stlouisfed.org
+ - ourworldindata.org
+ - www.eia.gov
+ - metaculus.com
+
+ Note that this means you can embed Metaculus questions:
+
+ {``}
+
+ will render as:
+
+
+
+
+ Note that for now this is only possible in question bodies, not in
+ comments.
+
+
+
+
+ Images
+
+
+ We also allow <img> images:
+
+
+ {`
`}
+
+ will render as:
+
+
+
+
+ Differences and limitations
+
+
+ The official Markdown specification lets users input raw HTML, but we
+ limit users to the elements described above. For example, if you try to
+ input an image using  the
+ output will look like <img alt="Alt text"
+ src="/path/to/img.jpg"/>, and something like{" "}
+ <script>doSomethingEvil()</script> certainly
+ won't work. We also employ a few markdown extensions that handle
+ fenced code blocks (described above) and make{" "}
+
+ lists
+ {" "}
+ and bolded text a little easier to manage.
+
+
+ );
+}
From 2f3eba91fd6f245e8447e0e31762811da04aaab5 Mon Sep 17 00:00:00 2001
From: aseckin
Date: Sat, 31 Aug 2024 01:57:03 +0200
Subject: [PATCH 7/8] Markdown page syntax
---
.../src/app/(main)/help/markdown/page.tsx | 64 ++++++++++---------
1 file changed, 34 insertions(+), 30 deletions(-)
diff --git a/front_end/src/app/(main)/help/markdown/page.tsx b/front_end/src/app/(main)/help/markdown/page.tsx
index d13a5a5114..674743a06c 100644
--- a/front_end/src/app/(main)/help/markdown/page.tsx
+++ b/front_end/src/app/(main)/help/markdown/page.tsx
@@ -70,8 +70,8 @@ export default function MedalsFAQ() {
Links can be produced using a{" "}
[link title](http://and-link-address.com) or by surrounding
- a link with < and >, like{" "}
- <http://www.example.com>. There are a number of
+ a link with < and {">"}, like{" "}
+ <http://www.example.com{">"}. There are a number of
shortcuts to make your life easier if you keep repeating the same link
(see the{" "}
docs),
@@ -82,30 +82,32 @@ export default function MedalsFAQ() {
and two asterisks will make the text **bold**.
Back-ticks denote fixed-width text. If you want small text,
you can wrap it in a literal{" "}
- <small>html tag</small>. Special characters (
- *_{}#+-.!\) can be escaped using a backslash, like{" "}
- \*, if they would otherwise be converted into a markdown
- element.
+
+ <small{">"}html tag</small{">"}
+
+ . Special characters (*_{}#+-.!\) can be escaped using a
+ backslash, like \*, if they would otherwise be converted
+ into a markdown element.
We also allow a limited subset of HTML tags, which you can mix with
- markdown syntax if you want. These include: <a>,{" "}
- <p>, <em>,{" "}
- <strong>, <small>,{" "}
- <ol>, <ul>,{" "}
- <li>, <br>,{" "}
- <code>, <pre>,{" "}
- <blockquote>, <aside>,{" "}
- <div>, <h1>,{" "}
- <h2>, <h3>,{" "}
- <h4>, <h5>,{" "}
- <h6>, <math-inline>,{" "}
- <math-display>, <hr>,{" "}
- <table>, <thead>,{" "}
- <tbody>, <tr>,{" "}
- <th>, <td>,{" "}
- <del>, <sup>,{" "}
- <sub>.
+ markdown syntax if you want. These include: <a{">"},{" "}
+ <p{">"}, <em{">"},{" "}
+ <strong{">"}, <small{">"},{" "}
+ <ol{">"}, <ul{">"},{" "}
+ <li{">"}, <br{">"},{" "}
+ <code{">"}, <pre{">"},{" "}
+ <blockquote{">"}, <aside{">"},{" "}
+ <div{">"}, <h1{">"},{" "}
+ <h2{">"}, <h3{">"},{" "}
+ <h4{">"}, <h5{">"},{" "}
+ <h6{">"}, <math-inline{">"},{" "}
+ <math-display{">"}, <hr{">"},{" "}
+ <table{">"}, <thead{">"},{" "}
+ <tbody{">"}, <tr{">"},{" "}
+ <th{">"}, <td{">"},{" "}
+ <del{">"}, <sup{">"},{" "}
+ <sub{">"}.
@@ -180,7 +182,7 @@ def hello_world():
> id sem consectetuer libero luctus adipiscing.`}
which would produce:
-
+
This is a blockquote with two paragraphs. Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Aliquam hendrerit mi posuere lectus.
@@ -346,7 +348,7 @@ def hello_world():
Embeds
- We allow <iframe> embeds from a limited list of
+ We allow <iframe{">"} embeds from a limited list of
trusted sites, currently including:
{`
`}
@@ -399,10 +401,12 @@ def hello_world():
limit users to the elements described above. For example, if you try to
input an image using  the
output will look like <img alt="Alt text"
- src="/path/to/img.jpg"/>, and something like{" "}
- <script>doSomethingEvil()</script> certainly
- won't work. We also employ a few markdown extensions that handle
- fenced code blocks (described above) and make{" "}
+ src="/path/to/img.jpg"/{">"}, and something like{" "}
+
+ <script{">"}doSomethingEvil()</script{">"}
+ {" "}
+ certainly won't work. We also employ a few markdown extensions that
+ handle fenced code blocks (described above) and make{" "}
lists
{" "}
From b3fdca037ea99821684fb6057ef9227ba0136703 Mon Sep 17 00:00:00 2001
From: aseckin
Date: Sat, 31 Aug 2024 02:55:48 +0200
Subject: [PATCH 8/8] Added Legacy Points Rankings page
---
.../(main)/legacy-points-rankings/page.tsx | 6032 +++++++++++++++++
1 file changed, 6032 insertions(+)
create mode 100644 front_end/src/app/(main)/legacy-points-rankings/page.tsx
diff --git a/front_end/src/app/(main)/legacy-points-rankings/page.tsx b/front_end/src/app/(main)/legacy-points-rankings/page.tsx
new file mode 100644
index 0000000000..1e3ab9e663
--- /dev/null
+++ b/front_end/src/app/(main)/legacy-points-rankings/page.tsx
@@ -0,0 +1,6032 @@
+import PageWrapper from "../components/pagewrapper";
+
+export const metadata = {
+ title: "Points Rankings Archive | Metaculus",
+ description:
+ "This page shows the cumulative points earned by top forecasters on all questions from October 2015 to April 2024. This page will not be updated in the future.",
+};
+
+export default function LegacyPointsRankings() {
+ return (
+
+ Points Rankings
+
+ This page shows the cumulative points earned by top forecasters on all
+ questions from October 2015 to April 2024. This page will not be updated
+ in the future. Our current scoring and leaderboard system can be found{" "}
+ here.
+
+
+
+ );
+}