From 4c1cdbdcbf0bda517d645c7ddde923a8a3240ade Mon Sep 17 00:00:00 2001
From: aseckin
+ {position}
+
+ Metaculus is an online forecasting platform and aggregation engine
+ working to improve human reasoning and coordination on topics of
+ global importance.
+
+ Metaculus is a{" "}
+
+ Public Benefit Corporation
+
+ .
+
+ This organizational structure enables us to serve the public good
+ through the following commitments in our charter:
+
+ To build epistemic infrastructure that enables the global community to
+ model, understand, predict, and navigate the world’s most important
+ and complex challenges.
+
+ Forecasting is the practice of putting explicit probabilities,
+ dates, and numbers on future events—calculating odds via both
+ models and human judgment.
+
+ Although such estimates are subjective, a substantial body of
+ scientific research demonstrates two points: that people who
+ forecast can improve, with some becoming expertly calibrated; and
+ that aggregating across many diverse opinions produces more
+ accurate forecasts than even the best individuals forecasting
+ alone. The resulting predictions give us a clearer sense of what
+ tomorrow will look like, allowing us to make better decisions
+ today, just as a good meteorologist can help us decide whether to
+ carry an umbrella.
+
+ Of course, subjects like geopolitics are not like meteorology.
+ Yet, the scientific validity of human forecasting holds true even
+ when it comes to subjects with a high degree of uncertainty, like
+ the war between Russia and Ukraine. In fact,{" "}
+
+ research
+ {" "}
+ by University of Pennsylvania psychologists Philip Tetlock and
+ Barbara Mellers found that the aggregated geopolitical predictions
+ of top forecasters were{" "}
+
+ more accurate
+ {" "}
+ than those of CIA analysts with access to classified information.
+
+ Because forecasts are expressed probabilistically, we can rarely
+ say that a particular forecast was “right” or “wrong,”. Rather, we
+ score forecasts mathematically by comparing forecasts to outcomes
+ over a large body of questions. This enables us to determine how
+ “well-calibrated” any given forecaster is—i.e., do things that
+ they believe are 70% likely actually happen 70% of the time-as
+ well as track the record of the whole Metaculus community over
+ thousands of questions.
+
+ Metaculus forecasts are well-calibrated. They provide greater
+ visibility into the future.
+
+ Metaculus is an online forecasting platform and aggregation engine
+ working to improve human reasoning and coordination on topics of
+ global importance. As a Public Benefit Corporation, Metaculus
+ provides decision support based on these forecasts to a variety of
+ institutions (learn more).
+
+ Metaculus features questions on a wide range of topics, with a
+ particular focus on{" "}
+ artificial intelligence,{" "}
+ biosecurity,{" "}
+
+ climate change
+
+ , and nuclear risk.
+
+ No, Metaculus is a forecasting platform and aggregation engine.
+ Like prediction markets, we collect people's forecasts and
+ reward them for accuracy. But in prediction markets, participants
+ place bets against each other for financial rewards, can only win
+ insofar as someone else loses. Metaculus forecasters are
+ incentivized only to make the most accurate forecasts, and they
+ often collaborate to do so.
+
+ Prediction markets produce forecasts via where the betting market
+ settles. Metaculus explicitly aggregates everyone's forecasts
+ together using algorithms we refine over time. We produce a
+ time-weighted median, the "Community Prediction," as
+ well as the more sophisticated "Metaculus Prediction".
+
+ Prediction market bettors can produce accurate forecasts because
+ they have “skin in the game.” But{" "}
+
+ research
+ {" "}
+ shows that forecasting platforms like Metaculus often outperform
+ prediction markets, while avoiding many of the downsides of market
+ incentives that lead to regulators{" "}
+
+ restricting their activity.
+ {" "}
+ And critically, the research, methods, and reasoning that
+ Metaculus forecasters produce are themselves valuable, as seen
+ both in question comments as well as in the{" "}
+ Metaculus Journal.
+
+ 1. Complement expert analysis.
+
+ News stories often rely on expert analysis to put developments
+ in context and to anticipate the future course of events.
+ Unfortunately, experts frequently offer imprecisely worded
+ forecasts—e.g., “If the United States provides it with F-16s,
+ there is a real possibility that Ukraine will regain control of
+ its airspace”—and{" "}
+
+ research
+ {" "}
+ shows that people interpret “real possibility” to mean anything
+ between 20% and 80%, confusing both journalists and their
+ audiences. Probabilistic forecasts eliminate this problem.
+
+ What’s more, to the extent that experts do put precise
+ probabilities on future events, their track record is poor. One
+ of Tetlock’s{" "}
+
+ early findings
+ {" "}
+ was that political experts are highly overconfident in their
+ predictions. In fact, although there is significant variance,
+ their aggregate forecasts perform little better than chance. By
+ contrast, Metaculus predictions perform significantly better
+ than chance.
+
+ 2. Serve as a check on the conventional wisdom.
+
+ Forecasts can suggest that the conventional wisdom may be wrong
+ and that strongly held beliefs are worth questioning strongly.
+ For example, the conventional wisdom within the American
+ military is that China will invade Taiwan in the next few years.
+ One four-star general went so far as to{" "}
+
+ suggest
+ {" "}
+ there was a 100% chance of the PRC attempting to seize the
+ island in 2025, leading to war with the United States. By
+ contrast, the Metaculus forecast for the same time period is{" "}
+ {/*
+ 3. Make sense of the big questions.
+
+ Metaculus forecasts can help both journalists and their readers
+ make sense of developments where there is tremendous uncertainty
+ by breaking large, difficult-to-answer questions into smaller,
+ more tractable ones.
+
+ The future of artificial intelligence falls into this category,
+ where the questions people are most interested in (e.g., “Will
+ AI lead to a more utopian or a more dystopian future?”) are
+ impossible to answer at this point. We can, however, provide
+ forecasts on{" "}
+ more targeted questions on AI
+ safety, the regulation of AI, technical progress on AI, and the
+ business of AI—all of which can help us better understand which
+ direction we are headed in and how fast. Forecasting questions
+ serve as clues as to what developments we should be paying
+ particular attention to. And a{" "}
+
+ thorough analysis
+ {" "}
+ of our track record on AI-related questions showed that
+ Metaculus predictions offer clear and useful insights into the
+ future of the field and its impacts.
+
+ Metaculus forecasts can also identify where there have been
+ significant changes in our anticipations of the future. For
+ example, the drastic reduction in the forecast arrival date of
+ transformative AI—from the early 2040s to the current
+ distribution, centered around{" "}
+ {/*
+ Simply paste the link to the Metaculus question URL, like{" "}
+
+ www.metaculus.com/questions/17096/us-tracks-training-runs-by-2026
+
+ , and the preview image with the graph will show up automatically.
+
+ On the question page, click “embed” at the top. Choose
+ your theme, width, and height, then copy the iframe onto your
+ site.
+
+ If you'd prefer to have a static image rather than an embed
+ that users can interact with, navigate to the URL of the embed,
+ e.g.{" "}
+
+ www.metaculus.com/questions/embed/17096/
+
+ . Then save the image, generally via right click + “save
+ image as”, and upload it to your preferred site.
+
+ See the full API documentation. You can also
+ see the raw question data in your browser, like{" "}
+
+ www.metaculus.com/api2/questions/17096/
+
+ .
+
+ Select the ‘...’ menu on the question page, and click
+ “Download CSV.” (Only available on questions with a
+ critical mass of predictions.) If you have more expansive data
+ needs, please reach out to{" "}
+
+ christian@metaculus.com
+ {" "}
+ and we can construct a custom dataset for you.
+
+ There are thousands of Metaculus questions. You can search on{" "}
+ the main feed by topic or keyword. Our AI
+ questions can be found here.
+
+ If you can't find what you're looking for, or want to
+ suggest a question to forecast, reach out to{" "}
+
+ christian@metaculus.com
+ {" "}
+ .
+
+ Metaculus users' individual forecasts are weighted by their
+ track records, which are rigorously evaluated. Our{" "}
+ own record is public and
+ robust:
+
+
+ Click here
+ {" "}
+ to download a copy of the Metaculus logo and monogram.
+
A Conditional Pair is a special type of{" "}
-
+
Question Group
{" "}
that elicits{" "}
@@ -1134,7 +1134,7 @@ export default function FAQ() {
. Each Conditional Pair sits between a Parent Question and a Child
Question. Both Parent and Child must be existing Metaculus{" "}
-
+
Binary Questions
.
@@ -1180,7 +1180,7 @@ export default function FAQ() {
+ {(userId || socials) && (
+
+ About Metaculus
+
+
+ {[
+ "Fostering the growth, learning, and development of the forecasting and forecasting research communities.",
+ "Supporting stakeholders who are serving the public good by informing their decision making.",
+ "Increasing public access to information regarding forecasts of public interest.",
+ ].map((text, i) => (
+
+
+ Mission
+
+
+ {groupName}
+
+
+
+
+ On Substack, Twitter, or most other social media:
+
+
+ On Other Sites:
+
+
+ Via the API:
+
+
+ Download Question Data:
+
+ Metaculus For Journalists
+
+ How to Reference Forecasts
+
+ News That Cited{" "}
+ Metaculus
+
+
+
+ {title}
+
+
+ ))}
+ Metaculus Track Record
+ Download Assets
+
+
For{" "} - - date range - {" "} + date range{" "} questions, the dates on the interface are in UTC. Typically the time of day makes little difference as one day is miniscule in comparison to the full range, but occasionally for shorter term questions the time of day might materially impact scores. If it is not clear what point in a specified period a date range question will be resolved as, it resolves as the{" "} - + midpoint of that period . For example, if a question says it will resolve as a certain day, @@ -1456,12 +1454,12 @@ export default function FAQ() {
Only Metaculus Administrators can resolve questions. Binary questions can resolve Yes, No,{" "} - + Ambiguous, or Annuled . Range questions can resolve to a specific value, an out-of-bounds value,{" "} - + Ambiguous, or Annuled . @@ -1485,7 +1483,7 @@ export default function FAQ() {
In the same way, when a Conditional turns out to be based on an
outcome that did not occur, it is Annulled. For example, when a{" "}
-
+
Conditional Pair
's parent resolves Yes, the if No
Conditional is Annulled.
@@ -1710,7 +1708,7 @@ export default function FAQ() {
This question relied on data published in a report by
Microsoft, however Microsoft's report for the year in
question no longer contained the relevant data. It's{" "}
-
+
Metaculus policy
{" "}
that by default if a resolution source is not available
@@ -1786,7 +1784,7 @@ export default function FAQ() {
This question relied on data published in a report by Microsoft,
however Microsoft's report for the year in question no
longer contained the relevant data. It's{" "}
-
+
Metaculus policy
{" "}
that by default if a resolution source is not available
@@ -2683,7 +2681,7 @@ export default function FAQ() {
If the Public Figure does not yet have a dedicated page, you can
- request that one be created by commenting on the
+ request that one be created by commenting on the{" "}
Joe Biden expressed that he plans to run for reelection
- .
+ .{" "}
- On the other hand, this question about whether the
+ On the other hand, this question about whether the{" "}
- If you would like to write for the Metaculus Journal, email
-
- christian@metaculus.com
- {" "}
+ If you would like to write for the Metaculus Journal, email{" "}
+ christian@metaculus.com{" "}
with a resume or CV, a writing sample, and two story pitches.
If you’re interested in hiring Metaculus Pro Forecasters for a
- project, contact us at
-
- support@metaculus.com
- {" "}
+ project, contact us at{" "}
+ support@metaculus.com{" "}
with the subject "Project Inquiry".
@@ -3132,7 +3126,7 @@ export default function FAQ() {
Does Metaculus have an API?
- The Metaculus API can be found here:
+ The Metaculus API can be found here:{" "}
https://www.metaculus.com/api2/schema/redoc/
@@ -3170,7 +3164,7 @@ export default function FAQ() {
Metaculus may—though this thankfully occurs very rarely—issue the
temporary suspensions of an account. This occurs when a user has
- acted in a way that we consider inappropriate, such as when our
+ acted in a way that we consider inappropriate, such as when our{" "}
terms of use are violated. At this
point, the user will receive a notice about the suspension and be
made aware that continuing this behavior is unacceptable. Temporary
@@ -3211,7 +3205,7 @@ export default function FAQ() {
matches articles to Metaculus questions.
- The article matching model is supported by
+ The article matching model is supported by{" "}
Improve the News, a
news aggregator developed by a group of researchers at MIT. Designed
to give readers more control over their news consumption, Improve
@@ -3247,7 +3241,7 @@ export default function FAQ() {
questions with large comment threads and will update regularly as
new discussion emerges in the comments. If you have feedback on
these summaries—or would like to see them appear on a wider variety
- of questions—email
+ of questions—email{" "}
support@metaculus.com.
From de9d7a9bc2f7c58aa08efed1b5b54e9ea0aa1dd4 Mon Sep 17 00:00:00 2001
From: aseckin
+ Description of Tournament. The Tournament, run by Metaculus,
+ Inc. ("Metaculus"), will test participants' ability to
+ create or assemble autonomous programs ("bots") that make
+ accurate probabilistic forecasts concerning future events.
+
+ How to Enter and Participate. Any person (or team of people
+ contributing to a single bot) wishing to participate must create a
+ free Metaculus bot account, if they do not already have one, and
+ register for the tournament via the Tournament landing page, thereby
+ accepting Tournament terms and conditions. To qualify for any of the
+ prizes, your bot needs to achieve a sufficiently high Peer Score, as
+ assessed by Metaculus's Tournament Scoring Rules (a copy of which
+ is available{" "}
+
+ here
+ {" "}
+ and incorporated herein by reference). No purchase or payment is
+ necessary to enter or win. No individual may enter more than one bot.
+ Entering more than one will void all entries and result in
+ disqualification. You may not enter more times than indicated by using
+ multiple email addresses, identities, or devices in an attempt to
+ circumvent the rules. Participants are not allowed to share accounts.
+ If multiple individuals have collaborated on building a single bot,
+ you may only have one entry per group.
+
+ Eligibility. All participants entering a bot or contributing to
+ the creation of a bot for this Tournament must be 18 years of age or
+ older as of July 8, 2024. Employees of Metaculus and their affiliates,
+ subsidiaries, and agencies, and members of their immediate family or
+ persons living in the same household are not eligible to participate.
+ Void where prohibited.
+
+ Competitions are open to residents of the United States and worldwide,
+ except that if you are a resident of Crimea, Cuba, Iran, Syria, North
+ Korea, Sudan, Russia, or any other place prohibited by applicable law,
+ you may not enter any Competition. Prize recipients will be required
+ to verify their identity, nationality, and residency. Payouts will not
+ be made to any participant or participating bot where such a payment
+ would be prohibited by U.S. law. Taxes and fees are paid by
+ recipients.
+
+ Timing. The Tournament webpage will be made available by July
+ 8, 2024. Practice questions will be available from June 8, 2024 to
+ July 7, 2024. Official scored forecasting begins July 8, 2024. Final
+ rankings will be determined and winners announced on Oct 5, 2024 or
+ shortly thereafter.
+
+ Prizes. The Tournament will have a total prize pool of $30,000,
+ which will be awarded to a number of bot accounts based on their
+ performance in the Tournament. In order to be eligible for the prize,
+ the participating bot needs to have written a comment response under
+ every single question that it is forecasting. To be eligible for a
+ prize, Bot makers must provide a description of how their bot works or
+ the actual code. Bots may not have a human in the loop when
+ forecasting. This includes running a bot on questions that are open or
+ upcoming in the competition, seeing the bot's output, and then
+ modifying the bot to improve its output. Participants, including any
+ person or entity collaborating on or contributing to a bot, represent
+ and warrant that: (a) the participating bot comprises only and
+ exclusively software code that is either (i) the original work of a
+ person collaborating on or contributing to that bot or (ii) legally
+ permitted to be used in that bot, for example, because of open source
+ licensing, and in either case, the participants have all rights and
+ permission to use the bot for this Tournament and disclose the source
+ and a description of the bot's operation to Metaculus; (b) no
+ source code or description of the bot's operation will qualify as
+ software or technical information subject to restrictions or controls
+ on export by the United States or by any jurisdiction from which the
+ participants operate or are associated. The individual who creates a
+ bot account and enters a bot into the Tournament represents and
+ warrants that such individual has explicit permission to participate
+ from each person or entity collaborating on or contributing to that
+ bot and has provided a copy of these rules to each such person or
+ entity. All participants, including any person or entity collaborating
+ on or contributing to a bot, shall indemnify and hold harmless
+ Metaculus for any and all damages, losses, costs, and expenses
+ (including the costs of defense and reasonable attorney's fees)
+ arising from or incurred in connection with any claim relating to a
+ breach of the foregoing representations and warranties.
+
+ Allocation of Prizes. Allocation of prizes will be determined
+ according to Metaculus's Tournament Scoring Rules, linked above.
+ Winning entrants will receive payouts that depend on their rank and
+ forecasting performance, according to their score. If a winning bot is
+ submitted by a team of people, the indicated prize payment will be
+ made to an entity associated with the team; if such an entity does not
+ exist, the prize will be split evenly among members of the team, as
+ identified by the individual who submitted the team's
+ registration. A total of $30,000 will be awarded.
+
+
+ Metaculus will decide the final resolution of all questions at its
+ discretion.
+ {" "}
+ Ambiguous questions resolutions will be avoided to the maximum extent
+ possible, but in the rare instances where they occur, those questions
+ will no longer be scored as part of the tournament.
+
+ Right to cancel or modify. Metaculus reserves the right to
+ cancel, suspend, or modify the Tournament if any unforeseen
+ problem—technical, legal or otherwise—prevents the Tournament from
+ running as planned.
+
+ Participation in the Tournament is additionally subject to the
+ following Terms and Conditions:
+
+ PLEASE NOTE THAT YOUR USE OF AND ACCESS TO OUR SERVICES (DEFINED
+ BELOW) ARE SUBJECT TO THE FOLLOWING TERMS AND CONDITIONS. IF YOU DO
+ NOT AGREE TO ALL OF THE FOLLOWING, YOU MAY NOT USE OR ACCESS THE
+ SERVICES IN ANY MANNER.
+
+ Welcome to Metaculus. Please read on to learn the rules and
+ restrictions that govern your use of our website(s), products,
+ services and applications (the "Services"). If you have any
+ questions, comments, or concerns regarding these Terms or the
+ Services, please contact Support [at] metaculus [dot] com. These Terms
+ of Use (the "Terms") are a binding contract between you and
+ Metaculus Inc. ("Metaculus," "we" and
+ "us"). You must agree to and accept all of the Terms, or you
+ don't have the right to use the Services. Your use of the
+ Services in any way means that you agree to all of these Terms, and
+ these Terms will remain in effect while you use the Services. These
+ Terms include the provisions in this document, as well as those in the
+ Privacy Policy. In these Terms, the words "include" or
+ "including" mean "including but not limited to,"
+ and examples are for illustration purposes and are not limiting.
+
+ Will these Terms ever change? We are constantly trying to
+ improve our Services, so these Terms may need to change along with the
+ Services. We reserve the right to change the Terms at any time, but if
+ we do, we will bring it to your attention by placing a notice on the
+ Metaculus website, by sending you an email, or by some other means.
+ Changes will become effective no sooner than the day after they are
+ posted. However, changes addressing new functions for a Service or
+ changes made for legal reasons will be effective immediately. If you
+ don't agree with the new Terms, you are free to reject them by
+ ceasing all use of the Services. If you use the Services in any way
+ after a change to the Terms is effective, that means you agree to all
+ of the changes. Except for changes by us as described here, no other
+ amendment or modification of these Terms will be effective unless in
+ writing and signed by both you and us.
+
+ What about my privacy? Metaculus takes the privacy of its users
+ very seriously. For the current Metaculus Privacy Policy, please{" "}
+ click here.
+
+ What are the basics of using Metaculus for this Tournament?
+
+ You are required to sign up for a bot account and select a password
+ and bot username ("Metaculus Bot User ID"). You promise to
+ provide us with accurate, complete, and updated registration
+ information about yourself. You may not select as your Metaculus Bot
+ User ID a name that you don't have the right to use, or another
+ person's name with the intent to impersonate that person. You may
+ not transfer your bot account to anyone else without our prior written
+ permission. You may not have or control more than one active Metaculus
+ Bot User ID at any time. If we determine that you are operating under
+ more than one Metaculus Bot User ID, we may disqualify you from the
+ Tournament without notice and revoke access to your Metaculus Bot User
+ ID.
+
+ You represent and warrant that you are of legal age to form a binding
+ contract and you have explicit permission from all persons and
+ entities who have contributed to or collaborated on the bot to
+ participate in this Tournament.
+
+ You will only use the Services for your own internal, personal, or
+ business use, and not on behalf of or for the benefit of any third
+ party, nor in a service bureau modality, and only in a manner that
+ complies with all laws that apply to you. If your use of the Services
+ is prohibited by applicable laws, then you aren't authorized to
+ use the Services. We are not responsible if you use the Services in a
+ way that breaks the law.
+
+ You will keep all your registration information accurate and current.
+ You will not share your account or password with anyone, and you must
+ protect the security of your account and your password. You're
+ responsible for any activity associated with your account.
+
+ Yes. You represent, warrant, and agree that you will not contribute
+ any Content or User Submission or otherwise use the Services or
+ interact with the Services in a manner that:
+
+ Without limitation to any other remedies available to Metaculus, a
+ violation of any of the foregoing is grounds for termination of your
+ right to use or access the Services. We reserve the right to remove
+ any Content or User Submissions from the Services at any time, for any
+ reason (including if someone alleges, and, Metaculus in its sole and
+ absolute discretion determines that, you contributed that Content in
+ violation of these Terms), and without notice.
+
+ What are my rights in Metaculus?
+
+ The materials displayed or performed or available on or through the
+ Services, including text, graphics, data, articles, photos, images,
+ illustrations, and User Submissions (collectively, the
+ "Content"), are protected by copyright and other
+ intellectual property laws. You promise to abide by all copyright
+ notices, trademark rules, information, and restrictions contained in
+ any Content you access through the Services, and you won't use,
+ copy, reproduce, modify, translate, publish, broadcast, transmit,
+ distribute, perform, upload, display, license, sell or otherwise
+ exploit for any purpose any Content not owned by you, (i) without the
+ prior consent of the owner of that Content or (ii) in a way that
+ violates someone else's (including Metaculus's) rights.
+
+ You understand that Metaculus owns the Services. You won't
+ modify, publish, transmit, participate in the transfer or sale of,
+ reproduce (except as expressly provided in this Section), create
+ derivative works based on, or otherwise exploit any of the Services.
+
+ The Services may allow you to copy or download certain Content; please
+ remember that just because this functionality exists, it doesn't
+ mean that all the restrictions above don't apply — they do!
+
+ Who is responsible for what I see and do on the Services?
+
+ Any information or content publicly posted or privately transmitted
+ through the Services is the sole responsibility of the person from
+ whom, or on whose behalf, such content originated, and you access all
+ such information and content at your own risk. We aren't liable
+ for any errors or omissions in that information or content or for any
+ damages or loss you might suffer in connection with it. We cannot
+ control and have no duty to take any action regarding how you may
+ interpret and use the Content or what actions you may take as a result
+ of having been exposed to the Content, and you release us from all
+ liability for you having acquired or not acquired Content through the
+ Services. We can't guarantee the identity of any users with whom
+ you interact in using the Services and are not responsible for which
+ users gain access to the Services.
+
+ You are responsible for all Content you contribute to the Services or
+ which is contributed to the Services on your behalf, and you represent
+ and warrant you have all rights necessary to contribute such Content.
+
+ The Services may contain links, information or connections to third
+ party websites or services that are not owned or controlled by
+ Metaculus. When you access third party websites or engage with third
+ party services, you accept that there are risks in doing so, and that
+ Metaculus is not responsible for such risks. We encourage you to be
+ aware when you leave the Services and to read the terms and privacy
+ policy of each third party website or service that you visit or
+ utilize.
+
+ Metaculus has no control over, and assumes no responsibility for, the
+ content, accuracy, privacy policies, or practices of or opinions
+ expressed in any third party websites or by any third party or third
+ party technology that you interact with through the Services. In
+ addition, Metaculus will not and cannot monitor, verify, censor or
+ edit the content of any third party site or service. By using the
+ Services, you release and hold Metaculus harmless from any and all
+ liability arising from your use of any third party website or service.
+
+ Your interactions with organizations, individuals, or technologies
+ found on or through the Services, including payment and delivery of
+ goods or services, and any other terms, conditions, warranties or
+ representations associated with such dealings, are solely between you
+ and such organizations or individuals. You should make whatever
+ investigation you feel necessary or appropriate before proceeding with
+ any online or offline transaction with any of these third parties. You
+ agree that Metaculus will not be responsible or liable for any loss or
+ damage of any sort incurred as the result of any such dealings. If
+ there is a dispute between participants on this site, or between users
+ and any third party, you agree that Metaculus is under no obligation
+ to become involved. If you have a dispute with one or more other
+ users, you release Metaculus, and their officers, employees, agents,
+ and successors from claims, demands, and damages of every kind or
+ nature, known or unknown, suspected or unsuspected, disclosed or
+ undisclosed, arising out of or in any way related to such disputes or
+ our Services. If you are a California resident, you are expressly
+ waiving California Civil Code Section 1542, which says: "A
+ general release does not extend to claims which the creditor does not
+ know or suspect to exist in his or her favor at the time of executing
+ the release, which, if known by him or her must have materially
+ affected his or her settlement with the debtor."
+
+ What are the rules for competitions on Metaculus?
+
+ Competitions are run according to rules that describe participation
+ guidelines, the criteria used to select a winner of the Competition as
+ posted on Metaculus' Web Site. The prize(s) awarded to such
+ winner(s), and when such prize(s) will be awarded will be posted on
+ our Site. Such rules and selection criteria must comply with all
+ applicable laws and these Terms (collectively, "Competition
+ Rules"). Such Competition Rules will also include how and when a
+ Participant User must submit Competition Entries (defined below) and
+ the rights the Host User will be granted in such Competition Entry
+ upon selecting any such Competition Entry as a winner ("Winning
+ Entry"). Certain rights granted in the Competition Entries and
+ Winning Entries are described in Section 9 (Do I have to grant any
+ licenses to Metaculus or to other users?) below. The Competition Rules
+ may impose additional restrictions or requirements for Competitions.
+ Each Participant User will comply with all Competition Rules.
+ You acknowledge and agree that Metaculus may, without any liability
+ but without any obligation to do so, remove or disqualify a
+ Participant User, if Metaculus believes that such Participant User is
+ in violation of these Terms or otherwise poses a risk to Metaculus,
+ the Service, or another user of the Service.
+
+ Regardless of anything to the contrary, Participant Users acknowledge
+ and agree that Metaculus has no obligation to hold a Competition Entry
+ in confidence or otherwise restrict their activities based on receipt
+ of such Competition Entry. Metaculus has no obligation to become
+ involved in disputes between users or between users and any third
+ party relating to the use of the Services. When you participate in a
+ Competition, you release Metaculus from claims, damages, and demands
+ of every kind — known or unknown, suspected or unsuspected, disclosed
+ or undisclosed — arising out of or in any way related to such disputes
+ and the Services. All content you access or submit via the Services is
+ at your own risk.
+
+ Do I have to grant any licenses to Metaculus or to other users?
+
+ Anything you post, upload, share, store, or otherwise provide through
+ the Services (including anything posted, uploaded, shared, stored, or
+ otherwise provided by a bot or autonomous program that you have
+ connected to the Services) is your "User Submission." Some
+ User Submissions are viewable or downloadable by other users. To
+ display your User Submissions on the Services, and to allow other
+ users to enjoy them (where applicable), you grant us certain rights in
+ those User Submissions. Please note that all of the following licenses
+ are subject to our Privacy Policy to the extent they relate to User
+ Submissions that are also your personal information.
+
+ For all User Submissions, you grant Metaculus a license to translate,
+ modify (for technical purposes, for example making sure your content
+ is viewable on a mobile device as well as a computer), display,
+ distribute and reproduce and otherwise act with respect to such User
+ Submissions, in each case to enable us to operate the Services, as
+ described in more detail below. You acknowledge and agree that
+ Metaculus, in performing the required technical steps to provide the
+ Services to our users (including you), may need to make changes to
+ your User Submissions to conform and adapt those User Submissions to
+ the technical requirements of communication networks, devices,
+ services, or media, and the licenses you grant under these Terms
+ include the rights to do so. You also agree that all of the licenses
+ you grant under these Terms are royalty-free, perpetual, irrevocable,
+ and worldwide. These are licenses only — your ownership in User
+ Submissions is not affected.
+
+ If you share a User Submission publicly on the Services or in a manner
+ that allows more than just you or certain specified users to view it
+ (such as a Dataset), or if you provide us with any feedback,
+ suggestions, improvements, enhancements, or feature requests relating
+ to the Services (each a "Public User Submission"), then you
+ grant Metaculus the license stated in the second paragraph of this
+ Section 8, as well as a license to display, perform, and distribute
+ your Public User Submission for the purpose of making that Public User
+ Submission accessible to all Metaculus users and providing the
+ Services necessary to do so, as well as all other rights necessary to
+ use and exercise all rights in that Public User Submission in
+ connection with the Services for any purpose. Also, you grant all
+ other users of the Services a license to access that Public User
+ Submission, and to use and exercise all rights in it, as permitted by
+ the functionality of the Services.
+
+ If you are a Participant User and submit an entry to a Competition
+ ("Competition Entry"), then you grant Metaculus the license
+ stated in the second paragraph of this Section 8, as well as a license
+ to display, perform, and distribute your Competition Entry for the
+ purpose of making that Competition Entry accessible to the Host User,
+ making that Competition Entry available to other Metaculus users as
+ part of a Dataset, and providing the Services necessary.
+
+
+ What if I see something on the Services that infringes my copyright?
+
+
+ You may have heard of the Digital Millennium Copyright Act (the
+ "DMCA"), as it relates to online service providers, like
+ Metaculus, being asked to remove material that allegedly violates
+ someone's copyright. We respect others' intellectual
+ property rights, and we reserve the right to delete or disable Content
+ alleged to be infringing, and to terminate the accounts of repeat
+ alleged infringers. To report potentially infringing content, contact
+ support@metaculus.com. To learn more about the DMCA, click here.
+
+ Will Metaculus ever change the Services?
+
+ We're always trying to improve the Services, so they may change
+ over time. We may suspend or discontinue any part of the Services, or
+ we may introduce new features or impose limits on certain features or
+ restrict access to parts or all of the Services. We'll do our
+ best to give you notice when we make a material change to the
+ Services, but we ask for your understanding in cases where this
+ isn't practical.
+
+ What if I want to stop using Metaculus?
+
+ You're free to stop using the Service at any time. Please refer
+ to our Privacy Policy, as well as the licenses above, to understand
+ how we treat information you provide to us after you have stopped
+ using our Services.
+
+ Metaculus is also free to terminate (or suspend access to) your use of
+ the Services or your account, for any reason in our discretion,
+ including your breach of these Terms. Metaculus has the sole right to
+ decide whether you are in violation of any of the restrictions in
+ these Terms.
+
+ Account termination may result in destruction of any Content
+ associated with your account, so please keep that in mind before you
+ decide to terminate your account.
+
+ Provisions that, by their nature, should survive termination of these
+ Terms will survive termination. By way of example, all of the
+ following will survive termination: any obligation you have to pay us
+ or indemnify us, any limitations on our liability, any terms regarding
+ ownership or intellectual property rights, and terms regarding
+ disputes between us.
+
+ What else do I need to know?
+
+ Warranty Disclaimer. Neither Metaculus nor its licensors or suppliers
+ makes any representations or warranties concerning any content
+ contained in or accessed through the Services (including
+ Competitions), and we will not be responsible or liable for the
+ accuracy, copyright compliance, legality, or decency of material
+ contained in or accessed through the Services. We (and our licensors
+ and suppliers) make no representations or warranties regarding
+ suggestions or recommendations of services or products offered or
+ purchased through the Services. Products and services purchased or
+ offered (whether or not following such recommendations and
+ suggestions) through the Services are provided “AS IS” and without any
+ warranty of any kind from Metaculus or others (unless, with respect to
+ such others only, provided expressly and unambiguously in writing by a
+ designated third party for a specific product). THE SERVICES AND
+ CONTENT ARE PROVIDED BY METACULUS (AND ITS LICENSORS AND SUPPLIERS) ON
+ AN “AS-IS” BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
+ IMPLIED, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR
+ A PARTICULAR PURPOSE, NON-INFRINGEMENT, OR THAT USE OF THE SERVICES
+ WILL BE UNINTERRUPTED OR ERROR-FREE. SOME STATES DO NOT ALLOW
+ LIMITATIONS ON HOW LONG AN IMPLIED WARRANTY LASTS, SO THE ABOVE
+ LIMITATIONS MAY NOT APPLY TO YOU.
+
+ Limitation of Liability. TO THE FULLEST EXTENT ALLOWED BY APPLICABLE
+ LAW, UNDER NO CIRCUMSTANCES AND UNDER NO LEGAL THEORY (INCLUDING TORT,
+ CONTRACT, NEGLIGENCE, STRICT LIABILITY, OR OTHERWISE) WILL METACULUS
+ (OR ITS LICENSORS OR SUPPLIERS) BE LIABLE TO YOU OR TO ANY OTHER
+ PERSON FOR (A) ANY INDIRECT, SPECIAL OR INCIDENTAL DAMAGES OF ANY
+ KIND, INCLUDING DAMAGES FOR LOST PROFITS, LOSS OF GOODWILL, WORK
+ STOPPAGE, ACCURACY OF RESULTS, OR COMPUTER FAILURE OR MALFUNCTION, OR
+ (B) ANY AMOUNT, IN THE AGGREGATE, IN EXCESS OF THE GREATER OF (I) $100
+ OR (II) THE AMOUNTS PAID BY YOU TO METACULUS IN CONNECTION WITH THE
+ SERVICES IN THE TWELVE (12) MONTH PERIOD PRECEDING THIS APPLICABLE
+ CLAIM, OR (C) ANY MATTER BEYOND OUR REASONABLE CONTROL. SOME STATES DO
+ NOT ALLOW THE EXCLUSION OR LIMITATION OF CERTAIN DAMAGES, SO THE ABOVE
+ LIMITATION AND EXCLUSIONS MAY NOT APPLY TO YOU.
+
+ Indemnity. To the fullest extent allowed by applicable law, you will
+ defend, indemnify and hold Metaculus and their affiliates, officers,
+ agents, employees, and partners harmless from and against any and all
+ claims, liabilities, damages (actual and consequential), losses and
+ expenses (including attorneys' fees) arising from or in any way
+ related to any third party claims (including from other users)
+ relating to (a) your submissions to the Services including any
+ Content, User Submissions or Competitions, (b) your use of the
+ Services (including any actions taken by a third party using your
+ account), and (c) your violation of these Terms.
+
+ Assignment. You may not assign, delegate or transfer these Terms or
+ your rights or obligations hereunder, or your Services account, in any
+ way (by operation of law or otherwise) without Metaculus's prior
+ written consent. We may transfer, assign, or delegate these Terms and
+ our rights and obligations without consent.
+
+ About These Terms
+
+ These Terms control the relationship between Metaculus and you. They
+ do not create any third party beneficiary rights.
+
+ If you do not comply with these Terms, and we don't take action
+ right away, this doesn't mean that we are giving up any rights
+ that we may have (such as taking action in the future).
+
+ If it turns out that a particular term is not enforceable, this will
+ not affect any other part of the Terms.
+
+ The laws of California, USA, excluding California's conflict of
+ laws rules, will apply to any disputes arising out of or relating to
+ these Terms or the Services. All claims arising out of or relating to
+ these Terms or the Services will be litigated exclusively in the
+ federal or state courts of San Francisco, California, USA, and you and
+ Metaculus consent to personal jurisdiction in those courts.
+
+ The guidelines below are general rules that should be followed in the
+ large majority of cases. There are always exceptions to the rules, but
+ if your question breaks a guideline below then it is worth checking
+ with another moderator or admin to get a second or third opinion that
+ it is the best path forward.
+
+ Not everyone reading/predicting a question is a domain expert, or has
+ even heard of the question's subject. Good background text should
+ draw the reader's interest and provide some glimpse into our
+ current understanding on the subject. Inevitably some of this
+ information may become out of date or irrelevant, but this is fine.
+ Metaculus is as much a record/evaluation of past predictions as it is
+ a tracker of current information on a subject.
+
+ Exceptions to this rule can be made in some scenarios, such as
+ tracking COVID-19 deaths while we're 1 year into the pandemic
+ (self-explanatory), or if a question needs to be written/opened
+ quickly, such as an outbreak of political violence (high value and
+ urgent).
+
+ Our forecasters, readers, and question authors come from all over the
+ world, and we should avoid the confusion of seeing different date
+ formats in different questions/contexts. Month, day year (for example,
+ "September 26, 2021" or "Sep 26, 2021") is aligned
+ with Metaculus' design elements, and is easy for people to read
+ and parse.
+
+ People come to our questions in many contexts, and may not be familiar
+ with terms like "resolution date" or "opening
+ date" or our site's interface. Many of our questions are
+ meant to remain open for a long time; 10 years from now, it may be
+ hard to parse the meaning of "in the next 100 years" or
+ "when this question was written".
+
+ Resolution should almost always come in the form of 1. Publication
+ from third-parties; or 2. Discretion of Metaculus Admins. Metaculus
+ moderators are valued members of our team, but they do not have
+ authority to resolve questions. Resolution text including "polls
+ from users in comments" or "if at least 1 moderator
+ says…" should be avoided. Possible exceptions:{" "}
+
+ Will at least one Metaculus user report a positive test result for
+ novel coronavirus by the end of 2020?
+ {" "}
+ -- users can submit verifiable claims for Admins to approve (though
+ alternative resolutions are generally preferred).
+
+ Though there are often exceptions to this rule, Metaculus has several
+ far-future forecasts where inflation can significantly change the
+ terms. When considering resolution dates 10 years out or more,
+ inflation indexing is generally preferred.
+
+ Sometimes we care about the behavior of a source, like{" "}
+
+ When will the WHO announce that the COVID-19 pandemic has ended?
+
+ , other times we care about the actual answer, like{" "}
+
+ Will 2021 be the hottest year on record according to NASA?
+
+ . We have a default policy in place: we will assume the question is
+ tracking the "true answer" unless the author specifically
+ stresses otherwise.
+
+ For example, a question might say "If this linked question on
+ Metaculus resolves true, then how many X by Y date?". This may
+ seem to have no harm and make the question briefer and simpler, but
+ this hides important complexity. Every term in the resolution criteria
+ is significant, and linking questions can lead to cascades of
+ simplifying and misunderstanding criteria. Even if criteria are copied
+ and pasted, this redundancy encourages predictors to re-review the
+ terms, potentially discovering ambiguities.
+
+ Examples of poor questions:{" "}
+
+ In the 2020 US Presidential election, when will the losing candidate
+ concede?
+ {" "}
+ And{" "}
+
+ Will any body of the US federal government conclude that COVID-19
+ originated in a lab in Hubei before June 1st 2022?
+ {" "}
+ In cases like these, a public figure might be under great pressure to
+ make a certain statement, and their reluctance to do so will often
+ lead them to vague or softened word choices. No matter how broad or
+ inclusive we might define resolution criteria, these situations
+ frequently lead to polarizing debates and unsatisfying resolutions. If
+ a question can't be defined on concrete actions or information,
+ it is a sign it should be avoided (With an exception for formal or
+ well-defined terms).
+
+ Example of a poor question:{" "}
+
+ Will the number of living humans who have walked on another world
+ fall to zero?
+ {" "}
+ This question can be easily rewritten to focus on future space
+ missions (a matter of public interest), rather than the health and
+ longevity of retired astronauts (not appropriate).
+
+ A good question:{" "}
+
+ When will the next US Supreme Court vacancy arise?
+ {" "}
+ Though an individual's death is a component of resolution, it is
+ arguably not the most likely component; regardless, the transition is
+ highly important to the public interest. Public interest can outweigh
+ this rule, for instance in a question like "When will Kim Jung Un
+ no longer be Dictator-For-Life?"
+
+ This is naturally somewhat more of an art than a science. Although
+ every detail in the resolution criteria is relevant, the more the
+ headline question matches the resolution criteria, the stronger the
+ question and forecasts will be. If a complicating detail does not make
+ a question more insightful, remove it.
+
+ Sometimes in controversial areas, Metaculus can offer a public service
+ in gathering high-quality information and giving falsifiable
+ predictions. If there is value or public interest in controversial
+ subjects, they can make for good questions. However, the stakes are
+ generally higher, and such questions will attract more attention. More
+ care is necessary in these cases.
+
+ Official Rules for the AI Forecasting Benchmark Q3 Tournament (the
+ “Tournament”))
+
+
+
+
+
+
+ Metaculus Question Approval Checklist
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Explanations/Rationales
+
+ Formatting and style
+
+
+ Does the question have at least 2 paragraphs / ~100 words of context?
+ If not, is the question self-explanatory or high value and urgent?
+
+
+ Are dates written in "Month, day year" format?
+
+
+ Are all dates written as absolute dates (not relatively, such as
+ "in this century" or "after the opening of this
+ question")?
+
+
+
+
+ Resolution and Scoring Details
+
+
+
+ Is the question resolution solely under the authority of Metaculus
+ Admins or an external third-party?
+
+
+ If the question involves currencies or prices, is the resolution
+ inflation-indexed?
+
+
+ When naming a resolution source, are we trying to forecast{" "}
+
+ that particular source
+ {" "}
+ more than the "true answer"? If we prefer the "true
+ answer", are any fallback sources listed?
+
+
+ Does the resolution avoid linking/dependence on other Metaculus
+ questions?
+
+
+ Does the resolution avoid dependence on any individual or group saying
+ particular words or phrases? If not, are they formal or well-defined
+ terms?
+
+
+
+ Sensitive Subjects
+
+
+ Does the question avoid predicting the mortality of an individual or
+ specific small group?
+
+
+ Does the headline question avoid making a stronger/weaker claim than
+ the resolution criteria?
+
+
+ Does the topic avoid highly controversial or potentially controversial
+ subjects? If not, has it been reviewed carefully by 2 mods/admins?
+
+
+
@@ -95,7 +95,7 @@ export default function QuestionChecklist() {
+
Explanations/Rationales
@@ -148,7 +148,7 @@ export default function QuestionChecklist() {
"when this question was written".
+ Examples:{" "} + "Who will be Japan's next Prime Minister?",{" "} + "Will NASA's Artemis 2 launch be successful?", … +
++ To predict, share the probability you give the outcome as a number + between 0.1% and 99.9%. On the question page, simply drag the + prediction slider until it matches your probability and click + “Predict”. You can also use the arrows to refine your probability or + select the field and type the probability. +
+
+ + Multiple choice questions ask about more than two (Yes/No) + possibilities. Predicting works the same, except your predictions + should sum to 100%. After inputting probabilities, select auto-sum to + guarantee they do. +
+
+ + The higher the probability you place on the correct outcome, the + better (more positive) your score will be. Give the correct outcome a + low probability and you'll receive a bad (negative) score. Under + Metaculus scoring, you'll get the best score by predicting what + you think the actual probability is, rather than trying to “game” the + scoring. +
+ ++ Examples: "When will humans land on Mars?",{" "} + "What will Germany's GDP growth be in 2025?", … +
++ To predict, provide a distribution, representing how likely you think + each outcome in a range is. On the question page, drag the slider to + change the shape of your bell curve, and focus your prediction on + values you think are likely. +
+
+ + If you want to distribute your prediction in more than one section of + the range, you can add up to four independent bell curves to build + your distribution and assign a weight to each of them. +
+
+ + The higher your distribution is on the value that ultimately occurs, + the better your score. The lower your distribution on the actual + value, the worse your score. To get the best score, make your + distribution reflect how likely each possible value actually is. +
++ Welcome to our question writing guide! Here, you'll learn about + our best practices for writing and submitting questions, as well as + our content rules and guidelines. +
++ We greatly value the contributions of our diverse community of + forecasters, question authors, and forum participants, and we hope + that these guidelines will promote, enhance, and safeguard a vibrant + community forecasting space for many years to come. +
++ Submitted questions are reviewed by a group of volunteer{" "} + Community Moderators. The Community + Moderators team tries its best to approve all the questions that + conform to our guidelines and best practices swiftly, typically within + a week or two. Questions that are not immediately ready for + publication are provided feedback by Community Moderators. +
+ ++ Metaculus hosts questions on almost any topic — science, technology, + politics, business, law, economics — you name it! That said, our + primary focus areas are Science,{" "} + + Technology + + ,{" "} + + Effective Altruism + + ,{" "} + + Artificial Intelligence + + ,{" "} + + Health + + ,{" "} + + Geopolitics + + , and{" "} + + Far-Future Forecasting + {" "} + (10 years or more in the future). +
++ Currently, we will consider any question that satisfies our guidelines + and rules for publication on the platform. +
+ ++ Writing incisive questions for forecasting can be challenging and + requires a keen eye for detail, careful precision, creativity and + imagination, and a host of other skills. Fortunately, many of these + skills can be cultivated with a bit of practice. +
++ For experienced question writers, we have a briefer checklist to + address common issues{" "} + here. + For a comprehensive guide to writing good questions, see our + guidelines below. +
+ ++ Resolution criteria are the backbone of any forecasting question—they + spell out how and when a question will resolve. It's therefore + key to spell these out clearly. +
++ See the FAQ{" "} + for more details about how questions resolve. +
+ ++ Metaculus has developed its own style and norms for writing and + submitting questions. Heeding these best-practices will be appreciated + by Metaculus forecasters and Moderators alike. Questions which + don't follow these best-practices may be rejected. +
++ When submitting a question, you are asked to specify the closing date + (when the question is no longer available for predicting) and + resolution date (when the resolution is expected to occur). The date + the question is set live for others to forecast on is known as the + open date. +
++ In some cases, questions must resolve at the resolution date according + to the best available information. In such cases, it becomes important + to choose the resolution date carefully. Try to set resolution dates + that make for interesting and insightful questions! The date or time + period the question is asking about must always be explicitly + mentioned in the text (for example, "this question resolves as + the value of X on January 1, 2040, according to source Y" or + “this question resolves as Yes if X happens before + January 1, 2040)". +
++ The close date must be at least one hour prior to the + resolution date, but can be much earlier, depending upon the context. + Here are some guidelines for specifying the close date: +
++ Note: Previous guidance suggested that a question + should close between 1/2 to 2/3 of the way between the open time and + resolution time. This was necessary due to the scoring system at the + time, but has been replaced by the above guidelines due to an{" "} + + update to the scoring system + + . +
+ ++ In addition to specifying precise resolution criteria and following + accepted formatting and practices, we have some rules that we adhere + to regarding question content. +
++ Question content rules (with the exception of the notability + requirement, and the no non-public figures rule) apply equally to + private questions as they do to public ones. +
+ ++ Metaculus Community Moderators are committed members of the community + who volunteer to help the platform run smoothly. We greatly appreciate + their time and talents, and we hope you'll do the same! +
+The main responsibilities of Community Moderators include:
++ Additionally, Community Moderators can help draw extra attention to + bugs, feature requests, or other issues that affect the platform. + Community Moderators are appointed democratically, by way of election. + If you'd like to register interest in becoming a Community + Moderator, please do so by commenting to this effect in the comment + section of this discussion thread, and we'll reach out to you + before the next election takes place. +
+