API: forecasting for numeric continuous values undocumented and example appears incorrect

[jkraybill]

Current behavior

I'm trying to figure out how to do numeric continuous forecasts in the API via trial and error. I was not expecting to have to supply both a CDF and the left/center/right values, but I'm trying to do that since the API seems to expect it.

According to the example in the API doc, the request I'm testing should be valid:

[{"question":28423,"continuous_cdf":[0.0,0.09424125589231841,0.19647063437408002,1.0],"slider_values":{"forecast":[{"left":0.25,"center":0.4,"right":0.75}],"weights":[1.0]}}]

But, when I submit this, I get

{"errors":[{"non_field_errors":["continuous_cdf invalid. Must be increasing, have 201 points, and respect the bounds of the question"]}]}

OK, so let's try 201 points that are increasing.

[{"question":28423,"continuous_cdf":[0.0,0.005,0.01,0.015,0.02,0.025,0.03,0.035,0.04,0.045,0.05,0.055,0.06,0.065,0.07,0.075,0.08,0.085,0.09,0.095,0.1,0.105,0.11,0.115,0.12,0.125,0.13,0.135,0.14,0.145,0.15,0.155,0.16,0.165,0.17,0.175,0.18,0.185,0.19,0.195,0.2,0.205,0.21,0.215,0.22,0.225,0.23,0.235,0.24,0.245,0.25,0.255,0.26,0.265,0.27,0.275,0.28,0.285,0.29,0.295,0.3,0.305,0.31,0.315,0.32,0.325,0.33,0.335,0.34,0.345,0.35,0.355,0.36,0.365,0.37,0.375,0.38,0.385,0.39,0.395,0.4,0.405,0.41,0.415,0.42,0.425,0.43,0.435,0.44,0.445,0.45,0.455,0.46,0.465,0.47,0.475,0.48,0.485,0.49,0.495,0.5,0.505,0.51,0.515,0.52,0.525,0.53,0.535,0.54,0.545,0.55,0.555,0.56,0.565,0.57,0.575,0.58,0.585,0.59,0.595,0.6,0.605,0.61,0.615,0.62,0.625,0.63,0.635,0.64,0.645,0.65,0.655,0.66,0.665,0.67,0.675,0.68,0.685,0.69,0.695,0.7,0.705,0.71,0.715,0.72,0.725,0.73,0.735,0.74,0.745,0.75,0.755,0.76,0.765,0.77,0.775,0.78,0.785,0.79,0.795,0.8,0.805,0.81,0.815,0.82,0.825,0.83,0.835,0.84,0.845,0.85,0.855,0.86,0.865,0.87,0.875,0.88,0.885,0.89,0.895,0.9,0.905,0.91,0.915,0.92,0.925,0.93,0.935,0.94,0.945,0.95,0.955,0.96,0.965,0.97,0.975,0.98,0.985,0.99,0.995,1.0],"slider_values":{"forecast":[{"left":0.25,"center":0.4,"right":0.75}],"weights":[1.0]}}]

Same error: {"errors":[{"non_field_errors":["continuous_cdf invalid. Must be increasing, have 201 points, and respect the bounds of the question"]}]}

Expected/desired behavior

I'm seemingly doing something wrong, but due to the API being undocumented, and this error message being cryptically vague, I have no idea what it is.

Please document this API, and improve the error message specificity. For example, in this code:

           if not (
                continuous_cdf_increasing
                and lower_bound_ok
                and upper_bound_ok
                and len(continuous_cdf) == 201
            ):
                raise serializers.ValidationError(
                    "continuous_cdf invalid. Must be increasing, have 201 points, "
                    "and respect the bounds of the question"
                )

it would be AWESOME if that were broken up into four different checks, each of which contribute a more detailed component to the validation error, like "provided value for continuous_cdf is invalid because it failed the "increasing" rule: every value must be greater than the one before, and elements 19 (0.66) and 21 (0.659) are the first to break this rule AND the "upper_bound" rule: blah blah etc, with as much detail as possible.

I've observed many exceptions in the API with this issue, but they're the most perplexing when these rules are not present in the documentation, so perhaps a doc fix would be a more reasonable solution than a code fix.

[jkraybill]

I actually worked out what I was doing wrong in this particular example -- first value can't be less than 0.001 and last value can't be more than 0.999. However i'm still very puzzled at why I must submit both a CDF and the slider settings!

[lsabor]

@jkraybill Sorry, I think the documentation (https://www.metaculus.com/api/) is wrong regarding slider values. The slider values are not required and not checked on the backend. We haven't yet set up support for having your custom cdf to be displayed on the prediction maker, so you won't see it reflected there.

The rules for CDF are as follows:

• if the question has a closed lower bound (open_lower_bound == False) then the first value of the cdf must be 0.0, otherwise it has to be no less than 0.001 (0.1% of mass must be assigned below lower bound)
• if the question has a closed upper bound(open_upper_bound == False) then the last value of the cdf must be 1.0, otherwise it has to be no greater than 0.999 (0.1% of mass must be assigned above upper bound)
• the cdf must be strictly increasing by at least 0.01 / 201 (I think it maybe should be 0.01 / 200 actually, but the code says 201) at each point. This sets a lower bound for the minimum of the equivalent pdf, and gaurantees that 1.0% of mass must be assigned within bounds

You did correctly find where it is in the code. I'll take this as a call to improve the error statements and correct the documentation.

[malginin]

I'm getting the same error when clicking Save Change on a continuous question in the regular UI when my forecast has more than 5 components.

[SylvainChevalier]

@lsabor we also need a maximum to the difference between consecutive cdf points, to keep scores comparable with the graphical UI. The maximum pdf achievable in the UI is currently ~40, so the maximum difference between consecutive cdf points should be 40/200=0.2.

[lsabor]

Ok, I'll see what the empirical value maximum is and set it to that, since this validation runs on the slider-produced input as well.
I'll post here with details before merging.

[SylvainChevalier]

Ah, just remembered that pdfs can go higher near closed bounds, so the empirical maximum may be higher than 0.2.

[lsabor]

I think putting this restriction makes sense in the interim, but I'd prefer to eventually solidify rules that we want to be true, and then derive a few methods of forecasting that can all be done within the UI that always obey those rules by default.
One such option is to allow users to just enter in as many cdf points as they want in the UI, making there be no advantage between ui users and api users. But again, that's not within the scope of this ticket and requires a fair bit of thought and design.

[lsabor]

@SylvainChevalier I was able to get a maximum pmf value of 0.5988 from slider positions: {left: -0.047809999999999984, center: 0.00001, right: 0.03501000000000001} on a question with closed lower bound.
Pretty ridiculously high.