API

evansolomon edited this page Jul 8, 2012 · 7 revisions
Clone this wiki locally

IsValid.org API

Everything is json encoded and accepts GET or POST requests.

There is only one endpoint, /api. All requests require the function parameter.

Confidence Interval

Use case: Calculate the estimated lower, middle and upper bound for a population's conversion rate using sample data. This test is used to determine the appropriate margin for error given performance and sample size.

Required Parameters:

  • conversions: Integer greater than or equal to 0
  • samples: Integer greater than or equal to conversions

Optional Parameters:

  • confidence: Decimal greater than 0 and less than 1, .999 used by default.

Response:

  • results
    • low: Lower bound of confidence interval
    • average: Middle of confidence interval, same as observed conversion rate
    • high: Upper bound of confidence interval
  • chart: Google chart API URL

Example query: http://isvalid.org/api?function=confidence&conversions=100&samples=1000&confidence=.95

Example response:

{"results":
  {"low":0.081405860999113,"average":0.1,"high":0.11859413900089},
  "chart":"http:\/\/chart.apis.google.com\/chart?&chxl=0:||1:||&chxt=x,y&chs=500x250&chls=1,5,5|1|1,5,5|0|0|0&cht=gm&chd=t:8.1405860999113|10|11.859413900089"
}

Significance Test

Use case: Calculate the likelihood that an experimental variant is better than a control variant. This test is used to determine if an experimental variant is likely to be better than the control, but not by how much.

Required Parameters:

  • conversions_control: Integer greater than or equal to 0
  • samples_control: Integer greater than or equal to control conversions
  • conversions_experiment: Integer greater than or equal to 0
  • samples_experiment: Integer greater than or equal to experiment conversions)

Optional Parameters:

  • None

Response:

  • results
    • control: Likelihood that the control is the better variant
    • experiment: Likelihood that the experiment is the better variant
  • chart: Google chart API URL

Example query: http://isvalid.org/api?function=significance&conversions_control=100&samples_control=1000&conversions_experiment=150&samples_experiment=1000

Example response:

{"results":
  {"control":0.00034911083604428,"experiment":0.99965088916396},
  "chart":"http:\/\/chart.apis.google.com\/chart?chxl=0:|&chxt=y&chs=500x250&chls=2|0&cht=gm&chd=t:0.99965088916396"
}

Improvement

Use case: Calculate the estimated lower, middle and upper bound for an experiment's improvement relative to a control for a population using sample data. This test is used to determine how much better (or worse) an experimental variant is, with appropriate margin for error based on performance and sample size.

Required Parameters:

  • conversions_control: Integer greater than or equal to 0
  • samples_control: Integer greater than or equal to control conversions
  • conversions_experiment: Integer greater than or equal to 0
  • samples_experiment: Integer greater than or equal to experiment conversions

Optional Parameters:

  • confidence: Decimal greater than 0 and less than 1, .8 used by default.

Response:

  • results
    • low: Lower bound of confidence interval
    • average: Middle of confidence interval, same as observed improvement of experiment
    • high: Upper bound of confidence interval
  • chart: Google chart API URL

Example query: http://isvalid.org/api?function=improvement&conversions_control=100&samples_control=1000&conversions_experiment=150&samples_experiment=1000&confidence=.9

Example response:

{"results":
  {"low":0.25740646488421,"average":0.5,"high":0.74259353511579},
  "chart":"http:\/\/chart.apis.google.com\/chart?&chxl=0:||1:||&chxt=x,y&chs=500x250&chls=1,5,5|1|1,5,5|0|0|0&cht=gm&chd=t:62.870323244211|75|87.129676755789"
}

Complete

Use case: Everything. This is a combination of all the other functions

Required Parameters:

  • conversions_control: Integer greater than or equal to 0
  • samples_control: Integer greater than or equal to control conversions
  • conversions_experiment: Integer greater than or equal to 0
  • samples_experiment: Integer greater than or equal to experiment conversions

Optional Parameters:

  • confidence: Decimal greater than 0 and less than 1, .8 used by default)

Response:

  • confidence
    • results
      • low: Lower bound of confidence interval
      • average: Middle of confidence interval, same as observed conversion rate
      • high: Upper bound of confidence interval
    • chart: Google chart API URL
  • significance
    • results
      • control: Likelihood that the control is the better variant
      • experiment: Likelihood that the experiment is the better variant
    • chart: Google chart API URL
  • improvement
    • results
      • low: Lower bound of confidence interval
      • average: Middle of confidence interval, same as observed improvement of experiment
      • high: Upper bound of confidence interval
    • chart: Google chart API URL

Example query: http://isvalid.org/api?conversions_control=100&samples_control=1000&conversions_experiment=120&samples_experiment=1000&function=complete

Example response:

{
    "confidence":{
      "results":{
        "control":{
          "low":0.0687829406354,
          "average":0.1,
          "high":0.131217059365
        },
        "experiment":{
          "low":0.0861855214456,
          "average":0.12,
          "high":0.153814478554
        }
      },
      "chart":{
        "control":"http:\/\/chart.apis.google.com\/chart?&chxl=0:||1:||&chxt=x,y&chs=500x250&chls=1,5,5|1|1,5,5|0|0|0&cht=gm&chd=t:6.878294|10.000000|13.121706",
        "experiment":"http:\/\/chart.apis.google.com\/chart?&chxl=0:||1:||&chxt=x,y&chs=500x250&chls=1,5,5|1|1,5,5|0|0|0&cht=gm&chd=t:8.618552|12.000000|15.381448"
      }
    },
    "significance":{
      "results":{
        "control":0.0763540531417,
        "experiment":0.923645946858
      },
      "chart":"http:\/\/chart.apis.google.com\/chart?chxl=0:|&chxt=y&chs=500x250&chls=2|0&cht=gm&chd=t:92.364595"
    },
    "improvement":{
      "results":{
        "low":0.0207548379314,
        "average":0.2,
        "high":0.379245162069
      },
      "chart":"http:\/\/chart.apis.google.com\/chart?&chxl=0:||1:||&chxt=x,y&chs=500x250&chls=1,5,5|1|1,5,5|0|0|0&cht=gm&chd=t:51.037742|60.000000|68.962258"
    }
}

Errors

Appropriate error messages will be returned for malformed data or functional arguments with this form:

{"error":{
  "message":"Invalid data",
  "debug":{
    "file":"api.php",
    "line":101,
    "source":"https:\/\/github.com\/evansolomon\/IsValid.org\/blob\/master\/api.php#L101"}
  }
}