Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Comparing changes

Choose two branches to see what's changed or to start a new pull request. If you need to, you can also compare across forks.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also compare across forks.
base fork: assaf/vanity.js
base: d52ec2ecd5
...
head fork: assaf/vanity.js
compare: 15f32ad3b4
  • 6 commits
  • 11 files changed
  • 0 commit comments
  • 1 contributor
View
57 server/README.md
@@ -164,3 +164,60 @@ events.addEventListener("activity", function(event) {
DELETE /v1/activity/:id
```
+
+### Participate In Split Test
+
+```
+PUT /v1/split/:test/:participant
+```
+
+Request path specifies the test and participant identifiers.
+
+The body is a JSON document (form parameters also supported) with the following
+properties:
+
+* alternative - Alternative number (0 ... n)
+* outcome - A numeric value
+
+To indicate that participant is taking part in this split test, send a request
+with alternative number. You can send this request any number of times, only
+the first update is stored.
+
+To indicate that participant converted, send a request with alternative number
+abd the outcome. You can send this request any number of times, only the first
+update is stored.
+
+If successful, this request returns status code 200 and a JSON document with the
+following properties:
+
+* participant - Participant identifeir
+* alternative - Alternative number
+* outcome - Recorded outcome
+
+If participant was already added with a different alternative, the request
+returns status code 409 (Conflict) and the above JSON document.
+
+If the test identifier, alternative number or outcome are invalid, the request
+returns status code 400.
+
+
+### Retrieve Participation In Split Test
+
+```
+GET /v1/split/:test/:participant
+```
+
+Request path specifies the test and participant identifiers.
+
+If successful, this request returns status code 200 and a JSON document with the
+following properties:
+
+* participant - Participant identifeir
+* joined - When participant joined this test (RFC3339)
+* alternative - Alternative number
+* completed - When participant completed this test (RFC3339)
+* outcome - Recorded outcome
+
+If the participant never joined this split test, the request returns status code
+404.
+
View
1  server/config/default.config
@@ -1,6 +1,5 @@
#!coffee
-
# ElasticSearch server and index name
elasticsearch:
hostname: "localhost"
View
2  server/config/production.config
@@ -0,0 +1,2 @@
+#!coffee
+
View
6 server/config/redis.coffee
@@ -0,0 +1,6 @@
+Redis = require("redis")
+
+redis = Redis.createClient() #port, hostname)
+
+
+module.exports = redis
View
192 server/models/split_test.coffee
@@ -0,0 +1,192 @@
+redis = require("../config/redis")
+
+
+# A split test. A split test records any number of participants, which
+# alternative was presented to each one, and if they converted, the outcome.
+#
+# Each participant is known by its unique identifier. The alternative is a
+# number starting with zero for the first alternative. When the participant
+# first joins the split test, the alternative is stored and cannot change
+# afterwards. In addition, the timestamp is also recorded.
+#
+# At some point the participant converts, and we record the outcome (a numeric
+# value) and timestamp. Only the first such occurrence is stored.
+class SplitTest
+
+
+ # Storage
+ #
+ # The hash vanity.split.:test.participants is a map from participant
+ # identifier to alternative number. We use this to note that a participant
+ # joined the test and what alternative was shown to them.
+ #
+ # The sorted set vanity.split.:test.joined.:alt stores when each participant
+ # joined that test, using the timestamp as the score. That allows us to
+ # retrieve all participants for a given time range. Having a set for each
+ # alternative also makes it cheap to count participants for each alternative.
+ #
+ # If failure happens, it is possible that a participant will be recorded in
+ # the hash but not the sorted set. This is rare enough to not skew any
+ # numbers, but statistics should always be presented from a consisted set
+ # (i.e. the joined sorted set).
+ #
+ # The hash vanity.split.:test.outcome is a map from participant identifier to
+ # outcome value. We use it to nore that a participant completed the test and
+ # with what outcome.
+ #
+ # The sorted set vanity.split.:test.completed.:alt stores when each
+ # participant completed the test. As with joined sorted set, it allows us to
+ # perform quick counts for each alternative.
+ #
+ # If failure happens, it is possible that a participant outcome will be
+ # recorded in the hash but not the sorted set.
+
+
+ # Namespace
+ @NAMESPACE = "vanity.split"
+
+ # Create a new split test with the given identifier. Throws exception is the
+ # identifier is invalid.
+ constructor: (id)->
+ unless id && /^[\w]+$/.test(id)
+ throw new Error("Split test identifier may only contain alphanumeric, underscore and hyphen")
+ @_base_key = "#{SplitTest.NAMESPACE}.#{id}"
+
+
+ # Adds a participant.
+ #
+ # Arguments are:
+ # participant - Participant identifier
+ # alternative - Alternative number
+ # callback - Receive error or response
+ #
+ # Throws an exception if participant identifier or alternative number are
+ # invalid.
+ #
+ # Callback recieves error and result object with:
+ # participant - Participant identifier
+ # alternative - Alternative number
+ #
+ # Note that only the first alternative number is stored, and the alternative
+ # passed to the callback is that first value.
+ addParticipant: (participant, alternative, callback)->
+ participant = participant.toString() unless Object.isString(participant)
+
+ unless Object.isNumber(alternative)
+ throw new Error("Missing alternative number")
+ if alternative < 0
+ throw new Error("Alternative cannot be a negative number")
+ unless alternative == Math.floor(alternative)
+ throw new Error("Alternative must be an integer")
+
+ # First check if we already know which alternative was presented.
+ redis.hget "#{@_base_key}.participants", participant, (error, known)=>
+ return callback(error) if error
+ if known != null
+ # Respond with identifier and alternative
+ callback error,
+ participant: participant
+ alternative: parseInt(known)
+ return
+
+ # Set the alternative if not already set (avoid race condition).
+ redis.hsetnx "#{@_base_key}.participants", participant, alternative, (error, changed)=>
+ return callback(error) if error
+ unless changed # Someone beat us to it
+ @getParticipant participant, callback
+ return
+
+ # Keep record of when participant joined
+ redis.zadd "#{@_base_key}.joined.#{alternative}", Date.now(), participant, (error)->
+ callback error,
+ participant: participant
+ alternative: alternative
+ return
+
+
+ # Sets the outcome, but also adds participant if not already in this split
+ # test.
+ #
+ # Arguments are:
+ # participant - Participant identifier
+ # alternative - Alternative number
+ # outcome - Outcome
+ # callback - Receive error or response
+ #
+ # Throws an exception if participant identifier, alternative number or outcome
+ # are invalid.
+ #
+ # Callback recieves error and result object with:
+ # participant - Participant identifier
+ # alternative - Alternative number
+ # outcome - Outcome
+ #
+ # Note that only the first alternative number and outcomes are stored, and the
+ # values passed to the callback are those first stored.
+ setOutcome: (participant, alternative, outcome, callback)->
+ unless outcome == null || outcome == undefined || Object.isNumber(outcome)
+ throw new Error("Outcome must be numeric value")
+
+ @addParticipant participant, alternative, (error, result)=>
+ return callback(error) if error
+ if outcome == null || outcome == undefined
+ callback null, result
+ return
+
+ redis.hsetnx "#{@_base_key}.outcomes", participant, outcome, (error, changed)=>
+ return callback(error) if error
+ unless changed # Someone beat us to it
+ @getParticipant participant, callback
+ return
+
+ result.outcome = outcome
+ redis.zadd "#{@_base_key}.completed.#{result.alternative}", Date.now(), participant, (error)->
+ callback error, result
+
+
+ # Retrieves information about a participant.
+ #
+ # Arguments are:
+ # participant - Participant identifier
+ #
+ # Callback recieves error and result object with:
+ # participant - Participant identifier
+ # alternative - Alternative number
+ # joined - When participant joined the test (Date)
+ # outcome - Outcome
+ # completed - When participant completed the test (Date)
+ #
+ # If the participant never joined this split test, the callback receives null.
+ getParticipant: (participant, callback)->
+ redis.hget "#{@_base_key}.participants", participant, (error, alternative)=>
+ return callback(error) if error
+ if alternative == null
+ # Identifier doesn't match any participant
+ callback()
+ return
+
+ # Get when participant joined this test
+ redis.zscore "#{@_base_key}.joined.#{alternative}", participant, (error, score)=>
+ return callback(error) if error
+ # We have enough result for participant with no outcome
+ result =
+ participant: participant
+ alternative: parseInt(alternative)
+ joined: Date.create(score)
+
+ redis.hget "#{@_base_key}.outcomes", participant, (error, outcome)=>
+ return callback(error) if error
+ if outcome == null
+ # Participant did not converat
+ callback(null, result)
+ return
+
+ result.outcome = outcome
+ # Get when participant compeleted this test
+ redis.zscore "#{@_base_key}.completed.#{alternative}", participant, (error, score)->
+ return callback(error) if error
+ result.completed = Date.create(score)
+ callback null, result
+
+
+module.exports = SplitTest
View
6 server/package.json
@@ -12,8 +12,8 @@
},
"dependencies": {
"async": "~0.1.18",
- "coffee-script": "~1.2",
- "connect": "~1.8",
+ "coffee-script": "~1.2.0",
+ "connect": "~1.8.6",
"eco": "~1.1.0",
"elastical": "~0.0.7",
"express": "~2.5.8",
@@ -26,7 +26,7 @@
"devDependencies": {
"crossfilter": "~1.0.2",
"d3": "~2.8.1",
- "mocha": "~0.14",
+ "mocha": "~1.0.0",
"replay": "~1.3.1",
"vanity": "~0.2.2",
"zombie": "~0.12.15"
View
67 server/routes/api_split_tests.coffee
@@ -1,15 +1,62 @@
logger = require("../config/logger")
+redis = require("../config/redis")
server = require("../config/server")
+SplitTest = require("../models/split_test")
-Redis = require("redis")
-redis = Redis.createClient() #port, hostname)
+# Add participant or record outcome.
+#
+# Path includes test and participant identifier.
+#
+# Request is a JSON document with the following properties:
+# alternative - Alternative number (0 ... n)
+# outcome - A number
+#
+# With only alternative, adds participant to split test.
+#
+# With alternative and outcome, records conversion with the specified value.
+#
+# If successful, returns the status code 200. If participant already added with
+# a different alternative, returns the status code 409. For invalid outputs,
+# returns status code 400.
+#
+# In all cases, returns a JSON document with the following properties:
+# alternative - Alternative number
+# outcome - Recorded outcome
+server.put "/v1/split/:test/:participant", (req, res, next)->
+ { alternative, outcome } = req.body
+ try
+ test = new SplitTest(req.params.test)
+ test.setOutcome req.params.participant, alternative, outcome, (error, result)->
+ if error
+ next(error)
+ else if result.alternative == alternative
+ res.send result, 200
+ else
+ res.send result, 409
+ catch error
+ res.send error.message, 400
-server.post "/v1/split/:test/:id", (req, res, next)->
- { test, id } = req.params
- redis.hsetnx "vanity.split.#{test}.joined", "joined", Date.create(), ->
- console.log arguments
- if req.body.alternative
- redis.hsetnx "vanity.split.#{test}.#{id}", "alt", req.body.alternative, ->
- console.log arguments
- res.send {}
+
+# Returns information about participant.
+#
+# Path includes test and participant identifier.
+#
+# Response JSON document includes the following properties:
+# participant - Identifier
+# joined - Timestamp when participant joined experiment
+# alternative - Alternative number
+# completed - Timestamp when participant completed experiment
+# outcome - Recorded outcome
+server.get "/v1/split/:test/:participant", (req, res, next)->
+ try
+ test = new SplitTest(req.params.test)
+ test.getParticipant req.params.participant, (error, result)->
+ if error
+ next(error)
+ else if result
+ res.send result
+ else
+ res.send 404
+ catch error
+ res.send error.message, 404
View
27 server/server.js
@@ -1,10 +1,10 @@
#!/usr/bin/env node
-var FS = require("fs"),
- Path = require("path"),
- HTTP = require("http"),
- Up = require("up"),
- OS = require("os"),
- port, workers, timeout, watch, pid;
+var FS = require("fs"),
+ Path = require("path"),
+ HTTP = require("http"),
+ Up = require("up"),
+ OS = require("os"),
+ workers, watch, port, pid;
// Default environment is development, saves us from accidentally connecting to
@@ -13,25 +13,26 @@ process.env.NODE_ENV = process.env.NODE_ENV || "development";
// Configuration for production and development.
if (process.env.NODE_ENV == "production") {
- port = 80;
workers = OS.cpus().length;
+ pid = Path.resolve(__dirname, "tmp/pids/server.pid");
+ port = 80;
timeout = 60000;
- pid = __dirname + "/tmp/pids/server.pid";
} else {
- port = 3000;
workers = 1;
- timeout = 1000;
watch = true;
+ port = 3000;
+ timeout = 1000;
}
+port = parseInt(process.env.PORT || port, 10);
+timeout = parseInt(process.env.TIMEOUT || timeout, 10);
// Fire up the workers.
var httpServer = HTTP.Server().listen(port),
- server = Up(httpServer, __dirname + "/config/worker.js", { numWorkers: workers, workerTimeout: timeout });
+ server = Up(httpServer, Path.resolve(__dirname, "config/worker.js"), { numWorkers: workers, workerTimeout: timeout });
-if (pid) {
+if (pid)
FS.writeFileSync(pid, process.pid.toString());
-}
process.on("SIGUSR2", function () {
console.log("Restarting ...");
View
187 server/test/api_split_test.coffee
@@ -4,38 +4,201 @@ Async = require("async")
{ EventEmitter } = require("events")
request = require("request")
EventSource = require("./event_source")
+redis = require("../config/redis")
describe "API split", ->
+ base_url = "http://localhost:3003/v1/split/foobar/"
+
before Helper.setup
# -- Adding participant --
describe "add participant", ->
- statusCode = body = headers = null
+ statusCode = body = null
before (done)->
- params =
- alternative: 2
- request.post "http://localhost:3003/v1/split/foobar/8fea081c", json: params, (_, response)->
- { statusCode, headers, body } = response
+ request.put base_url + "8fea081c", json: { alternative: 2 }, (_, response)->
+ { statusCode, body } = response
done()
it "should return 200", ->
assert.equal statusCode, 200
it "should return participant identifier", ->
- assert.equal body.id, "8fea081c"
-
- it "should return test identifier", ->
- assert.equal body.test, "foobar"
+ assert.equal body.participant, "8fea081c"
it "should return alternative number", ->
assert.equal body.alternative, 2
- it "should return joined timestamp", ->
- joined = Date.create(body.joined)
- assert.equal joined - Date.now() < 1000
+ it "should store participant", (done)->
+ request.get base_url + "8fea081c", (_, { body })->
+ result = JSON.parse(body)
+ assert.equal result.participant, "8fea081c"
+ assert.equal result.alternative, 2
+ assert Date.create(result.joined) - Date.now() < 1000
+ done()
+
+
+ # -- Change alternative --
+
+ describe "change participant", ->
+ statusCode = body = null
+
+ before (done)->
+ request.put base_url + "ad9fe6597", json: { alternative: 2 }, (_, response)->
+ request.put base_url + "ad9fe6597", json: { alternative: 3 }, (_, response)->
+ { statusCode, body } = response
+ done()
+
+ it "should return 409", ->
+ assert.equal statusCode, 409
+
+ it "should return participant identifier", ->
+ assert.equal body.participant, "ad9fe6597"
+
+ it "should return original alternative", ->
+ assert.equal body.alternative, 2
+
+
+ # -- Adding participant and settings outcome --
+
+ describe "add participant and set outcome", ->
+ statusCode = body = null
+
+ before (done)->
+ request.put base_url + "b6f34cba", json: { alternative: 3 }, (_, response)->
+ request.put base_url + "b6f34cba", json: { alternative: 3, outcome: 5.6 }, (_, response)->
+ { statusCode, body } = response
+ done()
+
+ it "should return 200", ->
+ assert.equal statusCode, 200
+
+ it "should return participant identifier", ->
+ assert.equal body.participant, "b6f34cba"
+
+ it "should return alternative number", ->
+ assert.equal body.alternative, 3
+
+ it "should return outcome value", ->
+ assert.equal body.outcome, 5.6
+
+ it "should store participant", (done)->
+ request.get base_url + "b6f34cba", (_, { body })->
+ result = JSON.parse(body)
+ assert.equal result.participant, "b6f34cba"
+ assert.equal result.alternative, 3
+ assert Date.create(result.joined) - Date.now() < 1000
+ assert.equal result.outcome, 5.6
+ assert Date.create(result.completed) - Date.now() < 1000
+ done()
+
+
+ describe "set outcome without adding participant", ->
+ statusCode = body = null
+
+ before (done)->
+ request.put base_url + "d5df3958", json: { alternative: 1, outcome: 78 }, (_, response)->
+ { statusCode, body } = response
+ done()
+
+ it "should return 200", ->
+ assert.equal statusCode, 200
+
+ it "should return participant identifier", ->
+ assert.equal body.participant, "d5df3958"
+
+ it "should return alternative number", ->
+ assert.equal body.alternative, 1
+
+ it "should return outcome value", ->
+ assert.equal body.outcome, 78
+
+ it "should store participant", (done)->
+ request.get base_url + "d5df3958", (_, { body })->
+ result = JSON.parse(body)
+ assert.equal result.participant, "d5df3958"
+ assert.equal result.alternative, 1
+ assert Date.create(result.joined) - Date.now() < 1000
+ assert.equal result.outcome, 78
+ assert Date.create(result.completed) - Date.now() < 1000
+ done()
+
+
+
+ # -- Error handling --
+
+ describe "missing alternative", ->
+ statusCode = body = null
+
+ before (done)->
+ request.put base_url + "fbb28a111", json: { alternative: "" }, (_, response)->
+ { statusCode, body } = response
+ done()
+
+ it "should return 400", ->
+ assert.equal statusCode, 400
+
+ it "should return error", ->
+ assert.equal body, "Missing alternative number"
+
+ describe "alternative is negative", ->
+ statusCode = body = null
+
+ before (done)->
+ request.put base_url + "b715d1f4e", json: { alternative: -5 }, (_, response)->
+ { statusCode, body } = response
+ done()
+
+ it "should return 400", ->
+ assert.equal statusCode, 400
+
+ it "should return error", ->
+ assert.equal body, "Alternative cannot be a negative number"
+
+ describe "alternative is decimal", ->
+ statusCode = body = null
+
+ before (done)->
+ request.put base_url + "76c18f432", json: { alternative: 1.02 }, (_, response)->
+ { statusCode, body } = response
+ done()
+
+ it "should return 400", ->
+ assert.equal statusCode, 400
+
+ it "should return error", ->
+ assert.equal body, "Alternative must be an integer"
+
+
+ describe "invlid test identifier", ->
+ statusCode = body = null
+
+ before (done)->
+ url = "http://localhost:3003/v1/split/foo+bar/76c18f432"
+ request.put url, json: { alternative: 1 }, (_, response)->
+ { statusCode, body } = response
+ done()
+
+ it "should return 400", ->
+ assert.equal statusCode, 400
+
+ it "should return error", ->
+ assert.equal body, "Split test identifier may only contain alphanumeric, underscore and hyphen"
+
+
+ describe "outcome is NaN", ->
+ statusCode = body = null
+
+ before (done)->
+ request.put base_url + "43c1c137", json: { alternative: 1, outcome: "NaN" }, (_, response)->
+ { statusCode, body } = response
+ done()
+ it "should return 400", ->
+ assert.equal statusCode, 400
+ it "should return error", ->
+ assert.equal body, "Outcome must be numeric value"
View
9 server/test/helper.coffee
@@ -1,14 +1,21 @@
process.env.NODE_ENV = "test"
+Async = require("async")
Browser = require("zombie")
Replay = require("replay")
server = require("../config/server")
+redis = require("../config/redis")
Activity = require("../models/activity")
Helper =
setup: (callback)->
- server.listen 3003,callback
+ Async.series [
+ (done)->
+ redis.flushdb done
+ , (done)->
+ server.listen 3003, done
+ ], callback
newIndex: (callback)->
Activity.deleteIndex (error)->
View
3  server/test/mocha.opts
@@ -0,0 +1,3 @@
+--compilers coffee:coffee-script
+--globals url
+--reporter spec

No commit comments for this range

Something went wrong with that request. Please try again.