Permalink
Commits on Mar 23, 2017
  1. Merge pull request #323 from steveluscher/patch-1

    Correct the ‘satisfaction survey’ example
    phillbaker committed on GitHub Mar 23, 2017
Commits on Mar 22, 2017
Commits on Jan 14, 2017
Commits on Dec 21, 2016
  1. Bump version for bugfixes.

    phillbaker committed Dec 10, 2016
Commits on Nov 29, 2016
  1. Merge pull request #317 from Futurelearn/deprecate-ab-seen-with-alter…

    …native-instance
    
    Deprecate calling #ab_seen with alternative instance
    phillbaker committed on GitHub Nov 29, 2016
Commits on Nov 28, 2016
  1. Use alternative id in ab_seen calls, not Alternative instance

    In the previous commit, calling `#ab_seen` with an alternative instance was
    deprecated in favour of calling with an alternative id, to bring the method
    signature into line with the rest of the abstract adapter API.
    
    This updates the only internal caller to the new style.
    urbanautomaton committed Nov 23, 2016
  2. Deprecate #ab_seen with Alternative instance

    Currently all methods in the storage adapter interface that accept an
    `alternative` argument accept it in the form of an alternative ID. The
    exception to this is `#ab_seen`, which accepts an Alternative instance.
    
    Since third-party code might depend on this, this change at first deprecates
    the old usage while allowing both calling styles. The old usage can then be
    removed in a future version.
    urbanautomaton committed Nov 23, 2016
Commits on Nov 23, 2016
  1. Silence deprecation warnings from legacy connecting test

    This test involves resetting Vanity, which sets a new logger at the default
    level. This means that the legacy configuration method it supports prints a
    deprecation to the test output.
    
    We can capture the IO and assert the presence of a deprecation message
    instead using minitest's #capture_io, setting a new logger within the captured
    block.
    urbanautomaton committed Nov 23, 2016
Commits on Nov 21, 2016
  1. Merge pull request #316 from Futurelearn/use-shared-adapter-tests

    Use shared tests for all adapters
    phillbaker committed on GitHub Nov 21, 2016
Commits on Nov 20, 2016
  1. Run ActiveRecord and Mongodb adapter tests only when told

    The new adapter tests are effectively new levels of integration test against
    the different adapters. Because some test setup is only performed when the
    relevant `DB` value is present in the environment, these tests will fail the
    travis build unless restricted like this.
    urbanautomaton committed Nov 20, 2016
Commits on Nov 19, 2016
  1. Ignore multi-valued metrics in shared adapter tests

    Multi-valued metrics were introduced some time ago[1], but the public API
    supporting them (Metric#values, which returns values binned by date) has only
    ever returned a single value per day.[2]
    
    Currently the different adapters' implementations of #metric_values behave
    differently for multi-valued metrics.
    
    * The mock and mongodb adapters return an array per day, with the totals for
      each metric value seen per day
    * The redis adapter returns an array summing only the first metric value seen
      per day, as a string
    * The ActiveRecord adapter returns an array containing the sum of *all* values
      seen per day
    
    While I think this may be completely unused behaviour, I can't rule out
    third-party code that uses multi-valued metrics, so for the time being I'm
    loosening the test to ignore multi-valued metrics, and ensuring that the Redis
    adapter returns integers instead of strings.
    
    [1] 10b1523
    [2] https://github.com/assaf/vanity/blob/0c241e3e78d5593824125031f897a6ca4bfe7170/lib/vanity/metric/base.rb#L211
    urbanautomaton committed Nov 19, 2016
  2. Merge pull request #311 from sdhull/docs-update

    Docs source: Add note about rebalance_frequency for bayes_bandit_score
    phillbaker committed on GitHub Nov 19, 2016
  3. Loosen specification on ab_seen

    Some adapters return nil, others false, so the tests need loosening to capture
    this discrepancy.
    urbanautomaton committed Nov 19, 2016
  4. Use exposed specification to set up adapters

    While most of the adapters work with an empty options hash, the mongo adapter
    requires more configuration to instantiate successfully.
    
    I've split out the test config in test_helper.rb to allow direct access to the
    different configurations, and used the previously-introduced
    Connection::Specification class to parse these strings into options that can
    be passed to the adapters.
    urbanautomaton committed Nov 17, 2016
  5. Refactor connection to allow specs to be built elsewhere

    Now that more adapter tests are being added, it's useful to be able to
    instantiate an adapter without going via the Connection class.
    
    However, most of the test setup provides connection specs as strings,
    requiring an extra parsing step that's currently baked into Connection, making
    it hard for others to use.
    
    This extracts the spec parsing into a Specification class for external use,
    without altering the behaviour of Connection.
    urbanautomaton committed Nov 17, 2016
  6. Fix `ActiveRecordAdapter#ab_add_conversion` with implicit=true

    Previously this added a participant for the alternative and recorded a
    conversion for them, but did not record the alternative they had seen, meaning
    the ab_counts were wrong.
    
    This updates the implicit creation to record the alternative seen.
    urbanautomaton committed Nov 16, 2016
  7. Set up 'real' experiment for shared adapter tests

    Previously the shared adapter tests simply passed experiment and metric labels
    to the various methods, since the MockAdapter (where these tests were
    introduced) didn't care too much about whether an experiment was properly set
    up before storing data.
    
    Other adapters are more picky, however, so it's useful to have a
    properly-initiated experiment, using the new_ab_test helper. Tests
    specifically requiring a non-existent experiment now use a different
    experiment label.
    urbanautomaton committed Nov 15, 2016
  8. Adapter test for `#is_persisted?` should use created_at

    While MockAdapter tests for the presence of any experiment data at all to
    check persistence, other adapters check specifically for the created_at date,
    so the test needs to match this behaviour.
    urbanautomaton committed Nov 15, 2016
  9. Relax resolution on adapter timestamp tests

    While the mock adapter stores Time instances, other adapters are limited by
    their storage layer, usually to 1s accuracy, so we can only assert that
    timestamps are accurate to the second.
    urbanautomaton committed Nov 19, 2016
  10. Apply shared adapter tests to RedisAdapter

    At present these don't all pass for the RedisAdapter for a variety of reasons
    - fixes will follow in individual commits.
    urbanautomaton committed Nov 19, 2016
  11. Move DummyAlternative definition out of dynamic context

    In its original location, this constant assignment caused a ruby warning, so
    hoist it to a static location.
    urbanautomaton committed Nov 19, 2016
  12. Extract mock adapter tests to shareable module

    Previously a set of behavioural tests for MockAdapter were introduced.
    However, these tests cover behaviour that should be shared by all adapters,
    and provides coverage that the other adapters don't yet have.
    
    This change extracts the tests to a module that can be included in the
    existing and any new adapter tests, as well as used by anyone developing their
    own adapter.
    
    Clients only need to define an `#adapter` method in their test class;
    including the module then imports all the tests.
    urbanautomaton committed Nov 15, 2016
  13. Merge pull request #309 from Futurelearn/fix-mock-adapter

    Implement #ab_seen and #ab_assigned in MockAdapter
    phillbaker committed on GitHub Nov 19, 2016
Commits on Nov 18, 2016
  1. De-duplicate alternative fetching in MockAdapter

    Previously there was some boilerplate initialization of the alternatives
    storage; I've extracted this to a couple of private helper methods and updated
    the relevant call-sites.
    urbanautomaton committed Nov 15, 2016
  2. Implement missing methods on mock adapter

    Previously the ab_seen and ab_assigned methods on the mock adapter were
    missing; as they were defined in the abstract adapter, they raised "not
    implemented" errors if called.
    
    This adds implementations for the extra methods.
    urbanautomaton committed Nov 15, 2016
Commits on Nov 17, 2016
  1. Merge pull request #313 from terracatta/patch-1

    Resolve Rails 5 deprecation warning for Mime::HTML
    phillbaker committed on GitHub Nov 17, 2016
Commits on Nov 16, 2016
  1. Resolve Rails 5 deprecation warning for Mime::HTML

    Changes Mime::HTML to Mime[:html] to avoid a dep warning.
    terracatta committed on GitHub Nov 16, 2016
Commits on Nov 15, 2016
  1. Add tests for mock adapter

    The mock adapter is currently untested, and lacks implementations for a couple
    of methods in the abstract adapter definition (specifically `#ab_seen` and
    `#ab_assigned`).
    
    This adds a set of tests for all implemented methods - the missing ones will
    be added subsequently.
    urbanautomaton committed Nov 15, 2016
Commits on Nov 10, 2016
  1. Correct test DB gitignore entry

    The generated sqlite3 file lives directly in test/dummy, but we can catch all
    sqlite3 files in the dummy directory to cover future changes.
    urbanautomaton committed Nov 10, 2016
Commits on Oct 21, 2016
  1. Merge pull request #308 from bazzargh/patch-1

    Fix 'base 17' typo, correct id bucketing
    phillbaker committed on GitHub Oct 21, 2016
Commits on Oct 20, 2016
  1. Fix 'base 17' typo, correct id bucketing

    MD5.hexdigest provides a flat (but noisy) distribution of hex digits, but the typo converting them back to integers with base 17 adds a very slight bias to the results, increasing the size of some low-order buckets. eg this calculation using a range instead of MD5 to provide the evenly distributed input: (0..9999999).each_with_object(Hash.new(0)) {|d, h| h[d.to_s(16).to_i(17) % 100 < 5] += 1 } => {true=>500025, false=>9499975} . It's only a slight loading of the dice, and would be very hard to spot in real experiments because it's lost in the noise of the MD5 distribution. Spotted via code review not tests.
    bazzargh committed on GitHub Oct 20, 2016
Commits on Oct 4, 2016