Permalink
Commits on Apr 16, 2014
  1. Cut version 2.1.0

    Shane da Silva committed Apr 16, 2014
    Change-Id: Ib0fb391b5393dbbc52da50ab7b3843bcfd555de4
    Reviewed-on: http://gerrit.causes.com/37318
    Reviewed-by: Shane da Silva <shane@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
  2. Add ability to exclude sets of specs

    Shane da Silva committed Apr 16, 2014
    We recently needed the ability to exclude a directory of specs (in our
    case, integration tests) from Buffet runs. Add an option to support
    this.
    
    Change-Id: I8ec2f4f6f9bcba50703058ec687c764a41cb4dc5
    Reviewed-on: http://gerrit.causes.com/37317
    Reviewed-by: Brian Mason <brian.mason@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
Commits on Apr 4, 2014
  1. Cut version 2.0.1

    Shane da Silva committed Apr 4, 2014
    Change-Id: If8f7fdc8f5c515ac68a4cd2a2135565bc29389f9
    Reviewed-on: http://gerrit.causes.com/36924
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Henric Trotzig <henric.trotzig@causes.com>
    Reviewed-by: Nebojsa Petrovic <nebs@causes.com>
  2. Fix infinite loop in spurious failure detection

    Shane da Silva committed Apr 4, 2014
    There was a situation where Buffet's detection of spurious failures
    could result in an infinite loop.
    
    For the simplest case, consider a spec with 2 examples A and B. If on
    the first run only example A fails, Buffet will run the spec again to
    confirm the failure. If then on every run thereafter example B fails,
    Buffet will continue to rerun the spec indefinitely.
    
    This was caused by the use of the `unconfirmed_failures?` method in
    `rerun_spec?`, which would always return a number > 0 for the case
    outlined above. This is because example A will always only have failed
    once, so it falls within the range of being "unconfirmed" as a failure.
    
    The solution was to add a clause which forces `rerun_spec?` to return
    false if the spec has already been run a minimum number of times.
    
    In the process, the failure threshold for spurious failures was upped to
    3 so that a quorum is forced in the event of a spurious failure--having
    1 failure and 1 success doesn't really tell us that the example is
    successful, so we force it to either be 2 failures and 1 success or vice
    versa.
    
    Also modified the `Runner` to output the spurious failures even if there
    are failures (changing an `elsif` to a separate `if` clause), since we
    still want to see the spurious failures.
    
    Change-Id: Ibfb66f52cde5eccc3713dbb8c77f8a8f7980cbd7
    Reviewed-on: http://gerrit.causes.com/36923
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Henric Trotzig <henric.trotzig@causes.com>
    Reviewed-by: Nebojsa Petrovic <nebs@causes.com>
Commits on Feb 11, 2014
  1. Cut version 2.0.0

    Shane da Silva committed Feb 10, 2014
    We've broken backwards compatibility by renaming some options and
    changing how the prepare_command and worker_command scripts are
    executed. While this perhaps isn't a fancy "shiny and new" major
    release, it is in line with the semantic versioning protocol.
    
    Change-Id: I73157abac00e066477df2420315b6173b617e404
    Reviewed-on: https://gerrit.causes.com/33872
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Shane da Silva <shane@causes.com>
  2. Rename prepare_script option to prepare_command

    Shane da Silva committed Feb 10, 2014
    To make the `prepare_script` configuration option more consistent with
    the `worker_command` option, rename it to `prepare_command`.
    
    Change-Id: I868afd1f3c4775fa63e9ad5fc872880d4d10e74e
    Reviewed-on: https://gerrit.causes.com/33871
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Shane da Silva <shane@causes.com>
  3. Remove support for RSpec 1

    Shane da Silva committed Feb 10, 2014
    RSpec 3 is about to be released, and maintaining the code for the old
    RSpec 1 seemed unnecessary.
    
    Removing support for RSpec 1 simplifies the code.
    
    Change-Id: Ie48297716da589e54d31ff37b9bfd3f3ceb46a06
    Reviewed-on: https://gerrit.causes.com/33870
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Shane da Silva <shane@causes.com>
  4. Specify Buffet master/project with environment variables

    Shane da Silva committed Feb 10, 2014
    Previously, Buffet would pass the master and project name via
    command-line arguments to the `prepare_script`, but not the
    `worker_command`.
    
    In order to make working with these scripts consistent, and also to
    allow the `worker_command` to have knowledge of the master and project,
    change the mechanism to pass these arguments via environment variables.
    
    Change-Id: I9b1d9570a985f6133500d3389c4624a4cdd43f55
    Reviewed-on: https://gerrit.causes.com/33869
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Shane da Silva <shane@causes.com>
Commits on Jan 16, 2014
  1. Cut version 1.4.0

    Shane da Silva committed Jan 16, 2014
    Change-Id: I50999668bbb8bb861d5ef1c72fa4bbd6b057c48f
    Reviewed-on: https://gerrit.causes.com/33030
    Reviewed-by: Shane da Silva <shane@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
  2. Allow display of progress dots to be disabled

    Shane da Silva committed Jan 16, 2014
    Add a `display_progress` option to `buffet.yml` which disables the
    display of progress dots. This is useful for tools like Jenkins where
    the dots are of no significant value.
    
    This is a bit of a hack for now. In the future, we'll want to have a
    bunch of formatters that can be specified instead of just
    enabling/disabling dots.
    
    Change-Id: Ia8856f80eccf77fefb37f284ff9e2673d309f654
    Reviewed-on: https://gerrit.causes.com/33029
    Reviewed-by: Shane da Silva <shane@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
  3. Fix broken --version flag

    Shane da Silva committed Jan 16, 2014
    This was broken due to the version file not being required ahead of
    time.
    
    Change-Id: I2d3efd785e13d12badbcedc91a24238a1f232cc5
    Reviewed-on: https://gerrit.causes.com/33028
    Reviewed-by: Shane da Silva <shane@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
  4. Add CHANGELOG

    Shane da Silva committed Jan 16, 2014
    Backfill the changelog file using the git history. We'll be keeping this
    up to date from now on.
    
    Change-Id: I1af3922ef5ec9804bd4d59df16e0183bb4d87b8a
    Reviewed-on: https://gerrit.causes.com/33027
    Reviewed-by: Shane da Silva <shane@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
  5. Upgrade RSpec 2.6.0 -> 2.14.1

    Shane da Silva committed Jan 15, 2014
    Buffet hasn't had it's version of RSpec updated in a long time. Update
    it now and fix any deprecation warnings.
    
    Change-Id: I0b6ea685189cd472bfbee3c4476d577d4578b49a
    Reviewed-on: https://gerrit.causes.com/33026
    Reviewed-by: Shane da Silva <shane@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
  6. Remove Gemfile.lock

    Shane da Silva committed Jan 15, 2014
    Gems shouldn't include a Gemfile lock, so remove it.
    
    At the same time, update the gemspec to include the exact dependencies,
    so that we can still pin dependencies to the desired version.
    
    Change-Id: I81b3cc2304f036d15b2923f7b35a32b6fe7fc5c0
    Reviewed-on: https://gerrit.causes.com/33025
    Reviewed-by: Shane da Silva <shane@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
Commits on Nov 12, 2013
  1. Add gem version badge to README

    Shane da Silva committed Nov 11, 2013
    Also prettied-up the documentation by adding some backticks so fragments
    are rendered as code.
    
    Change-Id: I9ba8dd05e0cfa2f75a18df7a65f789ffc7c6e6e5
    Reviewed-on: https://gerrit.causes.com/30889
    Reviewed-by: Adam Olsen <adam.olsen@causes.com>
    Reviewed-by: Joe Lencioni <joe.lencioni@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
  2. Rename buffet-gem to buffet

    Shane da Silva committed Nov 11, 2013
    The authors of the `buffet` gem were kind enough to relinquish the
    namespace for us to use.
    
    Rename the gem to `buffet` so we can push it to that namespace. We don't
    need to update the version since the `buffet` namespace doesn't have any
    versions posted.
    
    Change-Id: I9539b3c50ed7c95947f03e9d0e177848e38edc60
    Reviewed-on: https://gerrit.causes.com/30888
    Reviewed-by: Adam Olsen <adam.olsen@causes.com>
    Reviewed-by: Joe Lencioni <joe.lencioni@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
Commits on Sep 5, 2013
  1. Bump version to 1.3.0

    Shane da Silva committed Sep 4, 2013
    Change-Id: I8983e36901422760dfd87fd39d43b2286486de0d
    Reviewed-on: https://gerrit.causes.com/27927
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Shane da Silva <shane@causes.com>
  2. Reorder Buffet output to be more useful

    Shane da Silva committed Sep 5, 2013
    99% of the time, when you look at the output of Buffet, you're looking
    for spec failures in Jenkins, which displays output from the top, not
    the bottom like in the console. The current ordering of output doesn't
    match this 99% use case.
    
    This commit changes the order to display the summary stats immediately
    followed by any spec failures.
    
    Change-Id: I606fd35f849125a5dfd589b5e6b041ffd3fae1b8
    Reviewed-on: https://gerrit.causes.com/27990
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Shane da Silva <shane@causes.com>
  3. Add spurious failure detection

    Shane da Silva committed Sep 4, 2013
    Unfortunately, it becomes inevitable that spurious failures occur within
    a test suite. Even more unfortunate is that they are usually difficult
    to track down and are often extremely difficult to reproduce.
    
    However, if a test spuriously fails less than some percentage of time,
    then we'd rather not invest the time to fix it, so long as we're
    convinced that when it passes in isolation it's because it is doing what
    we expect it to.
    
    In order to save engineering time in dealing with these spurious
    failures, add support for spurious failure detection to Buffet.
    
    This adds support for a single threshold customization representing how
    many times we allow a spec to fail before we consider it an actual
    failure.
    
    When at least one example in a spec fails, we re-run the spec and see if
    the example fails again. We keep doing this until all examples pass or
    all the currently failing examples have exceeded the threshold for the
    number of allowed failures.
    
    The default (and recommended value) is 2, as that basically says "you
    only get one do-over".
    
    Note that with this implementation, the worst-case number of times a
    spec will be re-run is N*T, where N is the number of examples in the
    spec and T is the failure threshold. Practically, this will never
    happen, as the chances of every example in the spec failing at least T
    times independently each spec run is incredibly unlikely, and is
    probably a sign that your test suite is borked in a really bad way.
    
    For normal test suites, this should work as expected as most of the time
    the vast majority of the spec suite is green. In the event you somehow
    break every example in your suite, it will at most double the length of
    your test run as they all run twice.
    
    We collect data on each spec run because we can't assume that an example
    can be uniquely identified by its file and line number. For example:
    
        [1, 2, 3].each do
          it { ... }
        end
    
    This will have multiple examples reported for the same file/line number
    but they need to be treated as distinct.
    
    If we make the assumption that the number and order of examples that run
    never changes across multiple spec runs, e.g. we don't have
    
        [1, 2, (3 if (rand < 0.5))].compact.each do
          it { ... }
        end
    
    or
    
        [1, 2, 3].shuffle.each do
          it { ... }
        end
    
    then we can still detect spurious failures.
    
    The insight is that if you run examples A, B, and C in a spec,
    then over multiple spec runs you will see a history of results like:
    
        [A, B, C, A, B, C, ...]
    
    Thus if we store the history of every example that is run over the
    course of all spec runs, and keep track of how many spec runs we've had,
    we can still keep track of individual examples A, B, and C and their
    individual success/failure status history.
    
    Change-Id: I1bd9e30a83f4cda4dfcbea2f61d566891f89c859
    Reviewed-on: https://gerrit.causes.com/27926
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Shane da Silva <shane@causes.com>
Commits on Sep 3, 2013
  1. Bump version to 1.2.9

    Shane da Silva committed Sep 2, 2013
    Change-Id: I198fb9433788f30c1a31562efdc4df1eb2e1d572
    Reviewed-on: https://gerrit.causes.com/27829
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Shane da Silva <shane@causes.com>
  2. Order specs by number of lines

    Shane da Silva committed Aug 29, 2013
    This will hopefully speed up test runs by biasing specs which take
    longer (assuming a heuristic that the number of lines in a spec is
    indicative of how long that spec takes to run) to be run first. This in
    theory means that we won't end up with any "slow stragglers" holding up
    the test run, as all the specs that run at the end will be small.
    
    Preliminary experiments show that test times drop from about 3 minutes
    to 2.5 minutes, and there is also less variation in test times.
    
    Change-Id: Ic5c16259717b2945dd07e4a8983ac93367e515a6
    Reviewed-on: https://gerrit.causes.com/27828
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Tom Dooner <tom.dooner@causes.com>
Commits on Aug 29, 2013
  1. Bump version to 1.2.8

    johnskopis committed Aug 29, 2013
    Change-Id: Iffd6d0d628ce437780f2f6d6801e5bc94f1e9ef3
    Reviewed-on: https://gerrit.causes.com/27699
    Reviewed-by: Shane da Silva <shane@causes.com>
    Reviewed-by: John Skopis <john.skopis@causes.com>
    Tested-by: John Skopis <john.skopis@causes.com>
  2. Import ci_reporter

    johnskopis committed Aug 29, 2013
    The ci_reporter gem can output junit xml. Jenkins can natively display
    the junit reports providing timing/trending data for each test case.
    
    The presence of the setting, 'gather_junit', will enable collection of
    the junit xml created by the ci_reporter on the buffet slaves.
    
    Change-Id: Ieebfa8c0afd87271732e917de8abacda86cbd9a6
    Reviewed-on: https://gerrit.causes.com/27663
    Reviewed-by: Greg Hurrell <greg@causes.com>
    Reviewed-by: John Skopis <john.skopis@causes.com>
    Tested-by: John Skopis <john.skopis@causes.com>
  3. Run each spec only once

    johnskopis committed Aug 29, 2013
    When 'spec_helper' requires 'rspec/autorun' specs run more than once.
    
    Calling `disable_autorun!` ensures each spec only runs once.
    
    Change-Id: I06ba739ce0ac3c311e81b485c52b7600a1908ccb
    Reviewed-on: https://gerrit.causes.com/27662
    Reviewed-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Greg Hurrell <greg@causes.com>
    Reviewed-by: John Skopis <john.skopis@causes.com>
    Tested-by: John Skopis <john.skopis@causes.com>
Commits on Aug 28, 2013
  1. Add version flag

    Shane da Silva committed Aug 28, 2013
    For convenience in determining which version of Buffet is installed, add
    a `--version` flag.
    
    Change-Id: I1aac363acf2bc4518934038038f0f92c843e4b77
    Reviewed-on: https://gerrit.causes.com/27555
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Henric Trotzig <henric.trotzig@causes.com>
Commits on Aug 27, 2013
  1. Bump version to 1.2.7

    Shane da Silva committed Aug 27, 2013
    Change-Id: I961d30301c61c183ce65fc6df8fadb11c59f8003
    Reviewed-on: https://gerrit.causes.com/27500
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Lann Martin <lann@causes.com>
  2. Show spec order for slaves with failures

    Shane da Silva committed Aug 27, 2013
    In order to make debugging spurious test failures easier, we want to
    keep track of the order in which a slave ran specs.
    
    This adds a `specs` slave stat which is appended to each time a slave
    takes a spec, and prints it out if a slave has any failures.
    
    Change-Id: Ibf5a57a7ddb32ea93fffce4c5c3773b4dd739c77
    Reviewed-on: https://gerrit.causes.com/27499
    Tested-by: Shane da Silva <shane@causes.com>
    Reviewed-by: Lann Martin <lann@causes.com>
Commits on Mar 19, 2013
  1. Exit with non-zero status when no examples run

    Shane da Silva committed Mar 19, 2013
    For our purposes, there isn't really any good reason to return an exit
    code of zero when no specs are run (it almost invariably means something
    went wrong).
    
    This changes the behaviour of Buffet to return a non-zero status code in
    the event no specs are run.
    
    Change-Id: I0394e8c40fd60ce3d577551e8d631d046bbf5f6c
    Reviewed-on: https://gerrit.causes.com/19828
    Reviewed-by: Shane da Silva <shane@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
  2. Prevent buffet from clearing shared examples

    Tony Wooster committed Mar 19, 2013
    Commit 8e8fb2b of rspec-core introduced a change that would clear all
    shared examples when `Rspec.world.reset` was called. This causes any
    global shared examples (for example, loaded in `spec_helper` for all
    example groups) to only be present on the first run of buffet and
    missing on all subsequent runs.
    
    This commit addresses this by no longer calling `world.reset`, but
    instead reaching directly into `world.example_groups` to clear only
    the loaded example groups.
    
    Change-Id: I729c0ab1d26f0fda091f656459cb2574664ae608
    Reviewed-on: https://gerrit.causes.com/19817
    Reviewed-by: Shane da Silva <shane@causes.com>
    Tested-by: Tony Wooster <tony.wooster@causes.com>
Commits on Jan 23, 2013
  1. Bump version for license update

    sectioneight committed Jan 23, 2013
    Change-Id: Ifebed417056bb316e12e98e8897bf207980410f5
    Reviewed-on: https://gerrit.causes.com/17680
    Reviewed-by: Aiden Scandella <aiden@causes.com>
    Tested-by: Aiden Scandella <aiden@causes.com>
  2. Adds license to gemspec

    cknadler committed with sectioneight Jan 23, 2013
    This allows RubyGems to display buffet's license information.
    
    Change-Id: I690a129a7e7c4de44341e5ce87c0477484ea0b7c
    Reviewed-on: https://gerrit.causes.com/17679
    Reviewed-by: Aiden Scandella <aiden@causes.com>
    Tested-by: Aiden Scandella <aiden@causes.com>
Commits on Dec 8, 2012
  1. Bump version number to 1.2.3

    Shane da Silva committed Dec 8, 2012
    Change-Id: I7e80a947780b1270d9592f812acce6e7067c3354
    Reviewed-on: https://gerrit.causes.com/16174
    Reviewed-by: Aiden Scandella <aiden@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
  2. Add 'allowed_prepare_slave_failures' setting

    Shane da Silva + Lann Martin committed with Shane da Silva Dec 6, 2012
    This setting specifies how many slaves can fail in the prepare_slave
    step without stopping the entire spec run, making it more resilient as
    we increase the number of possible points of failure by adding more
    machines.
    
    At a high level, this commit:
    
      * Changes the messaging structure of the Master and its threads to use
        a ConditionVariable to signal/wait for updates from threads
    
      * Adds support for respecting the `allowed_prepare_slave_failures`
        setting to continue the test run unless the threshold has been
        exceeded
    
      * Updates the output from the Runner to display slave preparation
        failures
    
    The bulk of this commit is in the redesigning of the Master to use a
    condition variable as a signal mechanism. Previously, the Master spawned
    a thread for each slave and then joined each of those threads so the
    main thread would halt until all slaves had finished.
    
    The problem with this approach is that if a failure occurs during
    preparation, we don't find out about it until after we have joined the
    failing thread. Thus a situation could arise where with 10 threads the
    first 8 we spawn have no problems and run to completion, but the last 2
    threads immediately fail due to being unable to connect to a host. In
    this case, even though we could have stopped execution of the entire run
    due to the 2 failures, we had to wait for the first 8 threads to finish.
    
    The solution is to use signals so that the Master blocks on a single
    condition variable, and can be signaled by any of the threads at any
    time.
    
    Change-Id: Id1852b4b7910c59a4262990447858c6a532ec703
    Reviewed-on: https://gerrit.causes.com/16103
    Reviewed-by: Lann Martin <lann@causes.com>
    Reviewed-by: Greg Hurrell <greg@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
Commits on Dec 7, 2012
  1. Raise CommandError when shell command fails

    Shane da Silva committed Dec 6, 2012
    In the effort to make Buffet resilient to hosts going down, it was first
    identified that we swallow all failures in external shell commands. This
    becomes problematic if we want to bubble up the failure to the Master to
    properly handle.
    
    The solution was to add a `CommandError` exception that would wrap the
    result of a shell command in the event a non-zero exit status was
    returned.
    
    Related Story:
      http://www.pivotaltracker.com/story/show/40166629
      ("Buffet should gracefully continue when a node is down")
    
    Change-Id: I22f433d352c524637860f5a4c8a19285fa9cae67
    Reviewed-on: https://gerrit.causes.com/16102
    Reviewed-by: Lann Martin <lann@causes.com>
    Reviewed-by: Aiden Scandella <aiden@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>
Commits on Dec 6, 2012
  1. Fix indentation of Master#run_slave

    Shane da Silva committed Dec 6, 2012
    Noticed this while working on another commit.
    
    Related Story:
      http://www.pivotaltracker.com/story/show/40166629
      ("Buffet should gracefully continue when a node is down")
    
    Change-Id: Ia3aa4fb0d3cddcfbb185a0da3a66dd3875053a95
    Reviewed-on: https://gerrit.causes.com/16101
    Reviewed-by: Lann Martin <lann@causes.com>
    Tested-by: Shane da Silva <shane@causes.com>