Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Figure out the 4+ tests that sporadically fail #479

Open
carols10cents opened this Issue · 11 comments

6 participants

@carols10cents

We have one skipped test that sometimes fails on travis but doesn't seem to fail locally, and there's another one that is NOT currently skipped that has the same behavior. It's this one:

test_0002_unfollows_another_user(FollowingSpec::OnRstatUsSpec):
Capybara::ElementNotFound: no button with value or id or text 'unfollow-4f2c40089870160d4d000947' found

See these failing builds to see the test(s?) I'm talking about:

http://travis-ci.org/#!/hotsh/rstat.us/builds/625838
http://travis-ci.org/#!/hotsh/rstat.us/builds/622687
http://travis-ci.org/#!/hotsh/rstat.us/builds/622614

There's pretty much no way the changes in those builds could have caused those tests to fail, so there's something else going on here.

It's really getting annoying, so if anyone figures out what is going on here and fixes it, it would be VERY much appreciated.

@carols10cents

I skipped the other one too.

@carols10cents

UGH it looks like being able to find the delete link that has the text "I regret this" fails sometimes as well. I just skipped 2 more tests in acceptance/update_test.rb.

See Travis builds #129 and #135; those changes had nothing to do with being able to find the delete link.

@carols10cents

Ok, there are more of these now, I've had it.

@carols10cents

The new failures, and most of the old ones, appear to be tests that make external requests that would pass if you were using a previously-successful VCR cassette. That would explain why the failures are transient-- sometimes the external requests work and sometimes they don't.

I'm thinking about checking in the VCR cassettes I have in order to solve this problem. This introduces the possibility that we won't be notified through a test failure that an external site has changed, but I think the tradeoff of getting more stable test runs would be worth it.... Anyone have any thoughts before I do this?

One of the tests that we've skipped for a while does not use VCR; I'm trying a few travis runs that print out more of what the travis tests are seeing in my repo to try and figure that one out.

@carols10cents

Adding the VCR cassettes seems to have helped some, but there are still 2 tests that fail a lot (but not every time) on travis, and never locally.

For serious, if anyone can fix this, name your reward. I will buy you coffee or beer. Or chocolate. Or pizza. Or an edible bouquet of fruit.

@Spaceghost

Which tests are still broken on this? Any?

@carols10cents

There are four pending tests that are marked with "skip" (they show up as "S" when you run the tests right now).

I actually saw a NEW one that is NOT skipped currently the other day on travis here: http://travis-ci.org/#!/hotsh/rstat.us/jobs/2509375

@carols10cents carols10cents referenced this issue from a commit in carols10cents/rstat.us
@carols10cents carols10cents Add heisenbug logging that prints to STDOUT in the cases we know of
That way, we can get a little more information when these tests fail on Travis, which appears to be the only place they ever fail. See notes in Issue #479.
691c968
@geopet

@carols10cents I'm trying to reproduce this bug and I'm not seeing the skips when running rake test. I'm likely doing something wrong here. Any suggestions? :cat:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.