Figure out the 4+ tests that sporadically fail #479

Open
carols10cents opened this Issue Feb 4, 2012 · 11 comments

Comments

Projects
None yet
6 participants
Contributor

carols10cents commented Feb 4, 2012

We have one skipped test that sometimes fails on travis but doesn't seem to fail locally, and there's another one that is NOT currently skipped that has the same behavior. It's this one:

test_0002_unfollows_another_user(FollowingSpec::OnRstatUsSpec):
Capybara::ElementNotFound: no button with value or id or text 'unfollow-4f2c40089870160d4d000947' found

See these failing builds to see the test(s?) I'm talking about:

http://travis-ci.org/#!/hotsh/rstat.us/builds/625838
http://travis-ci.org/#!/hotsh/rstat.us/builds/622687
http://travis-ci.org/#!/hotsh/rstat.us/builds/622614

There's pretty much no way the changes in those builds could have caused those tests to fail, so there's something else going on here.

It's really getting annoying, so if anyone figures out what is going on here and fixes it, it would be VERY much appreciated.

Contributor

carols10cents commented Feb 8, 2012

I skipped the other one too.

Contributor

carols10cents commented Feb 19, 2012

UGH it looks like being able to find the delete link that has the text "I regret this" fails sometimes as well. I just skipped 2 more tests in acceptance/update_test.rb.

See Travis builds #129 and #135; those changes had nothing to do with being able to find the delete link.

@ghost ghost assigned carols10cents Apr 22, 2012

Contributor

carols10cents commented Apr 22, 2012

Ok, there are more of these now, I've had it.

Contributor

carols10cents commented Apr 22, 2012

The new failures, and most of the old ones, appear to be tests that make external requests that would pass if you were using a previously-successful VCR cassette. That would explain why the failures are transient-- sometimes the external requests work and sometimes they don't.

I'm thinking about checking in the VCR cassettes I have in order to solve this problem. This introduces the possibility that we won't be notified through a test failure that an external site has changed, but I think the tradeoff of getting more stable test runs would be worth it.... Anyone have any thoughts before I do this?

One of the tests that we've skipped for a while does not use VCR; I'm trying a few travis runs that print out more of what the travis tests are seeing in my repo to try and figure that one out.

Contributor

carols10cents commented May 15, 2012

Adding the VCR cassettes seems to have helped some, but there are still 2 tests that fail a lot (but not every time) on travis, and never locally.

For serious, if anyone can fix this, name your reward. I will buy you coffee or beer. Or chocolate. Or pizza. Or an edible bouquet of fruit.

Which tests are still broken on this? Any?

Contributor

carols10cents commented Sep 22, 2012

There are four pending tests that are marked with "skip" (they show up as "S" when you run the tests right now).

I actually saw a NEW one that is NOT skipped currently the other day on travis here: http://travis-ci.org/#!/hotsh/rstat.us/jobs/2509375

carols10cents added a commit to carols10cents/rstat.us that referenced this issue Nov 4, 2012

Add heisenbug logging that prints to STDOUT in the cases we know of
That way, we can get a little more information when these tests fail on Travis, which appears to be the only place they ever fail. See notes in Issue #479.

geopet commented Mar 1, 2013

@carols10cents I'm trying to reproduce this bug and I'm not seeing the skips when running rake test. I'm likely doing something wrong here. Any suggestions? 🐱

Contributor

hindenbug commented Oct 27, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment