Crashing against current Master - Broken Pipe

[nritholtz]

Webkit Debug output of crashing tests when ran against lastest commit, 7bedbc0, in master. As you can tell, the tests are running against a remote server.

All system information is included in the Webkit Debug output.

[jferris]

It seems likely that we're trying to free an already-deleted reference.

@nritholtz does this reliably crash for you on master but reliably work on a previous commit? If so, is there any way you could pinpoint the commit? The commit you linked to simply updated a NEWS file, and it seems unlikely for that to be the culprit.

Narrowing it down to a more specific test case would also be very valuable.

[nritholtz]

I'll try to run it against previous versions, since it is failing consistently on a specific test.

Unfortunately, it will take a little while since this is a bump upgrade from 1.3.0.

I will let you know.

[nritholtz]

@jferris f8f2048 is the offending commit
Works fine against previous commit, e49176b

Also, I'm working with proprietary code, otherwise I would've created this issue initially w a branch for a test scenario. Sorry about that.

[jferris]

Thanks. The commit that's causing issues is fortunately pretty small, so we may be able to debug with a few rounds.

I remember an earlier issue where the logger was causing issues because the URL string was somehow deleted. Based on that, I tried disabling just the logger change from that commit and pushed to a branch called jf-reset-crash.

Could you give that branch a shot?

[nritholtz]

@jferris Just tried it against your new commit but unfortunately it crashed again.

[jferris]

@nritholtz hmm, okay. I pushed another commit with some debugging statements in the reset statement. Can you give that a shot? That will hopefully tell us where the crash is occurring.

[nritholtz]

@jferris Tell me if you want the entire debug log, but here is what I think you want as the relevant log:

Received "Reset()"
Started "Reset()"
Beginning reset
Pending replies: 1
Aborting pending replies
    Portal User: auto1431657538
      end of file reached (EOFError)
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/bundler/gems/capybara-webkit-9240ce857f8c/lib/capybara/webkit/connection.rb:55:in `read_nonblock'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/bundler/gems/capybara-webkit-9240ce857f8c/lib/capybara/webkit/connection.rb:55:in `read'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/bundler/gems/capybara-webkit-9240ce857f8c/lib/capybara/webkit/connection.rb:46:in `gets'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/bundler/gems/capybara-webkit-9240ce857f8c/lib/capybara/webkit/browser.rb:287:in `check'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/bundler/gems/capybara-webkit-9240ce857f8c/lib/capybara/webkit/browser.rb:210:in `command'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/bundler/gems/capybara-webkit-9240ce857f8c/lib/capybara/webkit/browser.rb:38:in `reset!'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/bundler/gems/capybara-webkit-9240ce857f8c/lib/capybara/webkit/driver.rb:254:in `reset!'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/gems/capybara-2.4.1/lib/capybara/session.rb:103:in `reset!'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/gems/capybara-2.4.1/lib/capybara.rb:257:in `block in reset_sessions!'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/gems/capybara-2.4.1/lib/capybara.rb:257:in `each'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/gems/capybara-2.4.1/lib/capybara.rb:257:in `reset_sessions!'
      /home/qa/.rvm/gems/ruby-2.1.0@automated_testing/gems/capybara-2.4.1/lib/capybara/cucumber.rb:8:in `After'

[jferris]

@nritholtz thanks. That confirms my suspicion. I'll see if I can replicate this locally and push up a fix later this week.

[mhoran]

@jferris, perhaps it would make sense to listen to the destroyed() signal from the reply? Maybe some of these replies are getting aborted elsewhere, or not finishing for some reason -- and we have a stale reference?

[jferris]

Perhaps it would make sense to listen to the destroyed() signal from the reply?

That sounds like a good idea. It's definitely the abort call that's crashing it, so it must be that the replies are getting deleted without emitting finished.

[jwg2s]

Is the recommended short-term solution here to downgrade to 1.5.0? Getting this on 1.5.1

[jferris]

@jwg2s that will probably work for you. The only change in 1.5.1 was a fix for a crash in 1.5.0, but it unfortunately introduced a new crash. Hopefully you won't hit the other crash on 1.5.0. I'll try to get a fix for this out sometime this week.

[jwg2s]

Downgrading worked for us, will look for the next update and then upgrade again. Thanks for the help!

[jferris]

@nritholtz could you try the latest commit on that branch? I think I've managed to reproduce the crash.

[vkatz-ninthdecimal]

@jferris, we have had sporadic broken pipe issues in our tests for months now. i just tried your recent commit, and it looks like it may have fixed them!

[nritholtz]

Tried it a few times and it looks good.

[vkatz-ninthdecimal]

Hrm, actually i was wrong. we are still getting broken pipe issues

[vkatz-ninthdecimal]

@jferris, should i open a new issue or add stuff here?

[jferris]

@vkatz-ninthdecimal if you could open a new issue, that would be great.

[nritholtz]

Hey @jferris,

Any update on getting this into a release?

[jferris]

I was running into some issues on CI, and I haven't had a chance to address it. My next likely opportunity to work on this is Friday.

[jferris]

Fixed by #781. Will be released in 1.5.2.