Test fails in TFS Gated Checkin #308

Open
theoutlander opened this Issue Dec 20, 2012 · 7 comments

Comments

Projects
None yet
2 participants
Contributor

theoutlander commented Dec 20, 2012

I've spent the last two days on this issue and I'm not sure what is going on. My test case is failing during the TFS gated checkin.

I've validated that:

  1. The test passes locally on my machine via VS 2012 (on FF, Chrome & IE 10 Pre-reelase)
  2. It passes on the build machine via VS 2012 (on IE 9....we can't install other browsers)
  3. I ran it through the gated check-in and put the test method into a while loop (with the Mock HttpServer running) and then fired up IE 9 and went to the HTML Page that normally the test method would launch .... and it test PASSED.

However, for some reason during the gated check-in, the test case fails with an error that "The agent process was stopped while the test was running".

On the server, I see the following message in the EventViewer (the problem seems to occur when it flushes the data to the stream...is it trying to respond back to the browser?)

(QTAgent32.exe, PID 6516, Thread 57) Exception data being reported in Watson: Exception: IOException
Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
Stack Trace:
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Flush()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Flush()
at ScriptSharp.Testing.WebServer.HttpMessage.WriteStatus(HttpStatusCode statusCode)
at ScriptSharp.Testing.WebServer.HttpMessage.Run()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
Inner Exception Details
Exception: SocketException
Message: An existing connection was forcibly closed by the remote host
Stack Trace:
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)

Any idea why this is happening? Thanks!

Owner

nikhilk commented Dec 20, 2012

This looks like the same issue as #267.
There were changes made to the client script (in QUnitExt.js) and in the test web server, but looks like something might still be lingering.

Contributor

theoutlander commented Dec 20, 2012

So, I just figured the problem! The build machine had IE 9. What I noticed there was that sometimes the test wouldn't execute in the browser. If I refreshed the browser it would execute and then the test harness would exit with a Pass (if it was within timeout).

Questions

  1. I swapped IE 10 with IE 9 locally and was able to repro the problem! What info would you need to be able to fix this?
  2. Also, I meant to ask you about the purpose of change : 5455bf8#src/ZipX/ProjectTemplates/UnitTest/QUnitExt.js under the thread #267

What is the intent of this code? You're opening a blank window and closing it??

 xhr.onreadystatechange = function() {
    if (xhr.readyState == 4) {
      window.open('', '_self', '');
      window.close();
    }
  };

I noticed that the sample test cases weren't closing the window, but the ones I wrote were. The same code exists in QUnitExt.js in both places. Why would one close and not the other?

  1. In addition, when I started IE again it would state that my last session closed unexpectedly or something on those lines. Why does it think it crashed? I don't see anything in the debugger.
Owner

nikhilk commented Dec 20, 2012

The odd window.open is to try increase the chances of success for the subsequent window.close (as it still doesn't work 100% of times). For example discussion around this subject: http://stackoverflow.com/questions/57854/how-can-i-close-a-browser-window-without-receiving-the-do-you-want-to-close-thi

As far as I can tell this error suggests the client terminated the connection... earlier the script used to issue a fire-and-forget request, and I was hopeful that introducing a callback to track progress would ensure the browser didn't terminate the connection, and thereby letting the server send its response without running into this error.

In terms of finding workarounds to deal with this...

  • I wonder if the window operations were deferred using a setTimeout, and if that fixes any timing related issues that let the current event processing loop in the browser to take care of what is queued before the window is closed.
  • Worse, wrap the stuff on the server to swallow exceptions.

The code in WebTest.cs tries to forcibly close the browser in case it didn't close ... first by sending a close message to the window, and then by killing the process. Perhaps that is the cause of IE complaining the next time.

Contributor

theoutlander commented Dec 20, 2012

That makes sense. Thanks for the explanation.

I installed IE 10 on the build machine, but the test still fails during the gated check-in. I can run the same test via the command line (MSTest) on the build machine and it passes consistently (earlier it was sporadic with IE 9).

This issue happens only during a TFS Gated Checkin. I wonder if it has anything to do with running under the Network Service account (without a desktop). Is anyone using S# QUnit tests via TFS Gated Checkin?

Owner

nikhilk commented Dec 20, 2012

It might very well be a case of environment ... no desktop, different security context, and potentially other things as a consequence of how TFS might be setting up things. I haven't heard from anyone trying this before, so likely this is a totally new scenario here.

Have you been able to run other test infrastructures (eg. WatiN, or other commercial ones that launch browsers) under the context of similar TFS setup?

Could you try those couple of workarounds above to see if they help, even if its a remote possibility?

Contributor

theoutlander commented Dec 21, 2012

I'll try WatiN when I get a chance .... need to catch up with other stuff since my two hour UT task turned into a 3 day escapade.

For now, I've created a third configuration where the S# projects get built. That way the TFS Gated Checkin will ignore S# tests (this was the easiest way to bypass it without admin rights). In the near future, I'll work with the admin to play with settings (like running the gated checkin under the System account and/or as a logged in user).

Contributor

theoutlander commented Dec 21, 2012

Just realized what you meant the workaround from a previous thread. I'll certainly try those. I was in fact thinking about swallowing exceptions on the server, but didn't get to making that change as that would only mean that my checkin would work even if the test case failed.

One thing I noticed was that the test HTML page loaded showing the QUnit elements, but the tests weren't executing (sometimes even when I tried manually in IE 9). That could also be related to an onload event or timing issue....not sure how and when QUnit starts to process everything. I'll have to dig into that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment