WcfService: Investigate how to force-quit badly behaved services on CI server

[iamjasonp]

We've run into a couple of instances now where the CI server box has failed the prerequisite/cleanup steps because either git.exe was running for a long time and not proceeding, or w3wp.exe was holding onto files that was preventing the cleanup/sync necessary.

This has necessitated manual intervention in these cases to fix and stabilize the services.

We should find a way to force-quit and recover services/processes like this on the CI server to help keep our CI server system stable without the need for human intervention.

[zhenlan]

@iamjasonp could you please share more details of what are the errors you see and what is the root cause of the issues?

[zhenlan]

@iamjasonp do you still see this or get information of my earlier question?

[iamjasonp]

We root caused this as a permissions issue; Git expects that it has full permissions to the filesystem at all times, but there were some strange ACLing issues that were residual (I'm guessing #1090 will help to resolve this too).

I have seen w3wp have issues too recently, but have not found the root cause yet. The issue I found was that the app_code folder get locked and git chokes when trying to touch it. It's not apparent to me which file git is having trouble with though or if it's the folder itself