paket.exe restore hangs after (successfully) finishing its job

[mbodm]

Hey,

posted this already in #3412 but it seems this has no effect, since that Issue is rather old. So i decided to open a new issue.

We have somewhat same problem here, as in #3412 issue. All stuff is done and paket.exe restore simply not returns. But there seems to be no activity.

I attached 2 screenshots here:

• Console output of our build.cmd script (a simple batch file starting paket.exe restore)
  --> Screenshot1: ConsoleOutput.png
• Sysinternals Process Monitor Screenshot of paket.exe process
  --> Screenshot2: ProcessMonitor.png

After the CloseFile() (Screenshot2) nothing happens, but paket.exe in our script not returns (Screenshot1).

Additional informations:

• It doesn´t matter if paket.exe is started from cmd.exe, PowerShell, or the new Windows Terminal.
• It doesn´t matter if paket.exe restore needs to restore or not (in Screenshot1 all packages were up2date).

Used environment:

• Output of paket.exe --version: Paket version 7.2.1+8e4eb74b42fbd45f39f7afce9184c16ebb65f16c
• Windows 11 Pro Version 22H2 Build 22621.1848
• Visual Studio 2022 Version 17.6.3 (but doesn´t matter here)

We are a big company (but using my personal GitHub account here), developing and maintaining 1 giant .NET enterprise software product and we use paket.exe since years now. And this is a non-critical (but annoying) problem we have, since quite a while. The paket.exe restore does what it should (restore packages) and all is fine. The paket.exe simply not returns. We "fixed"it, by using a self-written tool, that scans for a paket.exe process and, if found, kills the process, so our script can continue then. But that´s more of a band aid patch than a solution. 😉

Any help is very appreciated!

Best regards

[smoothdeveloper]

Could you somehow:

• build a debug version of paket.exe (with the .pdb aside)
• when it hangs, attach to the process using your debugger and do a survey of the stack traces of each thread having something going on

I don't think I've faced this (as a long time user) but I'm never using custom sources that require anything fancy but plain nuget.org, public sources, or folder access for example.

It may be relevant that you share what may be relevant information about your paket.dependencies/lock files if you have ability.

[mbodm]

Hi,

thanks a lot for the fast reply! I will try to supply what you asked for. Could take a bit of time maybe.

FYI: I assume this issue is the same effect as in this issue: #4216

paket.exe restore is just using restore as argument, but i assume it is the exact same problem. So its maybe not a restoreproblem and more of a general paket.exe problem. Maybe it’s something Win11 related. I don’t know.

[smoothdeveloper]

It would be interesting also, to see if there are subprocesses that paket would launch (such as dotnet itself).

Maybe there is some kind of deadlock over files, when dotnet commands may be spawned by paket? (I'm not sure the implementation is doing this, just trying to look for hypothesis that the hanging could also be in an external process).

Also, some commands has to scan all paket.references in your tree, you may consider trying to identify, if a part of your tree is causing the issue in particular, it may help you do a surface binary search on this hypothesis.

[mbodm]

Should i not see the spawned child procs in ProcesssMonitor or ProcessExplorer then?

[smoothdeveloper]

Yes, you definitely would see such processes.

[mbodm]

Is there some documentation about compiling/building paket.exe so i can build it on my own (to inspect what’s going on)?

• Startup project?
• Prerequisites?
• etc.

PS: I’m an experienced (~20 years) C#/.NET software dev, but i never wrote a single line of F# 😄

[mbodm]

@smoothdeveloper

Just as some form of update:
We are still in the process of analyzing stuff. I will let you know, as soon as we have some further information. But it could take a while. Rather busy at the moment. 😉

[smoothdeveloper]

@mbodm no issue and sorry I didn't see your previous message up to now either...

To build, I recommend a windows machine with VS 2019 and the 4.6.1 targeting pack, it should work relatively smoothly with just a call to the build.cmd, and then in Visual Studio.

With VS 2022, somehow I'm facing issues with .net targeting pack which are cumbersome to figure out.

Stepping into the code / attaching to the process works the same as with C#/VB.NET so just overlook the syntax in meantime, I'm confident you'll be able to get to the bottom of where the process is hanging, form which point, people with more knowledge of the paket implementation may be able to provide guidance on ways to fix the issue.

Please bring build issues (text output) in this thread if you face anything that would feel like it is blocking you.

[mbodm]

Hmm,

we are still investigating and i am still not sure what it´s all about.

At the moment i can state the following facts:

• We download version 7.2.1 of paket.Bootstrapper.exe
• We rename paket.Bootstrapper.exe to paket.exe
• We run paket.exe --version

Result:

Then we do the following things:

• We download version 7.2.1 of paket.Bootstrapper.exe
• We run paket.bootstrapper.exe -v --version

Result:

As you can see, wether the renamed paket.exe nor the un-renamed paket.bootstrapper.exe return to the terminal. Even CTRL+C or CTRL+X can break it and you have to close the terminal.

The red output is the exact german translation of the one in this Paket GitHub issue: #1271

We are using paket.exe in magic mode, like it is described here:
https://fsprojects.github.io/Paket/bootstrapper.html#Magic-mode

Maybe you have some hints for us?
Do we something wrong?

[mbodm]

We finally can reproduce it! 😃

What we do (step1):

• We download version 7.2.1 of paket.Bootstrapper.exe
• We rename paket.Bootstrapper.exe to paket.exe
• We make sure that C:\Users\USERNAME\AppData\Local\Temp does not contain a paket folder.
• If it´s there, we delete it.
• We run paket.exe --version

Result of step1: All works fine and paket.exe returns to terminal.

Now we do the following instead (step2):

• We download version 7.2.1 of paket.Bootstrapper.exe
• We rename paket.Bootstrapper.exe to paket.exe
• We make sure that C:\Users\USERNAME\AppData\Local\Temp does contain a paket folder, created by some previouspaket.exe call.
• If it´s there, we do not delete it.
• We run paket.exe --version

Result of step2: The problem occurs and paket.exe does not return to terminal.

Used environment is a Windows 11 machine (details see in the opening post).

What we can say for sure:

• It´s not VirusScan-related. We´ve tested all of this with and without any active VirusScan software. In the Paket Magic Mode documentation is a part about that. And we analyzed it. We can say for sure: This problem is definitely not VirusScan-related.
• The downloaded "real" 10 MB paket_HASH.exe (in the temp folder, downloaded by the bootstrapper) always successfully finishes its job. It´s just the bootstrapper, not returning, after that.
• We can reproduce the above as often as we want and it always behaves that way. We tested it on 8-9 different Windows 11 machines. The result was always the same.

Any help is very appreciated.

PS:
We also tried using paket.exe.config and/or paket.Bootstrapper.exe.config files with BootstrapperOutputDir option. It seems to have no effect. The paket folder is always created under the user´s temp folder. We followed this guidlines:
https://fsprojects.github.io/Paket/bootstrapper.html But maybe we missunderstand the sense of this parameter. But that doesn´t change the facts anyway.

[ChrSteinert]

I tried to reproduce it on a Windows 10 with active Defender on it.
Even creating the paket folder in the Temp directory manually did not reproduce your issue.

I would also suggest, what has been before; get a version of paket with symbols, when it hangs, get a dump or attach a debugger and get us the line of code, where the hang occurs.

[mbodm]

First: Thx for the reply!

I have no news about the issue, but i just dropped in, to say: „Try it on a Windows 11 machine.“

I have no (easy) access to a Windows 10 machine (to compare behaviours) in my company. Not sure about Windows 11 is the problem here, but i only can say the above, already said, things. And i can say that i used a 4 weeks old Windows 11 machine, fresh installed. I downloaded and used Paket.exe in bootstrapper mode, the typical way, when i did the tests. I also made sure no Virus-Scan stuff was running or involved (not even Windows Defender).

Sadly, there is no time for debugging and hunting open source tool bugs, in my company, at the moment. At the moment, we fixed the problem, simply by using it in non bootstrapper mode or not using it at all. 😿

I just can advice you to follow the steps in the July 18th posts above, on a Windows 11 machine.

[ChrSteinert]

Ok – so you expect me/the maintainers to have a Win11 ready for testing, but you, using this free product for commercial gain, cannot invest the time to attach a debugger once, to assist us volunteers helping you fix your problem for free?

For someone with your experience it should be an easy task to get us the line of code, that hangs from attaching a debugger.
I would even take a process dump, and analyze that for you (which is literally right click -> Create Dump, and upload to GitHub), if you say it’s to much to ask of your company to invest the few minutes the debugger thing would take to further a free product, that they benefit from, and seem to, a certain extend, to rely on.

[mbodm]

Hi,

hmm... ok. Sorry, if i said something wrong above. Not sure about it, when i read between your lines.

Let me add 3 statements here:

Your first question: Yes. That´s exactly what i would assume. That you are capable to test your own software. "Assume", not "expect".

Don´t get me wrong: I am an open source contributor and a friend (i can´t find a better word, since english is not my native language) of open source software since more than 2 decades. And also the Paket tool is/was very appreciated. And i/we are totally aware of the fact, that it´s a helpful addition to a .NET (Classic Framework) toolbox.

But i want to make 1 thing absolutely clear here: I/We did not expect anything from you, or from the maintainers. And i also not really get, where, in my above posts, i ever showed an "i expect this from you guys" stance.

Also please don´t missunderstand my intension, with that issue. My major intention, way before our own commercial benefits, was exactly for the reason you talk about: To give something back and let you know that issue. I had 2 other possibilities: a) Do nothing and call it a day. b) Open a halfassed 2-statements-issue. Instead i clearly inspected it (besides the debug dumb) and openend a well-documented issue here, for your (or the open source mindset´s) pleasure, not mine/our.

Foremost related to the fact, that the classic .NET Framework is more and more fading away and we anyway are 80% through the steps to use the newer SDK project style format (which makes Paket obsolete).

So you are completely free to do, with this infos or this issue, whatever you want. It´s your software and your GitHub account. You are wether related to, nor dependent on, ppls like us/me. So feel free to close the issue and/or ignore the posts above. It´s totally up to you and in my opinion i never sounded like i don´t think or expect that.

I quickly tested the above steps yesterday, on a private Windows 10 machine, of a friend of mine. The issue not occurred. So i assume more and more it´s a Windows 11 issue. The rest is up to you.

By the way: GitHub says above i closed the issue. I did not. Just for the sake of correctness here.

Anyway, have a nice day.

---

PS (directly addressed to @ChrSteinert): When i think about the open-source-mindset in general and read your post above, i am really not sure, if you should invest your time into open-source. No offense here, but talking about "we should do your job???" sounds not very open-source-compatible, to be honest.

[ChrSteinert]

GitHub also shows to me, that you closed the issue. I took this into account while writing which looks like throwing your hands up.

If you can, reopen it. If not I suggest creating a new one (or using the other one opens by you).

I am still convinced that the best way forward is the debugger-thing or dump which I am still open to assist with.

[mbodm]

Hmm, no clue why GitHub stated that. I did not closed the issue.

I try to reopen the issue now, by using the „Reopen with comment“ button below, with this comment…

Edit:
Ok, that worked. Issue is open again. But feel free to close it, on your side, if you think it fits.

[smoothdeveloper]

@mbodm, there is a bit of C# code in this repository, and I find this particular while loop a bit suspect:

Paket/src/Paket.Bootstrapper/HelperProxies/FileSystemProxy.cs

Lines 41 to 69 in 11320d7

	private static Stream WaitForFileOpen(string path, FileMode filemode, FileAccess fileAccess)
	{
	Stopwatch watch = null;
	int wait = 100;
	while (watch == null || watch.Elapsed < WaitHighLimit)
	{
	try
	{
	var readOnly = fileAccess == FileAccess.Read;
	return File.Open(path, filemode, fileAccess, readOnly ? FileShare.Read : FileShare.None);
	}
	catch (Exception exception)
	{
	if (exception.HResult != HRESULT_ERROR_SHARING_VIOLATION)
	{
	throw;
	}
	if (watch == null)
	{
	watch = Stopwatch.StartNew();
	}
	Thread.Sleep(wait);
	wait = Math.Max(wait + 10, 1000);
	}
	}
	throw new TimeoutException("Timeout while waiting for a file to be available");
	}

Can you confirm if the hang is lasting 15 minute and then it doesn't hang anymore, or it does hang for significantly longer?

Also, why do you need to use the bootstrapper? Is it for project with dotnet SDK or dotnet framework/mono only and you can't use dotnet tool restore & dotnet paket ...?