Paket restore fails randomly when "msbuilding" solution in parallel #858

Closed
dzendras opened this Issue Jun 3, 2015 · 16 comments

Comments

Projects
None yet
4 participants
@dzendras

dzendras commented Jun 3, 2015

There seems to be a problem with paket restore when using msbuild/p:BuildInParallel=true switch.
It happens on our build machine randomly, but frequently (every day):

C:\SolutionDir.paket\paket.targets(36,5): error MSB3073: The command ""C:\SolutionDir.paket\paket.exe" restore --references-files "C:\SolutionDir\FooProject\paket.references" " exited with code 1. [C:\SolutionDir\FooProject\FooProject.csproj]

When using paket restore with --log-file (BTW why isn't it available on the paket's commands list displayed when you run paket without any arguments?) I could see the cause of the failures:

packages folder is locked by paket.exe (PID = 4220). Waiting...

Paket failed with:
The process cannot access the file 'C:\SolutionDir\packages\paket.locked' because it is being used by another process.

The workaround is to run paket restore before running msbuild.
Is it possible to make paket support such a scenario out of the box?

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Jun 3, 2015

Member

We tried to support this scenario, but there is no "before solution build"
event in MSBuild. So everything hooks into project build events.
The only way out was to try to serialize the requests and therefore all
paket.exe are waiting for the first to finish.

It seems the timeout is too short.
On Jun 3, 2015 11:52 AM, "Jedrzej Grabowski" notifications@github.com
wrote:

There seems to be a problem with paket restore when using
msbuild/p:BuildInParallel=true switch.
It happens on our build machine randomly, but frequently (every day):

C:\SolutionDir.paket\paket.targets(36,5): error MSB3073: The command
""C:\SolutionDir.paket\paket.exe" restore --references-files
"C:\SolutionDir\FooProject\paket.references" " exited with code 1.
[C:\SolutionDir\FooProject\FooProject.csproj]

When using paket restore with --log-file (BTW isn't it available on the
paket's commands list displayed when you run paket without any arguments?)
I could see the cause of the failures:

packages folder is locked by paket.exe (PID = 4220). Waiting...

Paket failed with: The process cannot access the file
'C:\SolutionDir\packages\paket.locked' because it is being used by another
process.

The workaround is to run paket restore before running msbuild.
Is it possible to make paket supports such a scenario out of the box?


Reply to this email directly or view it on GitHub
#858.

Member

forki commented Jun 3, 2015

We tried to support this scenario, but there is no "before solution build"
event in MSBuild. So everything hooks into project build events.
The only way out was to try to serialize the requests and therefore all
paket.exe are waiting for the first to finish.

It seems the timeout is too short.
On Jun 3, 2015 11:52 AM, "Jedrzej Grabowski" notifications@github.com
wrote:

There seems to be a problem with paket restore when using
msbuild/p:BuildInParallel=true switch.
It happens on our build machine randomly, but frequently (every day):

C:\SolutionDir.paket\paket.targets(36,5): error MSB3073: The command
""C:\SolutionDir.paket\paket.exe" restore --references-files
"C:\SolutionDir\FooProject\paket.references" " exited with code 1.
[C:\SolutionDir\FooProject\FooProject.csproj]

When using paket restore with --log-file (BTW isn't it available on the
paket's commands list displayed when you run paket without any arguments?)
I could see the cause of the failures:

packages folder is locked by paket.exe (PID = 4220). Waiting...

Paket failed with: The process cannot access the file
'C:\SolutionDir\packages\paket.locked' because it is being used by another
process.

The workaround is to run paket restore before running msbuild.
Is it possible to make paket supports such a scenario out of the box?


Reply to this email directly or view it on GitHub
#858.

@Stift

This comment has been minimized.

Show comment
Hide comment
@Stift

Stift Jun 3, 2015

Contributor

I just recommend to do a paket restore before you start msbuild and start msbuild with /p:RestorePackages=false. This works fine and fast.

Contributor

Stift commented Jun 3, 2015

I just recommend to do a paket restore before you start msbuild and start msbuild with /p:RestorePackages=false. This works fine and fast.

@dzendras

This comment has been minimized.

Show comment
Hide comment
@dzendras

dzendras Jun 3, 2015

@forki: I was pretty sure that the extra complexity was not worth it. Just asked to confirm :)
@Stift: The solution seems reasonable.

dzendras commented Jun 3, 2015

@forki: I was pretty sure that the extra complexity was not worth it. Just asked to confirm :)
@Stift: The solution seems reasonable.

@blumu

This comment has been minimized.

Show comment
Hide comment
@blumu

blumu Mar 10, 2016

Just hitting the same issue. Just out of curiosity: is it really necessary for paket to put a read access lock on this file when it just needs to restore packages?

blumu commented Mar 10, 2016

Just hitting the same issue. Just out of curiosity: is it really necessary for paket to put a read access lock on this file when it just needs to restore packages?

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Mar 10, 2016

Member

@blumu the lock is needed since we change the packages folder. If we don't serialize the requests then you get write conflicts there.

Member

forki commented Mar 10, 2016

@blumu the lock is needed since we change the packages folder. If we don't serialize the requests then you get write conflicts there.

@blumu

This comment has been minimized.

Show comment
Hide comment
@blumu

blumu Mar 10, 2016

@forki Is it not an overkill to have a global lock? Would not it be more appropriate to only lock any package directory that needs to be updated? That would significantly reduce the chance of lock contention and make this issue go away completely when the package folder is up to date.

blumu commented Mar 10, 2016

@forki Is it not an overkill to have a global lock? Would not it be more appropriate to only lock any package directory that needs to be updated? That would significantly reduce the chance of lock contention and make this issue go away completely when the package folder is up to date.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Mar 10, 2016

Member

yes in that case it would be enough to lock per nuget package.

issue go away completely when the package\ folder is up to date

that should already be no issue. since lock time would be really short and no timeout occurs.

Member

forki commented Mar 10, 2016

yes in that case it would be enough to lock per nuget package.

issue go away completely when the package\ folder is up to date

that should already be no issue. since lock time would be really short and no timeout occurs.

@blumu

This comment has been minimized.

Show comment
Hide comment
@blumu

blumu Mar 10, 2016

Large nuget packages can take some time to download so that would not completely eliminate contentions. It would still reduce them significantly though.

blumu commented Mar 10, 2016

Large nuget packages can take some time to download so that would not completely eliminate contentions. It would still reduce them significantly though.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Mar 10, 2016

Member

TBH I don't really think it's worth it. If you restore before build then everything should be fine.

Member

forki commented Mar 10, 2016

TBH I don't really think it's worth it. If you restore before build then everything should be fine.

@blumu

This comment has been minimized.

Show comment
Hide comment
@blumu

blumu Mar 10, 2016

Maybe. I would argue that restoring packages before building is just a workaround, not an actual fix. Paket.exe should ideally be just as bullet proof as nuget.exe.

blumu commented Mar 10, 2016

Maybe. I would argue that restoring packages before building is just a workaround, not an actual fix. Paket.exe should ideally be just as bullet proof as nuget.exe.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Mar 10, 2016

Member

How is nuget bullet proof here? What are they doing to solve this?
On Mar 10, 2016 19:41, "William Blum" notifications@github.com wrote:

Maybe. I would argue that restoring packages before building is just a
workaround, not an actual fix. Paket.exe should ideally be just as bullet
proof as nuget.exe.


Reply to this email directly or view it on GitHub
#858 (comment).

Member

forki commented Mar 10, 2016

How is nuget bullet proof here? What are they doing to solve this?
On Mar 10, 2016 19:41, "William Blum" notifications@github.com wrote:

Maybe. I would argue that restoring packages before building is just a
workaround, not an actual fix. Paket.exe should ideally be just as bullet
proof as nuget.exe.


Reply to this email directly or view it on GitHub
#858 (comment).

@blumu

This comment has been minimized.

Show comment
Hide comment
@blumu

blumu Mar 11, 2016

Ok maybe I did not choose the right terminology :-) I hope I did not offend anyone here. So nuget may not be "bullet proof" but maybe slightly more robust? I'm not sure how they implement it in nuget.exe but I never had to worry about this until I tried paket. I have many large nuget dependencies in my project and nuget manages to restore packages automatically at build time without a glitch. It would be preferable if paket could handle package restore just as well. Also the current workaround is not obvious and not clearly documented and I'm guessing many new paket users hit the same issue.

Auxiliary question: how would you use the workaround when building under Visual Studio? Do I need to configure the additional switch for every single project in my solution?

blumu commented Mar 11, 2016

Ok maybe I did not choose the right terminology :-) I hope I did not offend anyone here. So nuget may not be "bullet proof" but maybe slightly more robust? I'm not sure how they implement it in nuget.exe but I never had to worry about this until I tried paket. I have many large nuget dependencies in my project and nuget manages to restore packages automatically at build time without a glitch. It would be preferable if paket could handle package restore just as well. Also the current workaround is not obvious and not clearly documented and I'm guessing many new paket users hit the same issue.

Auxiliary question: how would you use the workaround when building under Visual Studio? Do I need to configure the additional switch for every single project in my solution?

@forki forki reopened this Mar 11, 2016

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Mar 11, 2016

Member

I looked at the code that acquires the lock. It waits for 25min. Is that what you see? Restore failing after 25min? So I guess something else is wrong.

I might have an idea where this race condition comes from.

Member

forki commented Mar 11, 2016

I looked at the code that acquires the lock. It waits for 25min. Is that what you see? Restore failing after 25min? So I guess something else is wrong.

I might have an idea where this race condition comes from.

@blumu

This comment has been minimized.

Show comment
Hide comment
@blumu

blumu Mar 11, 2016

No, my build normally takes less than 5 minutes overall (with msbuild running on 10 parallel threads).

blumu commented Mar 11, 2016

No, my build normally takes less than 5 minutes overall (with msbuild running on 10 parallel threads).

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Mar 11, 2016

Member

Ok please retry with 2.52.2 - I changed it slightly and also gave a little more verbose output.

Member

forki commented Mar 11, 2016

Ok please retry with 2.52.2 - I changed it slightly and also gave a little more verbose output.

@blumu

This comment has been minimized.

Show comment
Hide comment
@blumu

blumu Mar 11, 2016

Just tried 2.52.2. Paket does not fail and I don't get the msbuild warning MSB3073 anymore, however I still get a bunch of other warnings from paket itself:

error   Could not acquire lock to c:\src\packages\paket.locked.
  The process cannot access the file 'c:\src\packages\paket.locked' because it is being used by    another process.
  Could not acquire lock to c:\src\packages\paket.locked.
  Trials left: 77.

The solution is still suboptimal though, there really should not be a global lock around the package directory, especially when packages are all already up to date.

blumu commented Mar 11, 2016

Just tried 2.52.2. Paket does not fail and I don't get the msbuild warning MSB3073 anymore, however I still get a bunch of other warnings from paket itself:

error   Could not acquire lock to c:\src\packages\paket.locked.
  The process cannot access the file 'c:\src\packages\paket.locked' because it is being used by    another process.
  Could not acquire lock to c:\src\packages\paket.locked.
  Trials left: 77.

The solution is still suboptimal though, there really should not be a global lock around the package directory, especially when packages are all already up to date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment