Magick.net CreateProcess interacts with other created processes in application

[tomasr78]

This issue is follow-up from #1024 but I think it should be opened as an issue.

When I try to convert Image to PDF sometimes the file is not released after conversion and I get an error that the file is used.

I have tried to narrow down the problems and that's what I found. It seems the problem is not related directly to Magick.net but to Process.Start. I suppose you also use Process.Start to launch ImageMagick.

So the problem is that Magick.Net and my code use Process.Start at the same time to start other console apps and somehow they interact and lock each other files.

This problem only happens when Process.StartInfo.UseShellExecute is set to False.

    using (var p = new Process
    {
        StartInfo =
        {
            FileName = "c:\commandLine.exe",
            Arguments = "c:\file.jpg",
            UseShellExecute = false,
            RedirectStandardError = true,
            RedirectStandardOutput = true
        }
    })
    {
        try
        {
            p.Start();
            p.PriorityClass = ProcessPriorityClass.Normal;

            p.WaitForExit();

            var output = await p.StandardOutput.ReadToEndAsync();
            var error = await p.StandardError.ReadToEndAsync();
            var exitCode = p.ExitCode;
            return .....; 

        }
        finally
        {
            try
            {
                if (!p.HasExited)
                    p.Kill();
            }
            catch (InvalidOperationException)
            {
                //The process is not found, probably executable was not fount                    
            }

            p.Close();
        }
    } 

Dirk, could you show me the code where you run ImageMagick? I think there is a bug because if we remove Magick.Net our Processes start working fine and files are not locked anymore.

[dlemstra]

The Magick.NET API executes ImageMagick through it's C API. This library is not using any shell execution. It communicates directly with the library of ImageMagick.

When two process write to the same file at the same time there will be locking issues that you cannot prevent?

[tomasr78]

This locking problem happens with different files, I will try to explain:

My application runs Magick.NET(Image to PDF) and commandLine.exe(used ShellExecute code above, convert PDF to Image) and passes two different files for Magick.NET - Image.JPG->PDF, App2 - PDF->Image)

Sometimes, the App2 - PDF->Image finish processing the file, but for an unknown reason, the file produced with App2 is locked by the App2. This lock only happens when Magick.NET runs at the same time. I have tried to run other command-line apps and they also lock files when Magick.NET runs.

The summary - the Magick.NET(or ImageMagick) for unknown reason has side effects for command-line apps launched by Process.Start with the property Process.StartInfo.UseShellExecute set to False. The apps executed by Process.Start lock their files and do not release them until Magick.NET execution is finished on another thread.

All code is run from Console Application .NET 4.7.

[dlemstra]

I am sorry but I really don't understand what you are trying to tell me? It all looks like there is an issue with App2 not working properly.

[tomasr78]

There is no issue with App2 or App3 until Magick.NET is run on the separated thread, when Magick.NET is executed in any thread it has side effects on any console app which is run using Process.Start in separated threads and work with files, they get locked sometimes.
There is somehow related with in-process handles, I don't know.

I fully understand that it sounds very confusing and I will try to debug further.

Could you please send me the GitHub URL to the code where ImageMagick is being executed?

[tomasr78]

I could create a console application demo to demonstrate the issue, it is easy to reproduce on any Windows OS.

[dlemstra]

If you can create a console demo that demonstrates this issue that would help a lot.

And this library never executes ImageMagick on the command line. It uses this library to execute the ImageMagick code: https://github.com/dlemstra/Magick.Native.

[tomasr78]

@dlemstra I have created a demo project at https://github.com/tomasr78/FileLockingDemo
I am using two additional command-line utilities MuTool(https://mupdf.com/) and CPDF(https://community.coherentpdf.com/), which work with pdf files to simulate the problem.

When you run the project, you will find that files are being locked.

Fixes that solves the problem:

• Remove ImageMagick Task and run again. Everything would work fine. I am using ImageMagick, can't remove it :)
• Process.StartInfo.UseShellExecute to True instead of False. I need to have Process.StartInfo.UseShellExecute to set to False to be able to read console output.

[dlemstra]

Some weird is happening. When I checked the exception I got a message that one of the two executables (cpf or mutool) was locking the file.

I check to see what was happening with the procmon executable and I got the following output:

As you can see the executable closes the other file on exit. And this is also the only interaction with that file. But I have no idea what that is happening though. This almost feels like a bug in .NET. I did find a workaround for you:

using (var fs = File.Create(outTest1Pdf))
{
    image.Write(fs, MagickFormat.Pdf);
}

Because we don't create a file in the native code of Magick.NET it doesn't get closed by that other process.

[dlemstra]

It looks like file handles are inherited, see: dotnet/runtime#13943. Also found a possible work around: https://stackoverflow.com/a/62063412. Will need to investigate if we can/should use this.

[dlemstra]

Using fopen with the N flag seems to have resolved the issue. This issue will be resolved in the next release.

[tomasr78]

Great, thank you! I can't believe that such issues are in the framework and even not fixed in .NET 5.

[dlemstra]

Can this issue be closed @tomasr78?