Question: get percentage of progress in dzsave

[martinweihrauch]

Hi!
Thx so much for netVips!!
I would like to use dzsave. Is it possible to pass a C# Action to get the percentage of the process?
I would like to show the progress on the console or in a Windows desktop program.

Thanks

Martin

[kleisauke]

Hello @martinweihrauch,

You can just set the VIPS_PROGRESS environment variable for simple text feedback. I see this when setting VIPS_PROGRESS:

Unfortunately there is no option yet to get progress feedback in C#. I'll have to expand NetVips a bit for that.

TODO List:

• Create P/Invoke layer of the g_signal_connect and vips_image_set_progress functions.
• Marshal the VipsProgress and GTimer struct.
• Test the preeval, eval, and posteval signals. See here for an example.
• Create some convenience functions in Image.cs that wraps the above functions / structs. We could use the IProgress<T> interface for this.

[martinweihrauch]

Thx for the answer! I haven't worked with environment variables yet in C#. I am currently creating a WPF desktop application.
Could you please give me a hint: If I call NetVips dzsave with my WPF application, how can I set an environment variable and "catch" the text?

[kleisauke]

I've just added support for attaching progress feedback. For example:

// Load from a huge JPEG file
var image = Image.NewFromFile("huge.jpg", access: Enums.Access.Sequential);

var progress = new Progress<int>(value =>
{
    Console.Write($"\r{value}% complete");
});

// Attach progress feedback
image.SetProgress(progress);

// Save image pyramid
image.Dzsave("image-pyramid");

Showcase:

If you want to test this, you can use the nightly version of NetVips. Add the https://ci.appveyor.com/nuget/net-vips feed in the <packageSources> section of your NuGet.config:

<packageSources>
  <add key="netvips-nightly" value="https://ci.appveyor.com/nuget/net-vips" />
</packageSources>

And update NetVips to 1.1.0.

Thank for reporting this!

[martinweihrauch]

Thanks a lot.
I implemented everything, but experienced an error, which is about the assistant for managed debugging. Do you have any clue what that could be?

Assistent für verwaltetes Debuggen "CallbackOnCollectedDelegate"
Für den von der Garbage Collection gesammelten Delegaten vom Typ "NetVips!NetVips.Internal.GCallback::Invoke" wurde ein Rückruf durchgeführt. Dies kann Anwendungsabstürze, Datenbeschädigung und -verlust zur Folge haben. Beim Übergeben von Delegaten an nicht verwalteten Code müssen die Delegaten von der verwalteten Anwendung beibehalten werden, bis sichergestellt ist, dass sie nie aufgerufen werden.

Ausnahme ausgelöst: "System.NullReferenceException" in NetVips.dll
Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.

Das Programm "[15844] PathoZoomConverter.exe" wurde mit Code -1 (0xffffffff) beendet.

[martinweihrauch]

My code:
var image = Image.NewFromFile(wsiFilename, access: Enums.Access.Sequential);

            // Save image pyramid
            image.Dzsave(outFilename); 

[martinweihrauch]

And there is a "System.NullReferenceException" in NetVips.dll after some percent of running dzsave...

[kleisauke]

@martinweihrauch I had forgotten to protect the delegate against premature garbage collection. It's fixed with commit a5ce84a.

This code example now runs without any problems:

var im = Image.NewFromFile(Filename, access: Enums.Access.Sequential);

var progress = new Progress<int>(value =>
{
    Console.Write($"\r{value}% complete");
});
im.SetProgress(progress);

// Force garbage collection cycle to prove
// that the delegate doesn't get disposed
GC.Collect();
GC.WaitForPendingFinalizers();

// Save image pyramid
im.Dzsave("images/image-pyramid");

Thanks for reporting this dumbness!

[martinweihrauch]

:)))
This is not a dumbness, but small mishap in an otherwise great project :))
Do you ever plan to also accept a "cancel token"? But I believe, Vips doesnt support cancelling, correct?

[kleisauke]

If you want, I can adjust the method to this:

Example

/// <summary>
/// Attach progress feedback, if required.
/// </summary>
/// <remarks>
/// You can use this function to update user-interfaces with 
/// progress feedback, for example:
/// <code language="lang-csharp">
/// var image = Image.NewFromFile("huge.jpg", access: Enums.Access.Sequential);
/// 
/// var progress = new Progress&lt;int&gt;(value =>
/// {
///     Console.Write($"\r{value}% complete");
/// });
/// image.SetProgress(progress);
/// 
/// image.Dzsave("image-pyramid");
/// </code>
/// </remarks>
/// <param name="progress">A provider for progress updates.</param>
/// <param name="token">Cancellation token to stop progress updates.</param>
public void SetProgress(IProgress<int> progress, CancellationToken token = default)
{
    VipsImage.SetProgress(this, progress == null ? 0 : 1);

    var lastPercent = 0;

    GCallback evalCallback = (imagePtr, progressPtr, userDataPtr) =>
    {
        // Stop reporting the progress if a cancellation
        // has been requested for this token
        if (token.IsCancellationRequested)
        {
            return;
        }

        var progressStruct = progressPtr.Dereference<VipsProgress.Struct>();
        if (progressStruct.Percent != lastPercent)
        {
            progress?.Report(progressStruct.Percent);
            lastPercent = progressStruct.Percent;
        }
    };

    // prevent it from being re-located or disposed of by the garbage collector
    _progress = GCHandle.Alloc(evalCallback);

    this.Connect(Internal.Enums.VipsEvaluation.Eval, evalCallback);
}

For example, this allows you to stop the progress reporting after 5 sec:

var cts = new CancellationTokenSource();
cts.CancelAfter(5000);

var progress = new Progress<int>(value =>
{
    Console.Write($"\r{value}% complete");
});
im.SetProgress(progress, cts.Token);

Unfortunately, this will only stop the progress updates and not the image processing itself. libvips does support cancellation see:
https://github.com/libvips/libvips/blob/8e55732bcdcc9525a7ffbbde127298fb6a3523b8/libvips/iofuncs/image.c#L1639-L1659

But it's internal API.

@jcupitt Is it possible to move the vips_image_set_kill function to the public API? Or is this function for internal use only? It seems that this method is only used in the deprecated vips7compat.c. Making this method public can help users to interrupt long-running image processing tasks (such as vips_dzsave).

[jcupitt]

Sure, let's make it public.

I'll just check exactly how nip2 does cancel.

[kleisauke]

I just checked how nip2 does this exactly. It seems that if the callback in im_add_eval_callback returns -1 it will kill the image (with vips_image_set_kill), see:
https://github.com/libvips/nip2/blob/1b1b27f6b61096f0230b9d7db2d2d361fa4758ed/src/progress.c#L169
https://github.com/libvips/nip2/blob/1b1b27f6b61096f0230b9d7db2d2d361fa4758ed/src/imageinfo.c#L672-L674
https://github.com/libvips/nip2/blob/1b1b27f6b61096f0230b9d7db2d2d361fa4758ed/src/imageinfo.c#L728-L730
https://github.com/libvips/libvips/blob/4af242b59935a6f330c788d6dc0f4e90e1c1ad0f/libvips/deprecated/vips7compat.c#L409

I can create a PR to move the vips_image_set_kill function to the public API, if you like.

[jcupitt]

Ah yes, I think I remember now.

It's OK, I'll do it, it's a very simple change.

[kleisauke]

I've just released NetVips 1.1.0-rc1 and NetVips.Native 8.8.0-rc1 (which contains the pre-compiled libvips 8.8.0-rc1 binaries for Linux, macOS and Windows) on NuGet. This release adds support for progress feedback (image.SetProgress()) and signal handling (image.SignalConnect()).

Support for associated images is also included. You can attach all associated images as metadata by setting the second parameter of Openslideload, for example:

var image = Image.Openslideload("image.svs", attachAssociated: true)

^{(see relevant difference here)}

Or by doing:

var image = Image.NewFromFile("image.svs", kwargs: new VOption
{
    {"attach_associated", true}
});
image.Dzsave("image-pyramid", container: "szi");

Note that you need the the libvips -all variant on Windows (instead than the -web) to include OpenSlide support. The pre-compiled libraries within the new NetVips.Native package don't support OpenSlide images.

If you would like to see what's new in libvips 8.8, please visit the the release notes of libvips:
https://libvips.github.io/libvips/2019/04/22/What's-new-in-8.8.html

[kleisauke]

NetVips v1.1.0 is now available.