Support for rendering image markers with CairoMakie

[fatteneder]

Description

Overload CairoMakie.draw_marker for Matrix{<:Colorant} types which allows to render images as markers.

Type of change

• New feature (non-breaking change which adds functionality)

Checklist

• Added an entry in NEWS.md (for new features and breaking changes)
• Added reference image tests for new plotting functions, recipes, visual options, etc.

[fatteneder]

Tests are failing because of a failure in the conversion pipeline for the markers.

I retraced it to this PR #1981 which introduced the _marker_convert method that is now erroring:

# an array of markers is converted to string by itself, which is inconvenient for the iteration logic
_marker_convert(markers::AbstractArray) = map(m -> convert_attribute(m, key"marker"(), key"scatter"()), markers)
_marker_convert(marker) = convert_attribute(marker, key"marker"(), key"scatter"())

The issue is that when scattering with image markers you pass a marker of type Matrix{<:Colorant} <: AbstractArray for which _marker_convert tries to convert all elements as markers. I think adjusting the overload here should do the trick.

[asinghvi17]

Looks good, thanks! Couple of minor changes requested.

[asinghvi17]

Oops...I guess this breaks WGLMakie. Maybe we could exclude this test from there?

https://github.com/JuliaPlots/Makie.jl/blob/216a353a6c6f0c39078eb21d8a80e954eae39ad5/WGLMakie/test/runtests.jl#L41

[fatteneder]

Can't we fix this right now? Although I would not know how 😅

[SimonDanisch]

Is this really just for PDF, or for CairoMakie? Looks to me like the latter, no?

[fatteneder]

Aeeehhh, could be 😅. I only tested with PDFs. I guess @asinghvi17's 👍 to your answer indicates that it also works for CairoMakie. I'll update the news then.

Honestly, I am don't know yet which parts of Makie are responsible for saving scenes to files? And what file formats it can handle?
So far I only used CairoMakie to generate PDFs. Does it also work with SVGs and PNGs? Perhaps I should read the docs more carefully 😆

[asinghvi17]

Basically, you can think of it this way - Makie specifies the points to render, and the backend packages actually render them.

CairoMakie works with PNG and vector types (SVG, EPS, PDF). [W]GLMakie works only with bitmap/raster types (fundamentally PNG) and conversions to jpg, bmp, tiff, etc.

For example, a complex recipe will be degraded to atomic plots (lines, scatter, heatmap/image, surface, mesh, meshscatter, volume) by Makie. The backend's basic task is to render these atomic plots in the correct positions. Of course, you can then hook into higher level recipes to render those directly if you want, as CairoMakie does with poly. GLMakie renders only atomic level plots to OpenGL, WGLMakie renders them to browser. At one point we had a GRMakie which was trying to render Makie plots with GR, but it lost momentum and the packages weren't really that compatible.

[asinghvi17]

GLMakie CI actually passes, but since this post is from an external PR I guess the bot can't comment that there is a missing refimage. Otherwise this is good to go.

[asinghvi17]

Looks like there are some bugs here - shouldn't be too difficult to fix though.

Left is reference image, right is recorded image - this is from the Image scatter different sizes example.

The scale of the bottom marker looks weird and the top marker seems (a) misaligned and (b) somehow more compressed?

More detail on the top marker weirdness

[fatteneder]

I pushed a fix for the wrong scaling, at least now the Makie logo looks ok (the problem were swapped dimensions due to the permutedims call).

However, the logo and the doggy now both appear smaller than in the ref image (as you already noted above).
I don't know how to fix that given only the size of the picture and the scale factor.
Adding an artificial factor of 3/2 to the scaling (and offset) would give roughly the right sizes.
Any idea where this difference could origin from?

[asinghvi17]

Any idea where this difference could origin from?

Could you try rendering with different px_per_units and see what happens? My guess is that somehow you are drawing in device space, not Cairo's user space. Especially since the reference images seem to be the same.

[fatteneder]

Could you try rendering with different px_per_units and see what happens?

I already tried with different scaling. E.g. adding an artificial factor of 3/2 gives roughly the same size as the ref image.

My guess is that somehow you are drawing in device space, not Cairo's user space. Especially since the reference images seem to be the same.

Perhaps it is the difference between markerspace = :data, :pixel and the test needs to be updated?
Using markerspace=:data seems to work fine in terms of sizing and positioning:

using FileIO
using CairoMakie
CairoMakie.activate!()
using Makie

img = Makie.logo()
img2 = load(Makie.assetpath("doge.png"))
images = [img, img2]
# markersize = map(img-> Vec2f(reverse(size(img) ./ 10)), images)
markersize = [ ( 1.0,0.5 ), ( 1.0,1.0 ) ]
s = scatter(1:2, [0.75,2], marker = images, markersize=markersize, axis=(limits=(0.5
            markerspace=:data)

save("scatter.pdf", s)

[fatteneder]

Could you try rendering with different px_per_units and see what happens?

I already tried with different scaling. E.g. adding an artificial factor of 3/2 gives roughly the same size as the ref image.

Sorry, did not notice that there is an actual px_per_unit/pt_per_unit attribute for save.
Changing it does not change anything in the output though.

[fatteneder]

I spent more time on testing and finally compared the png outputs generate by GLMakie and CairoMakie:

using FileIO
using Makie

function test_scatter(filename)
  img = Makie.logo()
  img2 = load(Makie.assetpath("doge.png"))
  images = [img, img2]
  markersize = map(img-> Vec2f(reverse(size(img) ./ 10)), images)
  s = scatter(1:2, 1:2, marker = images, markersize=markersize, axis=(limits=(0.5, 2.5, 0.5, 2.5),))
  save(joinpath(@__DIR__, filename), s)
end

using CairoMakie
CairoMakie.activate!()
test_scatter("cairo_scatter.png")

using GLMakie
GLMakie.activate!()
test_scatter("gl_scatter.png")

• GLMakie output:
  

• Cairo output:
  

Booth pictures are identically, but they differ from the test output of test CairoMakie:

I guess the difference is in different settings in the test, but I could not figure out what those are. Where can I find them?

But since the tests now pass I guess the sizing is not the issue anymore. Only the colored edges in pdf output remain.

[fatteneder]

I opened an issue for the colored edges problem since it also appears when saving it as pdf using an image plot.

This PR should now be good to go.

[SimonDanisch]

Thank you all! :)

[SimonDanisch]

Just noticed, that the added test never worked in CairoMakie:

Was this working at some point in this PR?

[fatteneder]

Might be that I forgot to check the CairoMakie result, my bad.
I will take a look.