Seamless Blending and Saliency Blending

[fmw42]

Imagemagick 7.1.0-20 introduces two new masked compose methods: 1) seamless-blend and 2) saliency-blend.

Note, that both methods work best when HDRI is enabled. Saliency-blending will be a bit slower than seamless blending.

These compose methods are implementations of Poisson Image Editing by Perez et. al.. Imagemagick uses an iterative finite difference implementation.¹

The seamless-blend method simply tries to blend the overlay image smoothly over the background image. It is essentially a smoothly blended insert.

The saliency-blend method merges the background and overlay images such that the image that is the most salient at each pixel is preferentially put on top. That is it keeps the texture from whichever of the overlay or background image that has the most detail at each pixel.

The syntax is the same as any masked composite, namely,

magick background overlay mask -compose <method> -composite result

Both methods require an appropriate mask. Arguments are set using either -set option:args or as follows using -define:

-define compose:args=Max-IterationsxDistortion+<Print-Iterations> -compose seamless-blend -composite

and

-define compose:args=Max-IterationsxDistortion+<Print-Iterations> -compose saliency-blend -composite

where Max-Iterations is the maximum number of iterations to compute before stopping, Distortion is the RMSE convergence stopping value between iterations, and the optional <Print-Iterations> is the number of iterations between printout to the terminal.

The first example does a seamless-blend of the bear image into the water image at an offset using the mask that outlines outside the bear. Add -verbose to the command to see the iterations.

magick -verbose water.jpg bear.jpg bear_mask.png \
-define compose:args=400x0.00002+100 -compose seamless-blend \
-geometry +30+30 -composite bear_in_water.png

water.jpg JPEG 339x379 339x379+0+0 8-bit sRGB 27435B 0.020u 0:00.007
bear.jpg JPEG 297x163 297x163+0+0 8-bit sRGB 11636B 0.000u 0:00.001
bear_mask.png PNG 297x163 297x163+0+0 8-bit Gray 2c 607B 0.000u 0:00.001
seamless blending:
  0: 0.0035994
  100: 7.26026e-05
  200: 4.15708e-05
  300: 2.97342e-05
water.jpg=>bear_in_water.png JPEG 339x379 339x379+0+0 8-bit sRGB 27435B 0.240u 0:00.078

One thing to note is that the bear image gets brightened by the seamless blend, because the background is lighter. To mitigate this, one can use a second mask image that outlines inside the bear. Since this is implemented using a read-mask, the polarity must be inverted. That is the inside of the mask is black and the outside is white.

magick water.jpg \( bear.jpg -read-mask bear_mask2.png \) bear_mask.png \
-define compose:args=400x0.00002+100 -compose seamless-blend \
-geometry +30+30 -composite bear_in_water2.png

The next example does a saliency-blend of some text scribble with an image of a wall. Since the text is on a constant background (little background detail) the textured wall will show in the background region of the text. This is similar to a regular -compose over where the background region of the overlay image is transparent. However, with saliency-blending, we do not have to explicitly make the background transparent. Because the background of the scribble image is constant color, the mask simply can be the full image, i.e. white with a 1 pixel black border (or an outline).

magick wall.jpg scribble.jpg scribble_mask.png \
-define compose:args=400x0.00002+100 -compose saliency-blend \
-composite scribble_on_wall.jpg

Likewise, we can add a rainbow into an image of an island with palm trees using saliency blending. Since the sky is a relatively constant color in the rainbow image, we can again use either an outline mask or the full image mask.

magick palms.jpg rainbow.jpg rainbow_mask.png  \
-define compose:args=400x0.00002+100 -compose saliency-blend \
-geometry +60+0 -composite rainbow_over_palms.jpg

The final application creates seamless tiling of the follow image.

For comparison to seamless tiling, we first use the image to make a simple 3x3 tiled image by appending.

magick chalk.png \
-duplicate 2 +append \
-duplicate 2 -append \
chalk_tiled_3x3.png

Note that in the above result, the tiled image clearly shows the seams between the tiles.

Next, we make the image into a seamless tile. However, first we have to use the image to create a background image with the constraints that the opposite sides are identical. This is done by averaging the opposite 1-pixel wide outer sides and replacing opposing sides with these averages.

magick chalk.png +write mpr:img \
\( mpr:img -gravity east -crop 1x+0+0 +repage \) \
\( mpr:img -gravity west -crop 1x+0+0 +repage \) \
\( mpr:img -gravity north -crop x1+0+0 +repage \) \
\( mpr:img -gravity south -crop x1+0+0 +repage \) \
\( \
-clone 1,2 -evaluate-sequence mean +write mpr:LR +delete \
-clone 3,4 -evaluate-sequence mean +write mpr:TB +delete \
\) \
-delete 0-4 \
mpr:LR \( mpr:img -shave 1x0 \) mpr:LR +append \
-shave x1 mpr:TB +swap mpr:TB -append \
chalk_background.png

This background image is now used with the original image as the overlay in a seamless-blend composite followed by a 3x3 append. The mask is the full image with a one-pixel wide black outline.

magick chalk_background.png chalk.png chalk_mask.png \
-define compose:args=400x0.00002+100 -compose seamless-blend \
-composite \
-duplicate 2 +append \
-duplicate 2 -append \
chalk_seamless_tiled_3x3.png

Hard seams in this 3x3 tiling are hardly seen.

Footnotes

1. Thanks to Imagemagick users snibgo and fmw42 for the finite difference implementation concept, prototyping and testing. ↩