manip.Rmd

---
title: "Image Manipulations"
output: rmarkdown::html_document
resource_files:
  - vignettes/images/manip/*
opengraph:
  image: 
    src: "https://debruine.github.io/webmorphR/articles/images/manip/manip.png"
  twitter:
    card: summary
    creator: "@lisadebruine"
    site: "@webmorph_org"
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  out.width = "100%",
  fig.path = "images/manip/"
)

starttime <- Sys.time()
```

This vignette demonstrates the functions you can use to resize, rotate, crop, pad, and align images. Templates are also changed with the images.

```{r, message=FALSE}
# load packages and set maximum plot width
library(webmorphR)
library(webmorphR.stim) # for extra stimulus sets
wm_opts(plot.maxwidth = 850)
```


## Load your images

Load the demo stimuli, or read in your own images with `read_stim()`.

```{r}
stimuli <- demo_stim()
```


## Resize

Resizing your images at the start of a chain can make everything go faster if you're working with very large images, but can make the results blurry if you go too small.

```{r resize, fig.cap="Resized Images"}
stimuli |>
  resize(.1)
```

You can set the resize dimensions by proportions or pixels. Numbers less than 10 are interpreted as proportions, so the code above resized the images to 0.1 of their original size. Numbers larger than 10 are interpreted as pixels. If you set just width or just height, the other dimension is calculated proportionally to keep the aspect ratio.

```{r resize-pixel, fig.cap="Resized Images by pixel"}
stimuli |>
  resize(height = 200)
```


If you set both width and height, you may alter the aspect ratio. Make sure you really want to do this! Use `to_size()` instead if you want to resize and crop/pad all of your images to the same size without changing the image contents' aspect ratio.

```{r resize-both, fig.cap="Resized Images by width and height"}
stimuli |>
  resize(width = 200, height = 100)
```

## Rotate

Rotate images by degrees. Set the `fill` argument to fill in the corners with a specific colour, or use `patch(stimuli)` to sample the colour from 10x10 pixel square at the top left corner.

```{r rotate, fig.cap="Rotated Images and Templates"}
stimuli |>
  rotate(degrees = 45, fill = patch(stimuli))
```

If images aren't centered, you may want to rotate around the center of the image or the center of the face, and may want to keep the original image dimensions or expand to the full rotated image dimensions.

To demonstrate, we'll add some blue padding to the right side of the first demo image to make it off center. The contrasting fills used below are just to help you understand how each rotation option works; you'll usually set fill to the image background colour.

```{r}
offset <- stimuli[1] |> 
  draw_tem() |>
  pad(0, 500, 0, 0, fill = "dodgerblue") |>
  rep(4)

# set the rotations to show
degrees = seq(0, 90, by = 30)
```

By default, the rotate happens around the centre of the image. This can be undesirable if the face is off-centre.

```{r}
rotate(offset, degrees, origin = "image", fill = "pink")
```

Set `origin = "tem"` to rotate around the centre of the template.

```{r}
rotate(offset, degrees, origin = "tem", fill = "pink")
```

```{r, eval = FALSE, echo = FALSE}
# degrees to rotate
degrees <- seq(0, 360, 10)

# get an image, crop and duplicate it 
img <- demo_stim(1) |> 
  crop(0.5, 0.5) |> 
  pad(0, .5, 0, 0) |> 
  rep(length(degrees))

# rotate around centre of image
image_centered <- img |> 
  mask(fill = "dodgerblue") |>
  rotate(degrees, origin = "image", fill = "dodgerblue")|>
  pad(30, 0, 0, 0) |> 
  label('origin = "image"', size = 20)

# rotate around centre of template
tem_centered <- img |> 
  mask(fill = "hotpink") |>
  rotate(degrees, origin = "tem", fill = "hotpink")|>
  pad(30, 0, 0, 0) |> 
  label('origin = "tem"', size = 20)

# rotate around left eye point
lefteye_centered <- img |>
  mask(fill = "orange") |>
  rotate(degrees, origin = 0, fill = "orange")|>
  pad(30, 0, 0, 0) |> 
  label('origin = 0 (left eye)', size = 20)

# rotate around right eye point
righteye_centered <- img |> 
  mask(fill = "purple") |>
  rotate(degrees, origin = 1, fill = "purple")|>
  pad(30, 0, 0, 0) |> 
  label('origin = 1 (right eye)', size = 20)

# combine and animate
c(image_centered, tem_centered, 
  lefteye_centered, righteye_centered) |> 
  animate(fps = 20)
```

If you set `keep_size = FALSE`, the image and template will be rotated and keep the maximum rotated size (so it doesn't matter what you set `origin` to).

```{r}
rotate(offset, degrees, keep_size = FALSE, fill = "pink")
```


## Horizontal Eyes

If you want to rotate each image by the amount that makes the eyes horizontal, use `horiz_eyes()`.

```{r horiz-eyes, fig.cap = "Make eyes horizontal with horiz_eyes()"}
unstandard <- demo_unstandard(1:5)
horiz <- horiz_eyes(unstandard, fill = patch(unstandard))

c(unstandard, horiz) |> plot(nrow = 2)
```



## Crop

Crop images to a new width and height. You can specify sizes in proportions or pixels. Values < 2 will be interpreted as proportions. 

```{r crop, fig.cap="Cropped images with default center cropping."}
crop(stimuli, width = .6, height = .8)
```

The x- and y-offsets default to NULL, which centres the cropping. You can set the offsets to a proportion or pixel value and the cropping will start there. The origin is the upper left corner, so setting `x_off = 0` starts the cropping from the left side, and setting `y_off = 0` starts the cropping from the top.

```{r crop-offset, fig.cap="Cropped images with offsets."}
crop(stimuli, 
     width = 250, height = 300, 
    x_off = 0, y_off = 0)
```

## Pad 

Pad images with a border. Values less than 1 are interpreted as proportions. Set the colour with `fill`, otherwise its value defaults to the value of `wm_opts("fill")`, which is white unless you reset it.

```{r pad, fig.cap="Padded images with the same value for each side."}
pad(stimuli, 0.05, fill = "black")
```

Pad the top, right, bottom and left borders with different values, and set a different colour for each fill.

```{r pad-multi, fig.cap="Padded images with different values for each side."}
pad(stimuli,
    top = 10, right = 20, bottom = 30, left = 40,
    fill = c("dodgerblue", "hotpink"))
```

Or you can use the `patch` function to get the median colour from a patch of the image. The `patch()` function defaults to the top left 10 pixel square of each image.

```{r crop-patch, fig.cap="Cropped images with background matched to top left 10-pixel square"}
rb <- load_stim_rainbow(1:8)

rb |>
  pad(50, fill = patch(rb)) |>
  plot(nrow = 2)
```

Or you can set the boundaries of the patch manually. The code below sets the patch colour to the average colour from to 10% of width and height in the centre of each image.

```{r crop-match, fig.cap="Cropped Images with background matched to centre."}
patch_fills <- patch(rb, 
                     width = .1, 
                     height = .1, 
                     x_off = .45, 
                     y_off = .45)

rb |>
  pad(50, fill = patch_fills) |>
  plot(nrow = 2)
```

## Crop to Template

You can use the `crop_tem()` function to get the minimum and maximum x- and y-values from all of the image templates, then use that to set the cropping. In the example below, the images are cropped so there is 100 pixels of padding around the boundaries, calculated across all images. 

```{r bounds}
load_stim_lisa() |>
  crop_tem(100) |>
  draw_tem()
```

Set `each` to TRUE to calculate the boundaries separately for each template in the list instead of the full set.

```{r bounds-each}
load_stim_lisa() |>
  crop_tem(20, each = TRUE) |>
  draw_tem()
```

## To Size

The function`to_size()` combines crop and resize to get a batch of images to the same size.

```{r}
multisize <- demo_unstandard(1:5) |> pad(10, fill = "hotpink")
```


A 10-pixel pink border has been added to these original images so you can see what has been cropped in the subsequent images.

```{r fig-to-size, echo = FALSE, fig.cap="The images created above, with their width and height."}

lab <- paste0(width(multisize), "x", height(multisize))

multisize |> 
  pad(60, 0, 0, 0) |>
  label(lab, size = 40)
```

The code below resizes each image to fit into a 300 x 400 pixel image and fills in any extra background with blue. You'll probably choose your background colour for a real stimulus set, but this makes the demo clearer.

```{r fig-fit-pad, fig.cap="By default, to_size() will change relative sizes and cover the background without cropping any original image content."}
to_size(multisize, 300, 400, 
        fill = "dodgerblue")
```

Set `keep_rels = TRUE` to bring the largest image to fit in the new size, resize the others proportionally, and pad.

```{r fig-rel-pad, fig.cap="Set keep_rels = TRUE to keep relative size differences between original images."}
to_size(multisize, 300, 400, 
        fill = "dodgerblue", 
        keep_rels = TRUE)
```

Set `crop = TRUE` to resize all images until they cover the new size and crop the "overhang". There is no need to set the fill color for padding when crop is TRUE.

```{r fig-fit-crop, fig.cap="Set crop = TRUE to cover the new image size and crop off overhang."}
to_size(multisize, 400, 400, 
        crop = TRUE)
```

Set `keep_rels = TRUE` to bring the smallest image to cover the new size, resize the others proportionally, and crop any overhang. 

```{r fig-rel-crop}
to_size(multisize, 400, 400, 
        crop = TRUE, 
        keep_rels = TRUE)
```


## Align

You can align a set of images on two points. By default, these are the first two points in the template, which correspond to the left and right eye in the standard FRL template, the 106-point Face++ templates, and the dlib `auto_delin()` templates. For chaotic historical reasons, the 83-point Face++ template has the eye points as 63 and 81 (pupils) or 57 and 50 (eye centre).

If you don't set the x and y coordinates, the images will align to the average value across the set. The value of `fill` will fill in any borders from the rotation and cropping so the images stay the same size as the originals.

```{r align, fig.cap="Two-point alignment with align()"}
unstandard_men <- demo_unstandard(6:10)

unstandard_men |>
  align(fill = "dodgerblue")
```

You can manually set the values for the x and y coordinates, as well as the resulting width and height of the images. These values must be in pixels.

```{r align-manual, fig.cap="Two-point alignment with align(); setting values manually"}
unstandard_men |>
  align(x1 = 350, y1 = 350, x2 = 450, y2 = 450,
        width = 800, height = 800, fill = "dodgerblue")
```

Instead of aligning to the average of all the images, you may want to align to a reference image. Set `ref_img` to the index or name of the chosen image. 

```{r realign2, fig.cap = "Align images to a reference image."}
aligned5 <- align(unstandard_men, ref_img = 5, 
                  fill = "dodgerblue") 

plot_rows(
  "Original images" = unstandard_men,
  "Aligned to image 5" = aligned5,
  top_label = TRUE
)
```

You can align images to a single point by setting pt1 and pt2 the same. For example, the tip of the nose is point 55 in this template, so the following code moves the tip of the nose of all images to the average position, and alters the image dimensions to the average, but doesn't resize or rotate any of the faces

```{r align-1pt, fig.cap = "One-point alignment."}
align(unstandard_men, pt1 = 55, pt2 = 55, 
      fill = "dodgerblue")
```


## Procrustes Align 

You can set the `procrustes` argument to TRUE to use [Procrustes analysis](https://en.wikipedia.org/wiki/Procrustes_analysis) to align the images and templates to fit the average shape (or a reference image) as closely as possible.

```{r procrustes, fig.cap = "Procrustes alignment."}
align(unstandard_men, procrustes = TRUE, fill = "dodgerblue")
```

Currently, [webmorph](https://webmorph.org) does not have this capability, but you can download your images and tems, procrustes align them with the R package, save the images and templates, and upload them back to webmorph.



## Chaining

You can also chain image manipulation commands. 

```{r chain}
stimuli |>
  rotate(c(45, -45)) |>
  crop(.5, .5) |>
  pad(0.1, fill = "dodgerblue")
```

## Repeating images

You can use `rep()` to repeat images in a stimuli. Here, we repeat the faces 3 times each, apply 6 different rotations with different background colours, crop them to the same size, and plot them with 6 different template point colours.

```{r rep}
rainbow <- c(pink = "#983E82",
             orange = "#E2A458",
             yellow = "#F5DC70",
             green = "#59935B",
             blue = "#467AAC",
             purple = "#61589C")

stimuli |>
  resize(0.5) |>
  rep(each = 3) |>
  rotate(seq(10, 60, 10), fill = rainbow) |>
  crop(250, 250, fill = rainbow) |>
  draw_tem(pt.color = rainbow, pt.alpha = 1, pt.size = 3) |>
  plot(nrow = 2)
```


## Image functions

You can do so many things with the {magick} package that is installed with {webmorph}. The function `image_func()` is just a convenient wrapper that allows you to apply {magick} functions (or any other function that takes a magick image as the first argument) to each image in a stimulus list.

```{r fun}

stimuli <- demo_stim("m_")
imglist <- c(
  image_func(stimuli, "blur", radius = 10, sigma = 5),
  image_func(stimuli, "charcoal", radius = 5, sigma = 2),
  image_func(stimuli, "oilpaint", radius = 5),
  image_func(stimuli, "implode", factor = 0.25),
  image_func(stimuli, "implode", factor = -0.25),
  image_func(stimuli, "negate"),
  # canny edge detection format is {radius}x{sigma}+{lower%}+{upper%}
  image_func(stimuli, "canny", geometry = "0x1+10%+30%"),
  image_func(stimuli, "edge", radius = 2),
  image_func(stimuli, "modulate", brightness = 80),
  image_func(stimuli, "modulate", brightness = 120),
  image_func(stimuli, "modulate", saturation = 0),
  image_func(stimuli, "modulate", saturation = 110)
)

labs <- c("Blur", "Charcoal", "Oilpaint", "Implode",
          "Explode", "Negate", "Canny", "Edge", 
          "20% Darker", "20% Brighter", "Greyscale", "Saturated")

imglist |>
  pad(70, 0, 0, 0, fill = "dodgerblue3") |>
  label(labs, color = "white") |>
  plot(nrow = 3)

```

The possible {magick} functions are listed by `image_func_types()`. Check the magick package help for more info on each (e.g., `?magick::image_oilpaint`.

```{r}
image_func_types()
```


A potentially useful one is "fill" if your backgrounds are clean enough (the `mask()` functions are better for this, though). You need to experiment with the fuzz a bit. These functions are vectorised, so you can set a different point to start the flood fill or a different fuzz for each image.

```{r image-fill}
demo_stim(1) |>
  rep(5)  |>
  image_func("fill", "white", point="+10+10", 
             fuzz = seq(5, 25, 5)) |> 
  label(text = paste("fuzz =", seq(5, 25, 5)))
```



----------------------

```{r, echo = FALSE}
elapsed <- (Sys.time() - starttime) |>
  as.numeric(units="mins") |>
  round(1)
```

This script took `r elapsed` minutes to render all the included images from scratch.