GaussianBlur and ColorJitter ops.

[barucden]

This PR introduces first two non-geometric operations, Gaussian blur and contrast & brightness adjustment. It partially solves #16.

At first, I thought I could implement it easily but I encountered some issues. I would greatly appreciate help resolving them (cc @johnnychen94).

1. The introduced operations change the image eltype (e.g., ColorTypes.RGBA{FixedPointNumbers.N0f8} => ColorTypes.RGBA{Float64}). Is this something we need to worry about? This concern will perhaps apply for all non-geometric operations in future.
2. What is a good way to test these operations? It is difficult to come up with a test-case whose expected output is not based on the formulas present in the code (e.g., for Rotate90 we know the expected output without evaluating the operation, which is not the case for GaussianBlur).
3. For a segmentation task, these operations should be applied only on the input image, not the ground truth segmentation. How can we deal with that? Should we introduce a new abstract subtype for this purpose?

I have two additional issues regarding the AdjustContrastBrightness:

1. The name seems too long. What would be a shorter alternative?
2. The formula is now out[i] = clamp(α * img[i] + β * M), where M=mean(img). However, sometimes it is better to use M=the maximum value (e.g., for a grayscale image whose pixels are represented as 8-bit unsigned integer, M=255). I could not find a generic way how to get this value for different color types (yet, I am not too experienced). Is there a way?

[johnnychen94]

The introduced operations change the image eltype (e.g., ColorTypes.RGBA{FixedPointNumbers.N0f8} => ColorTypes.RGBA{Float64}). Is this something we need to worry about?

This is a separate issue. Most algorithms in JuliaImages don't take this assumption seriously so I think we can live this type of promotion without worrying about it too much. Generally, this issue can be fixed in upstream packages (e.g., ColorVectorSpace, FixedPointNumbers).

What is a good way to test these operations?

I would just use something as simple as ref = imfilter(img, gaussian((σ, σ), (k, k))) to be the test reference, but wrap it with assess_psnr(from ImageQualityIndexes) to allow some internal changes. This test looks trivial but it provides some guarantee that internal applyeager is correctly called.

For a segmentation task, these operations should be applied only on the input image, not the ground truth segmentation.

This is also a separate issue, maybe you can open a new issue with a minimal working example to discuss it? augment supports two usages:

augment(input_img, pipeline)
augment((input_img, ground_truth), pipeline) # augmentation applies uniformly to both images

I assume that you can just use the first version.

---

I have two additional issues regarding the AdjustContrastBrightness:

I forgot to mention this, maybe we can directly use https://github.com/JuliaImages/ImageContrastAdjustment.jl for the eager version, it introduces more contrast adjustment methods, including LinearStretching? Then open an issue in that repo for a lazy view. How do you think of this?

I would like to see more functionality of Augmentor being implemented in other packages, and leave Augmentor a pipeline composer.

[barucden]

I added tests for GaussianBlur but I use the standard pixel-wise comparison to check applyeager. I was trying to use assess_psnr but it seems that it lacks an implementation for RGBA images?

I like your idea about using ImageContrastAdjustment.jl. I am just not clear about the next step. Should I remove the proposed operation AdjustContrastBrightness for now? Or, as it seems that ImageContrastAdjustment.jl does not provide an operation equivalent to AdjustContrastBrightness, should I replace it with a different operation (such as LinearStretching) that comes from ImageContrastAdjustment.jl?

[johnnychen94]

I was trying to use assess_psnr but it seems that it lacks an implementation for RGBA images?

Could possibly just use RGB.(testpattern()) as image input.

It's not very clear to me how should we process the alpha channel of RGBA. Are there any references/conventions for this?

However, sometimes it is better to use M=the maximum value (e.g., for a grayscale image whose pixels are represented as 8-bit unsigned integer, M=255). I could not find a generic way how to get this value for different color types (yet, I am not too experienced). Is there a way?

gamutmin/gamutmax from ColorTypes, and is also available in ImageCore (via Reexport).

I like your idea about using ImageContrastAdjustment.jl. I am just not clear about the next step. Should I remove the proposed operation AdjustContrastBrightness for now? Or, as it seems that ImageContrastAdjustment.jl does not provide an operation equivalent to AdjustContrastBrightness, should I replace it with a different operation (such as LinearStretching) that comes from ImageContrastAdjustment.jl?

Ah never mind. I think this is for different purposes. Although clamp(α * img[i] + β * M) is mathematically a linear stretching operation, it seems a more convenient way for augmentation purposes; while the ImageContrastAdjustment LinearStretching output is more human-readable (e.g., stretch to [0, 1] range).

---

The implementation looks good to me, let me know when it is ready.

[johnnychen94]

One thing I forgot to mention, can you also add two cards to docs/operations? You can use other cards as templates and insert the filename to the corresponding config.json.

[barucden]

new changes:

• changed the name from AdjustContrastBrightness to ColorJitter (torchvision uses this name for a similar method)
• added the feature based on gamutmax
• added cards to docs/

Regarding the PSNR computed for RGBA images, I am not sure. I tried searching, but did not find any resources. Maybe the alpha channel could be treated just like any other channel?

I think that we can settle with converting to RGB. What is a reasonable PSNR threshold for which two images are considered equal?

[johnnychen94]

Maybe the alpha channel could be treated just like any other channel?

For JuliaImages that ranges consistently from [0,1] this is reasonable. But for other library that uses [0, 255] range, it doesn't make much sense to me. Anyway, we doesn't need to worry about this in this PR.

What is a reasonable PSNR threshold for which two images are considered equal?

larger than 25-30 should be okay. I use 25 in ReferenceTests.

[barucden]

Okay, I think it is ready now. Thanks for your advices @johnnychen94! Let me know if I should change something.

[johnnychen94]

Only some small suggestions and questions. (It's midnight here, I'll come back tomorrow and approve tests if there are new commits; Github action by default disable the test running for non-contributor because months ago there are people sending spam PRs to other repositories to run mining script)

[johnnychen94]

Really nice work! The detailed descriptions in this PR and other issues are really appreciated.