Converting single pixels to numeric values.

[orswan]

TLDR; Is there a straightforward way to convert a single pixel to a numeric value? If not, could one please be implemented (and mentioned in the quickstart)?

Explanation:
This works: channelview(rand(Gray,1))[1]
But this does not: channelview(rand(Gray))
This works: convert.(Float64,rand(Gray))
But this does not: convert.(Float64,rand(RGB))

As far as I can tell, there is no uniform way to get the numeric value of a single pixel other than wrapping it in an array, applying channelview, and then subscripting. This seems weirdly discombobulated.

More generally, Images.jl seems to try really hard to conceal the underlying numeric data in an image. I find this extremely irritating. I am a physicist, and a lot of my work consists in taking images of e.g. cold atomic gasses. Invariably, I don't really want to look at these images though, I just want to process them with my own custom numerical methods. (Concrete example: FFTW does not work on images with color values.) I would appreciate if the package made it easier to just "get in and out", importing an image and converting it to numerical values. If such methods exist, I would love for them to be mentioned in the quickstart. I have my own methods like imageToFloatArray(img::Matrix) = convert.(Float64,Gray.(img)) that I frequently add to the beginning of my notebooks, but it would be better if there were such functions conspicuously built in to the package.

[mkitti]

It is unclear exactly what you want when you say you want to convert a RGB pixel to Float64. A RGB{Float64} consists of three Float64 values, not a single value. You also used broadcast notation suggesting that a RGB{Float64} is iterable, but this is not the case.

The red, green, and blue components can be accessed as follows via the underlying package ColorTypes.jl. The documentation can be found in the Color Channels section of the Julia images function reference.

julia> using ColorTypes

julia> @which RGB
ColorTypes

julia> p = rand(RGB)
RGB{Float64}(0.6123750590186987,0.560934020985612,0.6942792994166388)

julia> red(p)
0.6123750590186987

julia> green(p)
0.560934020985612

julia> blue(p)
0.6942792994166388

julia> comp1(p), comp2(p), comp3(p)
(0.6123750590186987, 0.560934020985612, 0.6942792994166388)

If you wanted a single value, perhaps the grayscale conversion of the RGB value, you might do the following.

julia> gp = Gray(p) # convert(Gray, p)
0.5915162570283196

julia> gray(gp)
0.5915162570283196

julia> comp1(gp)
0.5915162570283196

The underlying implementation of RGB{Float64} is basically

struct RGBFloat64
    r::Float64
    g::Float64
    b::Float64
end

You can learn that from introspection.

julia> fieldnames(typeof(p))
(:r, :g, :b)

julia> fieldtypes(typeof(p))
(Float64, Float64, Float64)

Note that we have RGB types that have different layouts, but use a similar interface.

julia> p24 = RGB24(p)
RGB24(0.612N0f8,0.561N0f8,0.694N0f8)

julia> red(p24)
0.612N0f8

julia> green(p24)
0.561N0f8

julia> blue(p24)
0.694N0f8

julia> fieldnames(typeof(p24))
(:color,)

julia> fieldtypes(typeof(p24))
(UInt32,)

In either case, perhaps you would like to reduce such a pixel to a NTuple{3, Float64},

julia> p .|> (red,green,blue)
(0.6123750590186987, 0.560934020985612, 0.6942792994166388)

julia> p24 .|> (red,green,blue)
(0.612N0f8, 0.561N0f8, 0.694N0f8)

julia> p24 .|> (red,green,blue) .|> Float64
(0.611764705882353, 0.5607843137254902, 0.6941176470588235)

For an image, I can generalize this using the broadcast notation. Note that this eagerly extracts the values and allocates new memory for the result.

julia> img = rand(RGB, 2, 2)
2×2 Array{RGB{Float64},2} with eltype RGB{Float64}:
 RGB{Float64}(0.491911,0.880987,0.430793)  …  RGB{Float64}(0.633592,0.168878,0.61858)
 RGB{Float64}(0.962367,0.754885,0.873799)     RGB{Float64}(0.839588,0.606721,0.685134)

julia> red.(img)
2×2 Matrix{Float64}:
 0.491911  0.633592
 0.962367  0.839588

julia> blue.(img)
2×2 Matrix{Float64}:
 0.430793  0.61858
 0.873799  0.685134

julia> green.(img)
2×2 Matrix{Float64}:
 0.880987  0.168878
 0.754885  0.606721

It may also be worth discussing the underlying Julia array facilities, in particular the Base.reinterpret function. This essentially allows you to do virtual casting on the underlying dynamic memory. Note that this is lazy rather than eager. We have not allocated new memory.

julia> reinterpret(NTuple{3,Float64}, img)
2×2 reinterpret(Tuple{Float64, Float64, Float64}, ::Array{RGB{Float64},2}):
 (0.491911, 0.880987, 0.430793)  (0.633592, 0.168878, 0.61858)
 (0.962367, 0.754885, 0.873799)  (0.839588, 0.606721, 0.685134)

julia> reinterpret(reshape, Float64, img)
3×2×2 reinterpret(reshape, Float64, ::Array{RGB{Float64},2}) with eltype Float64:
[:, :, 1] =
 0.491911  0.962367
 0.880987  0.754885
 0.430793  0.873799

[:, :, 2] =
 0.633592  0.839588
 0.168878  0.606721
 0.61858   0.685134

julia> reinterpret(Float64, Gray.(img))
2×2 reinterpret(Float64, ::Array{Gray{Float64},2}):
 0.713331  0.359093
 0.830478  0.685288

The last output is essentially the same as what you get when you use channelview. Base.reinterpret is thus a generalization of channelview, although channelview is more aware of colors.

To convert the lazy views to an actual memory allocation, you can use Base.collect or Base.copy.

julia> reinterpret(reshape, Float64, img) |> collect
3×2×2 Array{Float64, 3}:
[:, :, 1] =
 0.491911  0.962367
 0.880987  0.754885
 0.430793  0.873799

[:, :, 2] =
 0.633592  0.839588
 0.168878  0.606721
 0.61858   0.685134

julia> reinterpret(reshape, Float64, img) |> copy
3×2×2 Array{Float64, 3}:
[:, :, 1] =
 0.491911  0.962367
 0.880987  0.754885
 0.430793  0.873799

[:, :, 2] =
 0.633592  0.839588
 0.168878  0.606721
 0.61858   0.685134

Regarding your imageToFloatArray you may want to consider a lazy version as follows.

julia> using MappedArrays, BenchmarkTools

julia> imageToFloatArray_lazy(img::Array{<:Colorant}) = mappedarray(p->gray(Gray(p)), img)
imageToFloatArray_lazy (generic function with 1 method)

julia> bigimg = rand(RGB, 1024, 1024);

julia> @btime imageToFloatArray($bigimg);
  4.669 ms (2 allocations: 8.00 MiB)

julia> @btime imageToFloatArray_lazy($bigimg);
  2.577 ns (0 allocations: 0 bytes)

[orswan]

Thanks for the detailed response--it's very helpful.

It is unclear exactly what you want when you say you want to convert a RGB pixel to Float64.

What I "want" ideally is for channelview to have a method for single pixels, akin to how it operates on arrays of pixels. Perhaps I should rephrase my initial question as "Why are arrays of pixels handled in a different way than individual pixels?" I want an easy way to see the underlying numerical data, regardless of the type and regardless of whether it's an array or single pixel.

Some context: I use Images.jl infrequently. When I do, my workflow is usually something like this:

1. Import several images to a Jupyter notebook. Display a few of them to check that they make sense (e.g. oriented the correct way).
2. Try to convert a few individual pixels to numerical values to check the overall scale of the data.
3. Spend 5 minutes searching how to convert my particular pixels to numerical values, because I forgot from last time.
4. Convert some of the images to numerical data for further processing. I keep the images around because displaying them in a notebook is faster than plotting e.g. a heatmap of the numerical data.

Step 3 is of course the bother. You could argue that the real problem is my workflow (a fair critique). And to be sure, if I used Images.jl every day, I would have the relevant functions memorized, but that's not the case. I always have to spend a few minutes googling to figure out how to make the relevant conversions (e.g. it's Gray, not Grey).

So basically, as an occasional user of this package, I find the distinctions between pixels and images unintuitive, and you would make my life a bit easier if they were handled more similarly.

Of course, the very act of writing this has probably solidified all the relevant issues in my mind, so that I won't need to be reminded for a long time. But I am trying to advocate for my fellow occasional users, and make a suggestion that I think would make the package easier to start using.

[mkitti]

One can do the following today.

julia> p = rand(RGB)
RGB{Float64}(0.8854174465282774,0.9495272204330198,0.6825188143513837)

julia> channelview([p])
3×1 reinterpret(reshape, Float64, ::Array{RGB{Float64},1}) with eltype Float64:
 0.8854174465282774
 0.9495272204330198
 0.6825188143513837

I suppose we could do the following.

julia> ImageCore.channelview(p::Color) = channelview([p])

julia> channelview(p)
3×1 reinterpret(reshape, Float64, ::Array{RGB{Float64},1}) with eltype Float64:
 0.8854174465282774
 0.9495272204330198
 0.6825188143513837

However, the following still seems more consistent to me.

julia> comp1(p)
0.8854174465282774

julia> comp1.(rand(RGB,2,2))
2×2 Matrix{Float64}:
 0.98307   0.626048
 0.725759  0.383848