Set of measured points to known coordinates

[yakir12]

I might have missed how to do this (and am glad to add this to the documentation), but how would one get a transformation from a (large) set of locations, say on an image, and the corresponding real-world coordinates?

[c42f]

Hmm, I'm not sure I understand this question. If you want to go from two-dimensional image coordinates into three dimensional world coordinates, you'll need a depth map to accompany your image. Without a depth map, each pixel corresponds to a ray in three dimensions. Put another way, PerspectiveMap is not invertible.

[yakir12]

I'll try to explain myself a little better: If I have a two dimensional grid (x & y), and I have the dimensions of this grid (each cell's width and height), then how would I use that information to create a Map?

In my specific case, I have an image of a checkerboard, and I want to transform locations in the image (that can only exist within the plane of the checkerboard) to real-world 2D coordinates.

[yakir12]

OK, I now know what I'm looking for in Matlab at least: fitgeotrans (see here)

[c42f]

Oh, ok, I understand what you're going for now.

So you have two grids of points pos1 and pos2, and you desire a smooth interpolating transformation trans such that

trans(pos1[i,j]) == pos2[i,j]

There's nothing in the package which does this yet, though I'd say this is more of a problem for something like the Interpolations package - if the appropriate type of interpolation was defined there, we could easily wrap it up into an InterpolatedTransformation type in this package.

[yakir12]

Sorry to bring this up again, but:
So this is especially relevant for camera calibrations, where we want to find (and apply) the transformation that converts image pixel coordinates to real-world coordinates. The idea is to have a function who's arguments are a bunch of known image points and their respective real-world points, and its output is this transformation.
The problem with having this functionality in Interpolations is that while the input is a set of x and y pairs, the output of the interpolation is not one number (e.g. z) but also a set of x and y pairs. There are other interpolation mechanisms out there that can do this kind of x, y -> x, y mapping (I think Dierckx.jl), but since you already have a whole projective camera section it feels like this really fits here.
After this post on Discourse, I found out that I can simply feed the interpolation with a vector (the set) of a vector (of the x and y pair). So problem solved.
This touches a subject that has received considerable attention but not too much structure: camera calibration. I'm referring to Interpolations.jl, ImageTransformations.jl, ImageProjectiveGeometry.jl and others.
What do you think? I guess this falls on the bigger question of whether CoordinateTransformations.jl should be able infer the transforms. Currently we can only apply them and build them directly (i.e. by suppling the offsets, angles, etc).

[andyferris]

Ah, yes, CoordinateTransformations.jl is definitely designed to be one piece of the puzzle for people doing calibrations, SLAM, etc. (This was precisely the kind of problem we were thinking of when developing it).

The way I imagine it to be used: Write a cost function describing how well your images match, and use Julia's autodifferentiation packages to optimize the calibration parameters (using your own optimizer or one of Julia's higher level ones).

(Another tip: use SVector{2,Float64} from StaticArrays.jl instead of Vector{Float64} to represent your image points to get a big speed-up.)

[yakir12]

The way I imagine it to be used: Write a cost function describing how well your images match, and use Julia's autodifferentiation packages to optimize the calibration parameters (using your own optimizer or one of Julia's higher level ones).

Wah! What is that..? The only two ways I know of are:

1. building a camera model that describes the intrinsic and extrinsic parameters, like here
2. building some warping scheme, like here

What you describe sounds super cool. Any chance you could spin up a MWE for how that would work?

(about the tip: Yes! Thank you!)

[yakir12]

Just a small step towards a solution:
For the case that there are exactly 3 fixed (known) coordinates, and 3 moving (new) coordinates, in 2D, the following Discourse topic has a solution (https://discourse.julialang.org/t/coordinate-transformation-from-three-known-points). Basically this:

using CoordinateTransformations
function CoordinateTransformations.AffineMap(ps::Vector{Pair{T,R}}) where {T, R}
    X = vcat(hcat(last.(ps)...), ones(1,3))'
    Y = hcat(first.(ps)...)'
    c = (X \ Y)'
    A = c[:, 1:2]
    b = c[:, 3]
    AffineMap(A, b)
end
fixed_points = [[0,0], [0,50], [130,50]]
moving_points = [rand(2) for _ in 1:3]
f = AffineMap(Pair.(fixed_points, moving_points))
@assert fixed_points ≈ f.(moving_points)

But I feel this can be generalized to n dimensions and perhaps optimized...