Subspaces

How should transforms be applied to subsets of axes?

Brain storming (direct mapping)

There are two ways to "directly" map inputs to outputs: by-index or by-axis. By index is the more general, but by axis can be more concise (and easier to understand?) if / when it maps to your mental model well.

Assuming the coordinate systems:

{ "name" : "ijk", "axes" : [ {"name" : "i"}, {"name" : "j"}, {"name" : "k"} ]},
{ "name" : "xyz", "axes" : [ {"name" : "x"}, {"name" : "y"}, {"name" : "z"} ]}
{ "name" : "xyz2", "axes" : [ {"name" : "x"}, {"name" : "y"}, {"name" : "z"} ]}
{ "name" : "xyz_perm", "axes" : [ {"name" : "y"}, {"name" : "z"}, {"name" : "x"} ]}
{ "name" : "ctxyz", "axes" : [ {"name" : "c"},{"name" : "c"}, {"name" : "x"}, {"name" : "y"}, {"name" : "z"} ]}
{ "name" : "zyxtc", "axes" : [ {"name" : "z"},{"name" : "y"}, {"name" : "x"}, {"name" : "t"}, {"name" : "c"} ]}

A

Want to describe the function

x = i
y = j
z = k

Set the value of the ith coordinate of of the output to the value of the ith coordinate of the input.

{ "type" : "identity", "input" : "ijk", "output" : "xyz" }

or

{ "type" : "map_by_index", "input" : "ijk", "output" : "xyz" }

where both of the above implicitly form the lists

{ 
  "inputAxes" :  ["i", "j", "k" ],
  "outputAxes" : ["x", "y", "z" ]
}

B

We want to describe the function

x = x
y = y
z = z

from the xyz coordinate system to the xyz2 coordinate system

{ "type" : "identity", "input" : "xyz", "output" : "xyz2" }

or

{ "type" : "map_by_index", "input" : "xyz", "output" : "xyz2" }

or

{ "type" : "map_by_axis", "input" : "xyz", "output" : "xyz2" }

where map_by_axis means:

x = // an array containing the input point coordinates
axes2coords = { inputAxes[i] : inputAxes[i] for i in range(len(x)) }
return [ axes2coords[axis] for axis in outputAxes ]

C

We want to describe the function

x = y
y = z
z = x

from the xyz coordinate system to the xyz2 coordinate system

Set the value of the ith coordinate of of the output to the value of the ith coordinate of the input.

{ "type" : "identity", "input" : "xyz", "output" : "xyz2", "inputAxes" : ["x", "y", "z"], "outputAxes" : ["y", "z", "x"] }

or

{ "type" : "map_by_index", "input" : "xyz", "output" : "xyz2", "inputAxes" : ["x", "y", "z"], "outputAxes" : ["y", "z", "x"] }

or

{ "type" : "permutation", "input" : "xyz", "output" : "xyz2", "indexes" : [2, 0, 1] }

where permutation means:

for i in range(len(indexes))
    output[indexes[i]] = input[i]

or

{ "type" : "inverse_permutation", "input" : "xyz", "output" : "xyz2", "indexes" : [1, 2, 0] }

where inverse_permutation means

for i in range(len(indexes))
    output[i] = input[indexes[i]]

D

We want to describe the function

x = x
y = y
z = z

from the xyz coordinate system to the xyz_perm coordinate system

Set the value of the ith coordinate of of the output to the value of the ith coordinate of the input.

{ "type" : "identity", "input" : "xyz", "output" : "xyz_perm", "inputAxes" : ["x", "y", "z"], "outputAxes" : ["x", "y", "z"] }

or

{ "type" : "map_by_index", "input" : "xyz", "output" : "xyz_perm", "inputAxes" : ["x", "y", "z"], "outputAxes" : ["x", "y", "z"] }

or

{ "type" : "map_by_axis", "input" : "xyz", "output" : "xyz_perm" }

or

{ "type" : "permutation", "input" : "xyz", "output" : "xyz_perm", "indexes" : [2, 0, 1] }

Brain storming (subsets)

Given the above, how

A

Want to describe the following function

x = f(i,j)
y = f(i,j)
z = k

from the ijk coordinate system to the xyz coordinate system.

where f is some complicated function from R2-> R2

{
    "type" : "sequence",
    "transformations" : [
        { "type" : "f", "inputAxes" : ["i", "j"], "outputAxes" : ["x", "y"]}
        { "type" : "map_by_index", "inputAxes" : ["k"], "outputAxes" : ["z"]}
    ]
}

or do we prefer a different "type"?

{
    "type" : "by_dimension",
    "transformations" : [
        { "type" : "f", "inputAxes" : ["i", "j"], "outputAxes" : ["x", "y"]}
        { "type" : "identity", "inputAxes" : ["k"], "outputAxes" : ["z"]}
    ]
}

B

Want to describe the following function

x_2 = f(x,y)
y_2 = f(x,y)
z_2 = z

from the xyz coordinate system to the xyz2 coordinate system.

{
    "type" : "sequence",
    "transformations" : [
        { "type" : "f", "inputAxes" : ["x", "y"], "outputAxes" : ["x", "y"]},
        { "type" : "map_by_index", "inputAxes" : ["z"], "outputAxes" : ["z"]}
    ]
}

is this shorthand reasonable?

{
    "type" : "sequence",
    "transformations" : [
        { "type" : "f", "inputAxes" : ["x", "y"], "outputAxes" : ["x", "y"]}
    ]
}

where it implies

{
    "type" : "sequence",
    "transformations" : [
        { "type" : "f", "inputAxes" : ["x", "y"], "outputAxes" : ["x", "y"]},
        { "type" : "map_by_axis", "inputAxes" : ["x", "y", "z" ], "outputAxes" : ["x", "y", "z"]},
    ]
}

the first transformation "overwrites" the values of x and y, and map_by_axis "passes through" the modified x and y axes and sets the value of the output z axis to the input z axis.

C

We want to define the function

c = c
t = t
z = z
y = f(x,y) // output 0 of f is y
x = f(x,y) // output 1 of f is x

from the ctxyz coordinate system to the zyxtc coordinate system.

{
    "type" : "sequence",
    "transformations" : [
        { "type" : "f", "inputAxes" : ["x", "y"], "outputAxes" : ["y", "x"]},
        { "type" : "map_by_axis", "inputAxes" : ["c", "t", "x", "y", "z" ], "outputAxes" : ["c", "t", "x", "y", "z"]},
    ]
}

With sequences

As of 9 June 2022, the way to do this is by using inputAxes and outputAxes tags. For all of our examples, lets use these

coordinate systems

{ "name" : "ijk", "axes" : [ {"name" : "i"}, {"name" : "j"}, {"name" : "k"} ]},
{ "name" : "xyz", "axes" : [ {"name" : "x"}, {"name" : "y"}, {"name" : "z"} ]}
{ "name" : "xyz2", "axes" : [ {"name" : "x"}, {"name" : "y"}, {"name" : "z"} ]}

If every axis is included, things are clear:

example 1

{
  "type" : "sequence",
  "transformations" : [
    { "type" : "scale", "scale" : [2, 3], "inputAxes" : ["i", "j"], "outputAxes" : ["x","y"] },
    { "type" : "identity", "inputAxes" : ["k"], "outputAxes" : ["z"] },
  ]
  "input" : "ijk",
  "output" : "xyz"
}

Our rule is that inputAxes and outputAxes should default to being the axes of the input / output spaces in order, so

example 2

this

{
  "type" : "scale", 
  "scale" : [2, 3,4], 
  "input" : "ijk",
  "output" : "xyz"
}

implies this

{
  "type" : "scale", 
  "scale" : [2, 3, 4], 
  "input" : "ijk",
  "output" : "xyz",
  "inputAxes" : ["i", "j", "k"],
  "outputAxes" : ["x","y","z"] ,
}

Cool. So far, so good. I think this should be invalid because not all output dimensions are appear.

example 3

{
  "type" : "scale", 
  "scale" : [2, 3], 
  "input" : "ijk",
  "output" : "xyz",
  "inputAxes" : ["i", "j"],
  "outputAxes" : ["x","y"] ,
}

Since the output axis "z" is missing in "outputAxes" the above is invalid.

but what should this mean

{
  "type" : "scale", 
  "scale" : [2, 3], 
  "input" : "xyz",
  "output" : "xyz2",
  "inputAxes" : ["x", "y"],
  "outputAxes" : ["x","y"] ,
}

Since the output axis "z" is missing in "outputAxes" the above is invalid.

Scope

Intermediate axes (1)

Note; Even if we want this, I expect it to be a difficult sell to the NGFF community right now.

Can axes exist that are defined in the sequence of transformations and are needed to get from input to output, but that are not themselves in the input or output.

For example

"space" : [
  { "name" : "input", "axes" : [ "i" ] },
  { "name" : "output", "axes" : [ "x" ] }
],
"transform" : 
{
  "type" : "sequence",
  "transformations" : [
    { "inputAxes" : ["i"], "outputAxes" : ["tmp"], "scale" : [2] },
    { "inputAxes" : ["tmp"], "outputAxes" : ["x"], "translate" : [0.5] },
  ],
  "inputSpace" : "input",
  "outputSpace" : "output"
}

In simple cases, it would not be so bad since the above could turn into

this

"space" : [
  { "name" : "input", "axes" : [ "i" ] },
  { "name" : "tmp", "axes" : [ "tmp" ] },
  { "name" : "output", "axes" : [ "x" ] }
],
"transforms" : 
[
    { "type" : "scale", "inputSpace" : "in", "outputSpace" : "tmp", "scale" : [2] },
    { "type" : "translate", "inputSpace" : "tmp", "outputSpace" : "output", "translate" : [0.5] },
]

This works fine if implementations are expected to compose transformations (and they should be). The downside is "extraneous" spaces. It's not bad in this case, but could get ugly in more complex situations, especially if we allow re-used axes.

01234 > abcde > z

{
  "spaces": [
    { "name": "", "axes": ["0", "1", "2", "3", "4" ] },
    { "name": "tmp1", "axes": ["a", "1", "2", "3", "4" ] },
    { "name": "tmp2", "axes": ["a", "c", "d", "e", "3", "4" ] },
    { "name": "tmp3", "axes": ["a", "c", "d", "e", "f" ] },
    { "name": "z", "axes": [ "z" ] }
  ],
  "transforms": [
    { "name" : "0>ab", "inputSpace": "", "outputSpace": "tmp1" },
    { "name" : "12>cde", "inputSpace": "tmp1", "dim_2" ], "outputSpace": "tmp2"},
    { "name" : "34>f", "inputSpace": "tmp2", "outputSpace": "tmp3" },
    { "name" : "acf>z", "inputSpace": "tmp3", "outputSpace": "z" }
  ]
}

or maybe this isn't even allowed, but rather:

01234 > abcde > z

{
  "spaces": [
    { "name": "", "axes": ["0", "1", "2", "3", "4" ] },
    { "name": "ab", "axes": ["a","b"] },
    { "name": "cde", "axes": [ "c", "d", "e", ] },
    { "name": "f", "axes": ["f" ] },
    { "name": "z", "axes": [ "z" ] }
  ],
  "transforms": [
    { "name" : "0>ab", "inputSpace": "", "outputSpace": "ab" },
    { "name" : "12>cde", "inputSpace": "ab", "dim_2" ], "outputSpace": "cde"},
    { "name" : "34>f", "inputSpace": "cde", "outputSpace": "f" },
    { "name" : "acf>z", "inputSpace": "f", "outputSpace": "z" }
  ]
}

but that feels wrong too. Better than that:

01234 > abcde > z

{
  "spaces": [
    { "name": "", "axes": ["0", "1", "2", "3", "4" ] },
    { "name": "abcdef", "axes": ["a","b","c","d","e","f"] },
    { "name": "z", "axes": [ "z" ] }
  ],
  "transforms": [
    { "name" : "0>ab", "inputSpace": "", "outputSpace": "abcdef" },
    { "name" : "12>cde", "inputSpace": "", "dim_2" ], "outputSpace": "abcdef"},
    { "name" : "34>f", "inputSpace": "", "outputSpace": "abcdef" },
    { "name" : "acf>z", "inputSpace": "abcdef", "outputSpace": "z" }
  ]
}

But what's weird about that is that the output spaces for the first 3 transforms don't make any sense outside the context of the sequence.

Re-used axes (2)

Can an input axis be used by multiple transformations?

For example

"space" : [
  { "name" : "input", "axes" : [ "i" ] },
  { "name" : "output", "axes" : [ "x", "y" ] }
],
"transform" : 
{
  "type" : "sequence",
  "transformations" : [
    { "inputAxes" : ["i"], "outputAxes" : ["x"], "scale" : [2] },
    { "inputAxes" : ["i"], "outputAxes" : ["y"], "scale" : [0.5] },
  ],
  "inputSpace" : "input",
  "outputSpace" : "output"
}

Non-unique axes (3)

Can an axes in two different spaces share a name? is this allowed

example

"space" : [
  { "name" : "input", "axes" : [ "x" ] },
  { "name" : "output", "axes" : [ "x" ] }
]

Sequences

Note: I'm omiting some things in the official spec and using shorthands instead.

Suppose we have the spaces:

"space" : [
  { "name" : "input", "axes" : [ "i", "j" "k" ] },
  { "name" : "output", "axes" : [ "x", "y", "z" ] }
]

Example 1. It's clear that this transformation sequence:

{
"type" : "sequence",
"transformations" : [
  { "inputAxes" : ["i"], "outputAxes" : ["x"], "scale" : [2] },
  { "inputAxes" : ["j"], "outputAxes" : ["y"], "scale" : [3] },
  { "inputAxes" : ["k"], "outputAxes" : ["z"], "scale" : [4] },
],
"inputSpace" : "input",
"outputSpace" : "output"
}

describes

x = 2*i
y = 3*j
z = 4*k

how should should this transform be built in code though (in a general way)?

Permute for imglib2

Add permutations so that the relevant axes always appear in the first position. For example 1, this would be building a transformation like this:

[i j k]
i -> x
[x j k]
permute [1,0,2]
[j x k]
j -> y
[y x k]
permute [2,1,0]
[k y x]
k -> z
[z y x]
permute [0,1,2]
[x y z]

If transformations have more output than input dimensions, implementation will need to detect that fact and pass points with the number of necessary dimensions. more dimensions.

{
"type" : "sequence",
"transformations" : [
  { "inputAxes" : ["i"], "outputAxes" : ["x","z"], "scale" : [2] },
  { "inputAxes" : ["j"], "outputAxes" : ["y"], "scale" : [3] },
],
"inputSpace" : "input",
"outputSpace" : "output"
}

then

[i j]
(notice that the first transform has 1 input and 2 outputs)
add dimension (call it #, but its just a placeholder)
[i j #]
permute [0,2,1]
[i # j]
i,# -> x,z
[x z j]
permute [2,0,1]
[j x z]
j -> y
[y x z]
permute [1,0,2]
[x y z]

If re-used axes (2) is in-scope, then permuting before and after is not sufficient. If for example:

"space" : [
  { "name" : "input", "axes" : [ "i" ] },
  { "name" : "output", "axes" : [ "x", "y" ] }
],
"transform" : 
{
  "type" : "sequence",
  "transformations" : [
    { "inputAxes" : ["i"], "outputAxes" : ["x"], "scale" : [2] },
    { "inputAxes" : ["i"], "outputAxes" : ["y"], "scale" : [0.5] },
  ],
  "inputSpace" : "input",
  "outputSpace" : "output"
}

[i]
(notice output is 2d, so pad)
[i a]
i -> x
[x a] # and now we don't have the correct input value for the second transformation

low dim transform to high dim

A similar idea is to wrap the transformations in a class that maps the inputs and outputs of a lowD transformationation into a higher dimensional point. I expect the performance of this to be similar to the above method using permutations, but the API might be different.

Using this example again:

{
"type" : "sequence",
"transformations" : [
  { "inputAxes" : ["i"], "outputAxes" : ["x","z"], "scale" : [2] },
  { "inputAxes" : ["j"], "outputAxes" : ["y"], "scale" : [3] },
],
"inputSpace" : "input",
"outputSpace" : "output"
}

then

[i j]
(notice that the first transform has 1 input and 2 outputs)
add dimension (call it #, but its just a placeholder)
[i j #]
i,# -> x,z
(first transform: take inputs from dimension [0], map output to indexes [0,2]
[x j z]
j -> y
(second transform: take inputs from dimension [1], map output to indexes [1]
[x y z]

All the axes all the time

Perhaps overkill, but safe. The values of all coordinates are tracked through the whole sequence. This or the option below are needed if axes can be reused. Same example as before:

"space" : [
  { "name" : "input", "axes" : [ "i" ] },
  { "name" : "output", "axes" : [ "x", "y" ] }
],
"transform" : 
{
  "type" : "sequence",
  "transformations" : [
    { "inputAxes" : ["i"], "outputAxes" : ["x"], "scale" : [2] },
    { "inputAxes" : ["i"], "outputAxes" : ["y"], "scale" : [0.5] },
  ],
  "inputSpace" : "input",
  "outputSpace" : "output"
}

then:

[i]
i -> x
[i x]
i -> y 
[i x y]
(permute and crop to output space)
[x y]

A more involved example with intermediate axes as well:

{
  "spaces": [
    { "name": "", "axes": ["0", "1", "2", "3", "4" ] },
    { "name": "f", "axes": [ "f" ] }
  ],
  "transforms": [
    { "name" : "0>ab", "input_axes": [ "dim_0" ], "output_axes": [ "a", "b" ] },
    { "name" : "12>cde", "input_axes": [ "dim_1", "dim_2" ], "output_axes": [ "c", "d", "e" ] },
    { "name" : "34>f", "input_axes": [ "dim_3", "dim_4" ], "output_axes": [ "f" ] },
    { "name" : "acf>z", "input_axes": [ "a", "c", "f" ], "output_axes": [ "z" ] }
  ]
}

then the sequence, tracking all coordinates would result in :

[0 1 2 3 4]
0 > a,b
[0 1 2 3 4 a b]
1,2 > c,d,e
[0 1 2 3 4 a b c d e]
3,4 > f
[0 1 2 3 4 a b c d e f]
a,c,f > z
[0 1 2 3 4 a b c d e f z]
(permute and crop to output space)
[z]

Axis-point

Store axis values in a Map<String,Double>, and wrap this structure in a RealPoint where values can be queried by strings (axis names), instead of longs.

SpacePoint p = new SpacePoint( ["x", "y", "z"], [1, 2, 3] )
p.get("z") // returns 3

p.setSpace(["y", "z", "x"]),
p.getPosition() // returns [2, 3, 1]

Track only required axes

Check which axes need to be used in subsequent transformations. Store those, omit others. Less overkill than the above method, but involves more bookkeeping, and analysis if the sequence of transforms before starting to apply things. Main benefit is not carrying around lots of unnecessary stuff, but that may not be a huge benefit.

[0 1 2 3 4]
0 > a,b 
( [0 b] are not used in any subsequent transformations)
[1 2 3 4 a]
1,2 > c,d,e
([1 2 d e] are not used in any subsequent transformations)
[3 4 a c ]
3,4 > f
[a c f]
([3 4] are not used in any subsequent transformations)
[a c f]
a,c,f > z
([3 4] are not used in any subsequent transformations)
[z]