Develop a way to make Ops usable easily from CompGraph framework

[ctrueden]

Here are some interfaces we came up with that could deliver what CompGraph needs on the SciJava/ImageJ Ops side:

Base interface for structured computations:

public ComputationStructure<S> {
	default S transform(S in) { return in1; } // e.g. identity
	S requiredInput(S out, S in);
}

public ComputationBiStructure<S> {
	default S transform(S in1, S in2) { return in1; } // e.g. identity
	default S requiredInput1(S out, S in1, S in2) { }
	default S requiredInput2(S out, S in1, S in2) { }
}

SciJava/ImageJ Ops need a way to deliver objects implementing the appropriate (unary or binary) base interface. The issue is that a higher-arity op needs to lock down its "secondary" parameters (i.e. non-images; things that KNIME would have a user specify in the configuration dialog), to reduce arity down to unary or binary. We could have interfaces like the following to accomplish this:

Structured, BiStructured, etc.

public final class Structured<S> {
	ComputationStructure<S> structure();
}

public final class BiStructured<S> {
	ComputationBiStructure<S> structure();
}

public final class Structured.With1Param<S, P> {
	ComputationStructure<S> structure(P param);
}

public final class Structured.With2Params<S, P1, P2> {
	ComputationStructure<S> structure(P1 param1, P2 param2);
}

public final class BiStructured.With1Param<S, P> {
	ComputationBiStructure<S> structure(P param);
}

public final class BiStructured.With2Params<S, P1, P2> {
	ComputationBiStructure<S> structure(P1 param1, P2 param2);
}

Then for ImageJ Ops, we need to specify the structure S as some metadata container. Probably something like this:

public class ImageInfo {
	Object elementType(); // i.e. the ImgLib2 pixel type T
	Interval interval();
}

Here is an example: gaussian convolution with locked down double[] sigmas:

GaussianConvolveWithSigmas

public class GaussianConvolveWithSigmas<T extends RealType<T>> implements Computers.Arity2<RAI<T>, double[], RAI<T>>, Structured.With1Param<ImageInfo, double[]> {
	public void compute(RAI<T> image, double[] sigmas, RAI<T> output) {
		// ...
	}

	public ComputationStructure<ImageInfo> structure(final double[] sigmas) {
		return new ComputationStructure<ImageInfo>() {
			@Override
			public ImageInfo transform(final ImageInfo in) {
				// e.g. identity
				return in;
			}
			@Override
			public ImageInfo requiredInput(final ImageInfo out, final ImageInfo in) {
				return new ImageInfo() {
					@Override
					public Object elementType() { return in.elementType(); }
					@Override
					public Interval interval() {
						// NB: Derive required input interval from output interval + sigmas.
						return requiredInputComputedFromSigmas(out.interval(), sigmas); // Could also rely on in.interval() if needed.
					}
				}
			}
		}
	}
}

The double[] sigmas state here is implicit in the new ComputationStructure anonymous class we define.

One question this does not answer is: how to go from an actual input object (e.g., RandomAccessibleInterval<DoubleType> to its corresponding structure (ImageInfo)? We could use an extensible service/plugin mechanism. Or just keep it simple and rely on the caller to take care of this somehow—in which case maybe ImageInfo has constructors that are helpful for this.

For now, let's assume we have method structureOf(inputValue)" available that returns ImageInfo`.

Here is another example that maps to a binary structured computation:

Matrix multiplication

public class MatMultiply<T extends RealType<T>> implements Computers.Arity2<RAI<T>, RAI<T>, RAI<T>>, BiStructured<ImageInfo> {
	public void compute(RAI<T> mat1, RAI<T> mat2, RAI<T> out) {
		// NB: We can validate the inputs+output structure here! Nice!
		checkStructure(mat1, mat2, out);
		S s1 = structureOf(mat1), s2 = structureOf(mat2);
		S expectedStructure = structure().transform(s1, s2);
		if (!expectedStructure.equals(structureOf(out))) throw new IllegalArgumentException("Output structure not compatible with input structures.");
		// OR: Better might be to do this the builder, to avoid repeated checking in loops over the same op instance with "known good" args.

		// Finally, do the algorithm here ...
	}

	public ComputationBiStructure<ImageInfo> structure() {
		return new ComputationBiStructure<ImageInfo>() {
			@Override
			public ImageInfo transform(final ImageInfo in1, final ImageInfo in2) {
				if (in1.numDimensions() != 2 || in2.numDimensions() != 2) alsoFail(); // also check matching offsets? what is a mat multiply with non-zero offset ???
				if (!in1.equals(in2)) throw new IllegalArgumentException("Input structures do not match.");
				return Interval(in1.interval().dimension(0) + in2.interval().dimension(1)); // too naive -- need to preserve offset
			}
			@Override
			public ImageInfo requiredInput(final ImageInfo out, final ImageInfo in) {
				return new ImageInfo() {
					@Override
					public Object elementType() { return in.elementType(); }
					@Override
					public Interval interval() {
						// NB: Derive required input interval from output interval + sigmas.
						//
					}
				}
			}
		}
	}
}

Notice that we can check the structure of a computation before trying to do it. We may want to do this outside the actual compute method, however, for performance reasons. The builder is a possible place this could happen. Or a new layer could be introduced, perhaps. Regardless, helper methods to validate seem to make sense.

A key question is whether things behave correctly when computing on non-zero offset tiles/blocks of inputs—since that is the entire point of CompGraph. More thought needed!

Next steps

• Consider whether to create composition interfaces for e.g. Computers.Arity2<I1, I2, O> + Structured.With1Param<S, I2(?)> ➡️ StructuredComputers.Arity2_With1Param<S, I1, I2, O>
• Ideally, the structure(...) implementations should be able to use a lambda in the common case of identity interval transform: we often have same-size input -> output—but the requiredInput computation varies more.
• Abstract base classes can define common behavior for e.g. neighborhood ops. Ideally we make this an interface again with default method(s), rather than abstract class, so lambdas are still usable.
• Consider what to do about @OpDependency chaining—can we leverage this somehow in CompGraph? Can we build multiple CompGraph nodes out of composed Ops?

[ctrueden]

Even if nothing else comes of this design, I ❤️ that the structure specification allows for input validation prior to each computation. This is otherwise missing from Ops, meaning computers become more "dangerous" and error prone. I think we want to, at minimum, introduce the out = transform(in) idea with structured metadata and then use it to check! The requiredInput is trickier...

[gselzer]

Time commitment: 4 weeks