Grids & combinatorial outputs

[space-nuko]

Adds a grids feature similar to XYZ-plots in webui, supporting an arbitrary number of axes

NOTE: It's still extremely WIP and assumes a uniform batch size, nodes that change the batch size partway or rebatch latents will almost certainly break it. Also I need to make sure the new PrimitiveNode widgets don't break existing workflows

Test workflow

Sets up a 5-axis grid from various parameters

https://files.catbox.moe/tym26a.json

Features added

• The executor now supports combinatorial inputs. This means it will generate a new prompt for every combination of one or more list of input values for each node, and pass each one of those prompts further down the graph combinatorially as well

• All nodes now show batch progress in addition to individual progress if they're being mapped over combinatorially

• PrimitiveNode gets new options for defining grid axis options, list for a comma separated list and range for numeric ranges (in supported input types). Each one of the two new value types will add a new grid axis to the output

(NOTE: The list type needs to change from comma-separated lists to support lists of text prompts, which almost always contain commas internally. Also I may just split this functionality into its own GridAxis node instead)

• PrimitiveNode can combine more than one attached input into a single axis, for example to treat both LoRA weights as one axis. And this works (or at least is supposed to) for more than one node attached

• SaveImage and PreviewImage can now show grids if combinatorial outputs were used

Implementation

Internally some new state was added to inputs and outputs in the partially executed graph to keep track of which grid axis they came from. Nodes with more than one output like LoraLoader treat all of their outputs as having originated from the same axis, such that both outputs (CLIP and MODEL) are treated as one axis instead of two separate axes. Otherwise the CLIP from one batch could be combined with the MODEL from another batch, which never happens during normal usage of LoraLoader, so it's an invalid state

PrimitiveNode internally "joins" all the inputs it's connected to into one axis as well, by associating each connection with a unique axis ID (which is the PrimitiveNode's ID for now). This ID is attached to the node's outputs and propagated downstream so long as those outputs have the only unique combinatorial axis ID across the inputs the other nodes receive, else the axis ID is reset so a new combination can be created

The frontend relies on the sorting of the nodes' inputs based on some criteria to correctly associate the final output images with their coordinates in the graph. New combinatorial inputs will be prepended to the set of combinations, such that earlier axes will be changed more frequently. As an example, if A is connected to B, and each has a weight axis with two values, then the order in the grid will be

(B1, A1)
(B1, A2)
(B2, A1)
(B2, A2)

If there is more than one axis attached to a node then they will be applied in the alphabetical order of the input names

This order is recreated in the frontend by traversing the graph backwards from the output image nodes, instead of the backend explicitly sending the mapping from output to grid coordinates, as it gets pretty arduous to keep track of. I'm not even sure it can be tracked in the general case if the batch size changes since there's no universal way to correlate a number of inputs to outputs in a given backend node

Basically while there are a lot of use cases that this built-in grid feature can support, there are bound to be many it can't support, and it's a matter of throwing as many prompts as possible at it until it breaks so that those edge cases can be uncovered

[mcmonkey4eva]

Comfy's general preference for nodes like this is to release it as a custom node unless there's a clear reason to put it in core.