Cancelling / invalidating complex batches

[epchris]

I have a few workflows that are implemented using nested batches and I'm struggling with figuring out what is the best way to allow for invalidating the entire structure. My system tracks the top-level batch ID tied to a model, so I have easy access to that, but if I invalidate the top-level batch it doesn't seem that a job in a child batch has a true response to valid_within_batch?.

Is there some "good" way to propagate batch invalidation to the entire tree of batches if I invalidate the top-level batch manually? One approach that has occurred to me is for a job to check its own batch for invalidation and then "walk" the parent tree upwards checking each batch above itself.

[mperham]

There's no easy way to invalidate a tree. Your app code which invalidates a batch can also easily recurse up and invalidate parent, gp, etc if the situation calls for it.

[epchris]

Thanks for chiming in, @mperham ! In a situation where we only track the top-level batch key, and that batch may or may not have child batches, can you think of a consistent pattern that would allow for cancelling the top level batch (which we track the ID of) and having that "trickle down" to all the child batches (if they exist)?

[mperham]

There's effectively no easy way to do that. The Batch data structure in Redis makes it easy to go "up" the Batch tree from leaf / child node to the parent node, but does not have an index to go from parent to children. You'd need to write some lower-level Redis code to iterate through all open Batches.

Alternatively you'd need to create your own BID tracking for each Batch in the tree so you can invalidate them all with a simple loop.