ref(workflows): Cache assignee lookup in-memory

[cmanallen]

Its the same lookup on repeat. We can reuse the results from the previous iteration.

[kcons]

Yeah, this is a known perf issue in our condition evaluation that we've got special-cased for Snuba-dependent conditions but mean to eventually generalize; ideally, we could bulk gather required data, then run an evaluation phase.
If it's causing issues I'm certainly not against mitigating it before we get around to fixing it structurally, but I'd feel comfortable with a bit more structure, like a doc on _cache and maybe a TypedDict showing the shape of the data. Or, keying it by a private module level object() so the data is borderline unreachable by any other means. WorkflowEventData is used in lots of places and is really intended to be immutable, so I don't want to make it a bag-of-vals without some safeguards. Or maybe this is a case for local_cache()? Nobody seems to use that, but if roundtrips to memcache are a problem, that seems build for that.

[cmanallen]

@kcons

If it's causing issues I'm certainly not against mitigating it before we get around to fixing it structurally,

No issues per se. I was looking at the top span emitters by count (and duration) and I spotted the trace I linked above. Figured I'd fix it rather that leave it.

As for structure, sure I can do that. The goal was to be minimally intrusive so I wouldn't exhaust potential collaborators. We can do whatever here. If you have a preference let me know. Adding stronger typing is certainly an easy value add.

[cmanallen]

@kcons Any thoughts?