Reduce replication traffic due to reflected cache stream POSITION

[erikjohnston]

This was exacerbated by #16473

[clokep]

Does this mean that workers don't need to track other workers cache stream positions?

[erikjohnston]

The difference is nothing uses the global "persisted upto position" for the caches stream, as everything that uses the caches streams only care about each streams position, not any sort of "combined" position. The reason we care about that for other streams is there is some code that still thinks that all streams can be considered as a single integer (e.g. federation and application sending code).

[clokep]

global "persisted upto position" for the caches stream

Just to refresh my memory -- this is the minimum persisted position across all workers, correct?

everything that uses the caches streams only care about each streams position

Because the caches stream gets blasted to everyone and they just clear their caches but we don't really care about tracking it? Do we even care about each stream's position in it?

[erikjohnston]

(Note I've just pushed a commit that changes things a bit)

global "persisted upto position" for the caches stream

Just to refresh my memory -- this is the minimum persisted position across all workers, correct?

Yup.

everything that uses the caches streams only care about each streams position

Because the caches stream gets blasted to everyone and they just clear their caches but we don't really care about tracking it? Do we even care about each stream's position in it?

There are two usages of the caches stream:

1. Reliably receiving updates about cache invalidations that happened on other workers. We use the stream position here purely to detect when gaps happened.
2. We sometimes want to wait for the caches stream to get to a particular position (this where we do a HTTP request and we want to wait for the action to replicate to the requesting worker after receiving a response).

[clokep]

I'm fairly certain this is OK.