Unmaterialized S3 sources with SQS notifications will lose data if multiple views depend on them

[quodlibetor]

Since there is no way to specify different SQS queues for different
source instances, there is no way to ensure that all objects are
discovered as new source instances come up.

Mentioned in #7414, there are a few things we can do:

1. Probably fastest, not great: refuse to create unmaterialized S3 sources that have SQS notifications enabled
2. Middle of the road: create a way to share S3 objects discovered via SQS notifications to all instances of a given source. While there are multiple possible implementations, this should be doable by adding a map of OID -> MPSC queue that is shared by all source instances, and sending new object discovered messages on each of those queues in addition to the existing queue that the SQS discovery task sends on.
3. Most complex: fully implement s3 sources: Support specifying a catalog of objects to download/ingest #6605

IMO the second option should work pretty well and be easy to implement.

[quodlibetor]

Update after speaking with @frankmcsherry: option 2 is doable by stashing an Rc<..> on our Worker or RenderState struct and ensuring that all instances of the same source render to the same worker.

This is actually another instance where things would compose more naturally if SQS notifications were their own potentially-hidden dataflow: we wouldn't need to do any worker-pinning in that case.

[benesch]

Closing as stale, as we no longer support S3 sources.