Support `FilterPolicy` in worker service

[Lou1415926]

Copilot should provide a native way for user to specify subscription filter policy for their worker service.

Related: discussion on Gitter

[lsnape]

To dig into the use case, we have a CPU- and memory-intensive worker that has a min task count of 0. There can be several hours without events, so we'd like to not run the task when there's no work to do.

We have multiple workers subscribed to the same events topic. Not all workers are interested in every event on that topic. We could circumvent this by configuring the upstream producer write to multiple topics, but that doesn't take advantage of the decoupling between topics and subscriptions.

[rfma23]

It would be really useful to have a way of specifying a FilterPolicy in a similar way as in AWS SAM, something along the lines of:

FilterPolicy:
  fare:
    - numeric:
        - '>='
        - 50
  distance:
    - numeric:
        - '>='
        - 20

[iamhopaul123]

Proposal

Thanks @rfma23 for giving this brilliant idea. To put everything together, we have a new field in our manifest for each topic as

filter_policy: json/yaml

For example:

subscribe:
  topics:
    - name: events
      service: api
      filter_policy: {"store":["example_corp"],"event":[{"anything-but":"order_cancelled"}],"customer_interests":["rugby","football","baseball"],"price_usd":[{"numeric":[">=",100]}]}
      queue: # Define a topic-specific queue for the api-events topic.
        timeout: 20s
    - name: events
      service: fe
      filter_policy:
        store:
          - example_corp
        event:
          - order_cancelled
        encrypted:
          - false
        customer_interests:
          - basketball
          - baseball
  queue: # By default, messages from all topics will go to a shared queue.
    timeout: 45s
    retention: 96h
    delay: 30s

Referred to https://docs.aws.amazon.com/sns/latest/dg/example-filter-policies.html

[rfma23]

Proposal

Thanks @rfma23 for giving this brilliant idea. To put everything together, we have a new field in our manifest for each topic as

filter_policy: json/yaml

For example:

subscribe:
  topics:
    - name: events
      service: api
      filter_policy: {"store":["example_corp"],"event":[{"anything-but":"order_cancelled"}],"customer_interests":["rugby","football","baseball"],"price_usd":[{"numeric":[">=",100]}]}
      queue: # Define a topic-specific queue for the api-events topic.
        timeout: 20s
    - name: events
      service: fe
      filter_policy:
        store:
          - example_corp
        event:
          - order_cancelled
        encrypted:
          - false
        customer_interests:
          - basketball
          - baseball
  queue: # By default, messages from all topics will go to a shared queue.
    timeout: 45s
    retention: 96h
    delay: 30s

Referred to https://docs.aws.amazon.com/sns/latest/dg/example-filter-policies.html

If you could implement this exactly as proposed it would be greatly appreciated 😊

[iamhopaul123]

If you could implement this exactly as proposed it would be greatly appreciated 😊

Yeah I am working on it! Please stay tuned.

[efekarakus]

Hi folks! This feature is now released in v1.16 https://github.com/aws/copilot-cli/releases/tag/v1.16.0 🥳
See announcement: https://aws.github.io/copilot-cli/blogs/release-v116/#define-messages-filter-policy-in-publishsubscribe-service-architecture