-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Consumers with Multiple Filters ADR-34 #192
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| # Title JetStream Consumers Multiple Filters | ||
|
|
||
| | Metadata | Value | | ||
| |----------|---------------------------| | ||
| | Date | 2023-01-18 | | ||
| | Author | @Jarema | | ||
| | Status | Approved | | ||
| | Tags | jetstream, client, server | | ||
|
|
||
| ## Context and Problem Statement | ||
|
|
||
| Initially, JetStream Consumers could have only one Filter Subject. | ||
| As the number of feature requests to specify multiple subjects increased, this feature was added. | ||
| That could also reduce the number of Consumers in general. | ||
|
|
||
| ## Context | ||
|
|
||
| Server PR: https://github.com/nats-io/nats-server/pull/3500 | ||
|
|
||
| ## Design | ||
|
|
||
| ### Client side considerations | ||
|
|
||
| To implement the feature without any breaking changes, a new field should be added to Consumer config, both on the server and the clients. | ||
|
|
||
| 1. The new field is: | ||
| `FilterSubjects []string json:"filter_subjects"` | ||
|
|
||
| 2. Subjects can't overlap each other and have to fit the interest of the Stream. | ||
| In case of overlapping subjects, error (10136) will be returned. | ||
|
|
||
| 3. Only one of `FilterSubject` or `FilterSubjects` can be passed. Passing both results in an error (10134) | ||
|
|
||
| 4. Until future improvements, only old JS API for consumer creation can be used (the one without `FilterSubject`) in consumer create request. Using new API will yield an error (10135). | ||
|
|
||
| 5. Each client, to support this feature, needs to add a new field that Marshals into a json array of strings. | ||
|
|
||
| 6. To ensure compatibility with old servers that are not aware of `filter_subjects` field, client's should check the returned info (from update or create) if the filters are set up properly. | ||
|
|
||
| **Example** | ||
| ```json | ||
| { | ||
| "durable_name": "consumer", | ||
| "filter_subjects": ["events", "data"] | ||
| } | ||
| ``` | ||
| 6. Client does not have to check if both, single and multiple filters were passed, as server will validate it. | ||
| Client should add new errors to return them in client language idiomatic fashion. | ||
|
|
||
| ### Server side | ||
|
|
||
| To make this change possible and reasonable peformant, server will have a buffer of first message for each subject filtered and will deliver them in order. After delivering message for a subject, buffer for that subject will be filled again, resulting in close to no overhead after the initial buffer fill. | ||
|
|
||
| This can be optimized but will not affect API. | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$JS..CONSUMER.CREATE.>There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More importantly, if it is an update, the consumer could have been created using the new API, so it could have a
name, depending on how the client is handling this it may reject thename. Effectively, ifArray.isArray(config.filter_subjects), it should use the old API. Server will properly catch iffilter_subjectand filter_subjects` are both set.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we prevent a consumer update from switching between FilterSubjet and FilterSubjects?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No.
User can switch the consumer from being single-filtered to multi-filtered.
Considering above, we might want to deny it until we land solution for multiple filters and new API.