Switching emitter. This will allow for a per feed emitter designation.#13363
Conversation
This will work by looking at an event's feed and direct it to a specific emitter. If no specific feed is specified for a feed. The emitter can direct the event to a default emitter.
server/src/main/java/org/apache/druid/server/emitter/SwitchingEmitterConfig.java
Show resolved
Hide resolved
core/src/main/java/org/apache/druid/java/util/emitter/core/SwitchingEmitter.java
Outdated
Show resolved
Hide resolved
| public void flush() throws IOException | ||
| { | ||
| boolean fail = false; | ||
| log.info("Flushing Switching Emitter."); |
There was a problem hiding this comment.
what frequency is this logging going to be at?
server/src/main/java/org/apache/druid/server/emitter/SwitchingEmitterConfig.java
Show resolved
Hide resolved
docs/configuration/index.md
Outdated
| |Property|Description|Default| | ||
| |--------|-----------|-------| | ||
| |`druid.emitter.switching.emitters`|Map of feed to list of emitter modules to load that will be used for each feed, e.g., {"metrics":["http"], "alerts":["logging"]}|{}| | ||
| |`druid.emitter.switching.defaultEmitters`|List of emitter modules to load that will be used if there is no emitter specifically designated to the event feed, e.g., ["logging","http"].|[]| |
There was a problem hiding this comment.
so that event feed will go to all the default emitters?
There was a problem hiding this comment.
So this where naming gets weird. Default is probably the wrong name. since it could be read that no matter what every event goes to the default emitters but i see it as it as the list of emitters to use if there isn't a match for an event's feed
Example of how it's working as coded:
So with a hypothetical config and hypothetical event feed names, and emitters
druid.emitter.switching.defaultEmitters=[http, foo]
druid.emitter.switching.emitters={"metrics":["statsd", "logging"], "alerts":["bar"]}
if an event with feed metrics is emitted. it will only be emitted to statsd and logging emitters and will not go to the default emitters.
if an event with feed requestLog it would be emitted to the http and foo emitters
if an event with feed alerts it would be emitted to bar emitter
There was a problem hiding this comment.
@abhishekagarwal87 One thing that can be done here is i add this per feed functionality onto the composing emitter instead of making a new one...
There was a problem hiding this comment.
I think the current structure is fine since the final JSON would get a bit unwieldy. what do you think?
There was a problem hiding this comment.
ok, no complaints from me then
core/src/main/java/org/apache/druid/java/util/emitter/core/SwitchingEmitter.java
Show resolved
Hide resolved
…d docs more clear
|
Thank you @TSFenwick for the contribution. |
3 participants
This will work by looking at an event's feed and direct it to a specific emitter. If no specific feed is specified for a feed. The emitter can direct the event to a default emitter.
Description
The goal of this PR is to allow there to be a way to filter events to a different emitter. For example druid cluster might be sending metrics with the statsd emitter, but we want to emit request log events to an http emitter. Currently in druid there is no way to do this without sending all the events to both of them. And each emitter doesn't necessarily have the ability to filter out events it doesn't want to send/hold onto in memory.
Create and Wire Up Switching Emitter ...
We chose to use a linear search for finding the correct emitter based on feed because the number emitters and feeds is likely to be low enough that a linear search will likely be the fastest.
Release note
New you can now use a specific emitter for each feed. For example this means that alerts can go to http emitter, metrics can go to statsd emitter, and requestLog events can go to a kafka emitter.
Key changed/added classes in this PR
SwitchingEmitterSwitchingEmitterConfigSwitchingEmitterModuleThis PR has: