Notification groups, lists and relationship matrices

[proditis]

Hi everyone,

This thread is with regards to the a new feature we were discussing at PR #78

The idea is for us to expand the current probe and notify commands in such a way that would allow us to create custom notification relationships between probes and notify endpoints.

Some examples of the different structures that came during the initial discussions include (in order they arrived so that the progress can be identified):

From @proditis Add support to define desired notification on probes

tcp:
  - name: Memcached
    host: 10.7.0.253:11211
    notify: "Server #Alert" # notify.discord.name

From @nullsimon support custom probe with different notifies, notifies group or notifies list are both great

http:
  # Some of the Software support the HTTP Query
  - name: ElasticSearch
    url: http://elasticsearch.server:9200
    notify: GroupA,telegram # all of groupA + individual notify endpoint named telegram
  - name: Prometheus
    url: http://prometheus:9090/graph
    notify: telegram,discord
groupA:
 discord:
 slack:
 GroupB:
groupB:
 discord:
 slack:

From @haoel to cover all of the scenarios, the best model is the n:m relationship between probe and notification, but this would introduce the complexities of configuration - we need an extra configuration to describe the relationship.

relationsip:
- probe_id_1 : notification_id_1
- probe_id_2 : notification_id_1
- probe_id_1 : notification_id_2

So now we have to decide the Hows and Whats for the future of this feature 👍

Feel free to contribute your ideas and concerns here so that they will be considered and addressed 😃

[nullsimon]

for example, maybe not appropriate, I want to use linux file and user models

probe = file
probes = folder (a group of file)
notify = user
notifies = user group

probe to probes group is relation, notify to notifies group is relation, and probe to notify or probe group to notify are n:m relation indeed.

using relationship from @haoel are good for implements and for offline data analysis

while using my idea are easy to understand and use, but not good for analysis

[proditis]

tried working out an example configuration with some of the ideas proposed, here is what it looks like so far,

http:
  - name: ElasticSearch
    url: http://elasticsearch.server:9200
    notify:
        - emergency_channels
        - emergency_channels!telegram (exclude telegram from the group)
        - emergency_channels>discord (Only discord notify from emergency_channels group)
        - discord (default_channels discord only)
        - recovery_channels!&SLA (do not call this channel when we submit our daily SLA report)
  - name: Prometheus
    url: http://prometheus:9090/graph
    # omit notify: uses default notify channels like before
notify:
  recovery_channels:
        - elastic restart notify endpoint
  emergency_channels:
      discord:
        - name: "#alerts channel"
        - name: "#support channel"
      telegram:
  default_channels: # From this point we operate with the same logic as before, unless somehow excluded by the admin all these notifies will be triggered.
      slack:
        - name: "MegaEase#Alert"
      discord:
        - name: "MegaEase#Alert"
      telegram:
        - name: "MegaEase Alert Group"

Some things that come to mind as i was working on this

• We are going to need a "default" notify group that behaves like before and that we can fall back there is NO explicit notify for a probe (default_channels: here)
• seems more practical for the future if we add a notify: list to avoid complex string parsing and conversions
• SLA daily reports seem to get in the way and we may need a way to avoid sending it to all notify channels. In the example above i excluded the sla reports from some notify endpoints with the pseudo-syntax !&SLA
• The recovery triggers we were discussing can now become normal notify endpoints recovery_channels which we trigger if and when we want them

Let me know what you think.

[haoel]

I still need more time to think about this. especially, balance the complexity and the requirement.

[haoel]

I quite like the channel solution, this solution is intuitive in how we send the probers data to notifiers.

But we can have several different ways to configure it.

Solution A - Group the notifiers to channel

This is the proposal from @proditis (refer link)

http:
   - name: probe_a
     channels: [ chan_a, chan_b]
notify:
   chan_a:
      - discord: ....
      - slack: ...
   chan_b:
      - telegram: ...

Solution B - Define the ID of the channels for both probers and notifications

http:
   - name: probe A
     channels : [ Chann_A, Chann_B]
shell:
   - name: probe B
     channels: [ Chann_C ]
notify:
   - discord: Dev
     channels: [Chann_A, Chann_C]
   - email: Gmail
     channels: [Chann_B]

Solution C: Dedicated configuration for the channel

http:
...
shell:
...
notify:
...
channels:
   - name: channel_one
      probers: [A, B, C]
      notify: [X, Y]
   - name: channel_two
      probers: [AA, BB, CC]
      notify: [Z]

Solution Comparsion

The following comparison might not be objectives, please feel free to raise your concerns.

Aspects	A Notify Group	B Channel ID	C Channel Section	Notes
Ease to configure	Good	Average	Bad	The solution B could confuse people. and the darkside of C is if we changed the probe name we have to go to channels to change the probe name
Backwords Compatibility	Bad	Good	Good	The new changes could be optional, and the old configuration file can be compatible by using the default values. Solution A is not the case.
Errors Checking	Good	Bad	Good	Solution B needs more checks. A and B are easy to check.
Coding efforts	Medium	Medium	Medium	the code efforts are the same.

@proditis @nullsimon please feel free to share any of your ideas.

[nullsimon]

Thanks for your great insight and comparisons.

I'm prefer solution B,it's store relation metadata in both probers and notifications,and decoupled by channels.And further more,it can support a slightly problem level feature like @proditis proposed before.

[proditis]

Awesome analysis thank you for sharing with us.

A introduces some repetitions of settings, if we want the same discord notify we have to copy to 2 groups

With C, although is the most complete of them all, has the side effect to be error prone in configuring, i had to go back and forth to my notify and probes to get their names. It got confusing really fast 😄

I agree with @nullsimon B seems to be the better candidate as it covers more use cases with less repetition of config segments. And is straight forward both in the way of configuring and "reading" the config.

For the backwards compatibility issue with B case, we can use an internal channel name (say default) that we can assign any probes or notify segments that are without a channels key. This way when we bring forward configuration files from previous versions the probes and notify can work without a change?

I'll run a few more config "simulations" 😄 and see if I missed anything in each case but so far the most promising seems to be the B

[haoel]

Cool, I will work on Solution B after we release v1.5.0 ;-)

[haoel]

PR #108 is ongoing...

[proditis]

Not to worry, i just got excited when i saw it 😄 and jumped in to check. not reviewing anything yet until you get the PR out of draft 👍