Deduplicate alert manager alerts by caching their hashes #1396
Conversation
RobertSzefler
force-pushed
the
feature/main-1609/dedup-alertmgr-alerts
branch
from
ae16603
to
fb37d09
Compare
RobertSzefler
force-pushed
the
feature/main-1609/dedup-alertmgr-alerts
branch
4 times, most recently
from
93f16f1
to
8b07759
Compare
RobertSzefler
force-pushed
the
feature/main-1609/dedup-alertmgr-alerts
branch
from
8b07759
to
c28ddc5
Compare
| alert_hash = Web.get_compound_hash([ | ||
| alert.fingerprint.encode('ascii'), | ||
| alert.status.encode('utf-8'), | ||
| str(alert.startsAt.timestamp()).encode('ascii'), |
There was a problem hiding this comment.
please also add the endsAt
I don't think you can get 2 alerts with everything the same, and only endsAt is different, but I don't see a downside in adding it
src/robusta/runner/web.py
Outdated
|
|
||
| Web.event_handler.get_telemetry().last_alert_at = str(datetime.now()) | ||
| return jsonify(success=True) | ||
|
|
||
| @staticmethod | ||
| def processed_alerts_cache_flusher(): |
There was a problem hiding this comment.
I think we don't need this.
I think the TTL Cache handles expiration
There was a problem hiding this comment.
As discussed, this is implemented to handle some worst case scenarios when the built-in expiration mechanism in TTLCache would be run and take a very long time. To guard against this, we run expire periodically to distribute the workload.
There was a problem hiding this comment.
Okay, upon further inspection and bencharking it looks like the "bulk expire", if it happens, in practice won't take so long to warrant having this mechanism implemented. Removed.
src/robusta/runner/web.py
Outdated
| @staticmethod | ||
| def init(event_handler: PlaybooksEventHandler, loader: ConfigLoader): | ||
| Web.metrics = QueueMetrics() | ||
| Web.api_server_queue = TaskQueue(name="api_server_queue", num_workers=NUM_EVENT_THREADS, metrics=Web.metrics) | ||
| Web.alerts_queue = TaskQueue(name="alerts_queue", num_workers=NUM_EVENT_THREADS, metrics=Web.metrics) | ||
| Web.event_handler = event_handler | ||
| Web.loader = loader | ||
| threading.Thread( |
There was a problem hiding this comment.
I think we don't need this
No description provided.