[TE] ThirdEye doesn't update Anomalies after editing alert

[suvodeep-pyne]

ThirdEye still shows old detected anomalies after the 'alert' is updated.

Steps to reproduce:

1. Load a dataset in TE
2. Create an alert. Say PERCENTAGE_RULE so that it generates an anomaly
3. Update the alert. Example, Increase the threshold.
4. After successful update, the overview still shows the old anomalies, while the preview shows the new anomalies.

Example alert:

detectionName: pc
description: Capture percentage change in values.

# Tip: Type a few characters and look ahead (ctrl + space) to auto-fill.
metric: views
dataset: pageviews

# Configure multiple rules with "OR" relationship.
rules:
- detection:
  - name: detection_rule_1
    type: PERCENTAGE_RULE             # Configure the detection type here. See doc for more details.
    params:                     # The parameters for this rule. Different rules have different params.
      offset: wo1w
      percentageChange: 0.1 # Change the threshold to alter the set of anomalies.
      # pattern: down.  # Can be used to capture cases only when metric goes down.

[suvodeep-pyne]

@harleyjj @kishoreg
seems like a bug. thoughts?

In preview, it shows like this:

In Overview, it shows this:

[harleyjj]

@suvodeep-pyne This is behaving as designed. The preview tool's purpose is to show the user the anomalies it would have generated on historical data, so that the user can visualize the detection configuration and decide whether it behaves correctly for the user's specific use case (what is an anomaly to them?). In the case of an existing alert, it shows the new anomalies as compared to those saved in the database (the actual anomalies that were generated by the alert in the past). So the user can decide if the new detection configuration would perform better or worse than the alert's older detection configuration (i.e. Does it omit the false positives and/or detect the false negatives? Does the new detection still detect true positives? Does it detect false positives that the old config didn't?).

The Alert Overview's role is to show the detection as configured with the historical anomalies overlaid on it. The historical anomalies are actual real-time anomalies that were fired in the past, and alerts sent to any subscribed subscription groups.

[Jackie-Jiang]

Closing third-eye related issue as the project has been moved out of the pinot repo. For unsolved issue that still applies, please create a new one under third-eye repo: https://github.com/project-thirdeye/thirdeye