Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snapshot Management sends notification false positive #720

Open
DominikMoelter opened this issue Jan 26, 2023 · 1 comment
Open

Snapshot Management sends notification false positive #720

DominikMoelter opened this issue Jan 26, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@DominikMoelter
Copy link

I'm using a snapshot policy which was created via Opensearch Dashboards that backups a bunch indices. If a policy run fails I want to get a email notification. The policy runs every morning. The status of every run was successfull but I receive every morning a mail regarding to a failed policy run.

Mail:
image

Policy with "GET _plugins/_sm/policies/" in Dev Tools:

{
  "_id": "<replaced>-sm-policy",
  "_version": 2,
  "_seq_no": 28845332,
  "_primary_term": 869,
  "sm_policy": {
    "name": "<replaced>",
    "description": "",
    "schema_version": 17,
    "creation": {
      "schedule": {
        "cron": {
          "expression": "0 4 * * *",
          "timezone": "UTC"
        }
      }
    },
    "deletion": {
      "schedule": {
        "cron": {
          "expression": "0 4 * * *",
          "timezone": "UTC"
        }
      },
      "condition": {
        "max_age": "8d",
        "min_count": 5
      }
    },
    "snapshot_config": {
      "indices": "<replaced>",
      "repository": "minio"
    },
    "schedule": {
      "interval": {
        "start_time": 1668774573304,
        "period": 1,
        "unit": "Minutes"
      }
    },
    "enabled": true,
    "last_updated_time": 1668774945727,
    "enabled_time": 1668774573304,
    "notification": {
      "channel": {
        "id": "L0lLioQB70SDhg8KcAJe"
      },
      "conditions": {
        "creation": false,
        "deletion": false,
        "failure": true,
        "time_limit_exceeded": false
      }
    }
  }
}

Policy Explain with "GET _plugins/_sm/policies//_explain" in Dev Tools:

{
  "policies": [
    {
      "name": "<replaced>",
      "creation": {
        "current_state": "CREATION_START",
        "trigger": {
          "time": 1674792000000
        },
        "latest_execution": {
          "status": "SUCCESS",
          "start_time": 1674705633482,
          "end_time": 1674708933420,
          "info": {
            "message": "[2023-01-26T04:55:33Z]: Snapshot <replaced>-2023-01-26t04:00:33-kul920n5 creation has finished successfully."
          }
        }
      },
      "deletion": {
        "current_state": "DELETION_START",
        "trigger": {
          "time": 1674792000000
        },
        "latest_execution": {
          "status": "SUCCESS",
          "start_time": 1674708930014,
          "end_time": 1674708933619,
          "info": {
            "message": "[2023-01-26T04:55:33Z]: Snapshot(s) [<replaced>-2023-01-18t04:00:33-taqr9s94] deletion has finished."
          }
        }
      },
      "policy_seq_no": 28845332,
      "policy_primary_term": 869,
      "enabled": true
    }
  ]
}

As you can see in the policy explain the both creation of new snapshot and deletion of new snapshot were successfull.
Any ideas why the mail is nevertheless sent?

@DominikMoelter
Copy link
Author

Also checked the logs as mentioned in the mail. Log looks OK.

[2023-01-26T04:54:45,745][INFO ][o.o.s.SnapshotsService ] [os-master-1] snapshot [minio:-2023-01-26t04:00:33-kul920n5/-a5NFzGBSiyzNUmKHPYpZw] completed with state [SUCCESS]

@CEHENKLE CEHENKLE transferred this issue from opensearch-project/OpenSearch Mar 28, 2023
@Angie-Zhang Angie-Zhang added bug Something isn't working and removed untriaged labels Mar 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants