Alert history #3191

alexandre-allard · 2021-03-10T18:50:18Z

Component: images, build, salt

Context:

Summary:

Acceptance criteria:

Closes: #3180

bert-e · 2021-03-10T18:50:20Z

Hello alexandre-allard-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2021-03-10T18:50:24Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

bert-e · 2021-03-12T13:15:29Z

Branches have diverged

This pull request's source branch feature/alert-history has diverged from
development/2.9 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

Merge origin/development/2.9 into feature/alert-history
Rebase feature/alert-history onto origin/development/2.9

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.

bert-e · 2021-03-12T13:19:00Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge feature/alert-history into
development/2.8
w/2.9/feature/alert-history will be merged into development/2.9

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6
development/2.7

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2021-03-12T13:19:02Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

bert-e · 2021-03-17T10:59:11Z

History mismatch

Merge commit #f6cf8807ddc2744994944db8283b45725c082587 on the integration branch
w/2.9/feature/alert-history is merging a branch which is neither the current
branch feature/alert-history nor the development branch
development/2.9.

It is likely due to a rebase of the branch feature/alert-history and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

bert-e · 2021-03-17T11:00:01Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

bert-e · 2021-03-19T08:32:13Z

Branches have diverged

This pull request's source branch feature/alert-history has diverged from
development/2.9 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

Merge origin/development/2.9 into feature/alert-history
Rebase feature/alert-history onto origin/development/2.9

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.

bert-e · 2021-03-19T10:28:36Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

gdemonet

Mostly peripheral comments, some things should change in this PR though:

add the logger Go code to the linter
adjust the K8s manifests to use standard labels
adjust the test to check alert format (using the Watchdog "example")

docs/developer/architecture/alert-history.rst

gdemonet · 2021-03-24T10:10:43Z

docs/developer/architecture/alert-history.rst

+   As for now, there is no retention based on labels, streams, tenant or
+   whatever (on-going discussion `GH Loki #162`_).


We need to follow this closely, and maybe contribute. Having tenant-based retention could be really nice for separating different kinds of log streams (not only alerts, but also e.g. Salt events, audit logs, ...).

docs/developer/architecture/alert-history.rst

images/metalk8s-alert-logger/main.go

salt/metalk8s/addons/alert-logger/deployed/deployment.sls

gdemonet · 2021-03-25T08:19:59Z

salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls

+                Name        container
+                Format      regex
+                Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]+) (?<message>.+)$


I think this parser should always be used for containerd-formatted logs, no? And as such, likely this commit should be backported to the first version where Loki was introduced (2.6, I guess?)

Yes, I agree, we should probably backport it.
Will do it once this PR is merged.

PR opened here #3276

tests/post/steps/test_logging.py

bert-e · 2021-03-25T08:25:18Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@gdemonet

bert-e · 2021-04-01T15:18:11Z

Branches have diverged

This pull request's source branch feature/alert-history has diverged from
development/2.9 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

Merge origin/development/2.9 into feature/alert-history
Rebase feature/alert-history onto origin/development/2.9

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.

Move all `.uml` to a `diagrams` directory, so the directory only contains architecture document which is cleaner.

bert-e · 2021-04-06T07:59:33Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@gdemonet

bert-e · 2021-04-06T07:59:37Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@gdemonet

gdemonet

Some relatively minor comments, feel free to put them in a debt ticket.

images/metalk8s-alert-logger/main.go

images/metalk8s-alert-logger/Dockerfile

bert-e · 2021-04-06T08:11:52Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

This is a simple HTTP server, listening on a port (default to 19094), waiting for HTTP post request from alertmanager. It then logs the content of these requests to stdout. Refs: #3180

This container image is used for alert history. Refs: #3180

Refs: #3180

Remove the date from the logs once parsed by fluent-bit as we do not need it anymore and it allows to improve the logs readability. Plus, it is also need by the alert history feature, this way we only end up with a JSON formatted alert in logs which makes it easier to parse/use by other components. Refs: #3180

We now send all the alerts to the alert-logger receiver. Refs: #3180

We also need to enable CORS to allow Cross Origin requests coming from the web UI. Refs: #3180

This scenario checks that we can retrieve the cluster alerts from the Loki API. Refs: #3180

Refs: #3180

We were passing a CmdAction object to target actions, so doit was not doing anything. We now pass the execute method of the object.

We need a more recent version of plantuml to be able to build alert-history diagram and this version is not shipped in ubuntu:18.04

There is an issue with mktexpk not being able to create a directory, so wo create these fonts prior to launch the build. Also fix an issue by always changing ownership of doc build artifacts to avoid having root owned files when doc build fails.

alexandre-allard · 2021-04-06T18:34:12Z

/approve

bert-e · 2021-04-06T18:34:17Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following options are set: approve

gdemonet

Don't forget about https://github.com/scality/metalk8s/pull/3191/files#r601234682 😉

bert-e · 2021-04-07T07:03:11Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.9

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6
development/2.7
development/2.8

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2021-04-07T15:36:40Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.9

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6
development/2.7
development/2.8

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.

alexandre-allard force-pushed the feature/alert-history branch 2 times, most recently from 80bd91d to 3873910 Compare March 12, 2021 13:15

This comment has been minimized.

Sign in to view

alexandre-allard changed the base branch from development/2.9 to development/2.8 March 12, 2021 13:18

alexandre-allard force-pushed the feature/alert-history branch from 3873910 to 1a333a3 Compare March 17, 2021 10:59

alexandre-allard changed the base branch from development/2.8 to development/2.9 March 17, 2021 10:59

alexandre-allard marked this pull request as ready for review March 19, 2021 08:32

alexandre-allard requested a review from a team as a code owner March 19, 2021 08:32

alexandre-allard force-pushed the feature/alert-history branch from 1a333a3 to 10f8ed3 Compare March 19, 2021 10:28

alexandre-allard force-pushed the feature/alert-history branch 4 times, most recently from c6320f9 to d196d3c Compare March 19, 2021 16:59

gdemonet suggested changes Mar 25, 2021

View reviewed changes

alexandre-allard force-pushed the feature/alert-history branch 2 times, most recently from 54a973f to 4af620b Compare March 26, 2021 12:55

alexandre-allard force-pushed the feature/alert-history branch from c11fa8b to b660dcb Compare April 1, 2021 15:50

docs: Move architecture diagrams into sub directory

3471561

Move all `.uml` to a `diagrams` directory, so the directory only contains architecture document which is cleaner.

gdemonet previously approved these changes Apr 6, 2021

View reviewed changes

images/metalk8s-alert-logger/main.go Outdated Show resolved Hide resolved

images/metalk8s-alert-logger/Dockerfile Outdated Show resolved Hide resolved

alexandre-allard dismissed gdemonet’s stale review via 864d947 April 6, 2021 09:01

alexandre-allard force-pushed the feature/alert-history branch from 1dd396f to 864d947 Compare April 6, 2021 09:01

gdemonet previously approved these changes Apr 6, 2021

View reviewed changes

alexandre-allard added 14 commits April 6, 2021 11:32

images: Add metalk8s-alert-logger

0db7354

This is a simple HTTP server, listening on a port (default to 19094), waiting for HTTP post request from alertmanager. It then logs the content of these requests to stdout. Refs: #3180

images: Add metalk8s-alert-logger container image

91f3eb3

This container image is used for alert history. Refs: #3180

build: Add build of metalk8s-alert-logger container image

acc2e53

Refs: #3180

salt: Add metalk8s-alert-logger deployment

4a23e50

Refs: #3180

salt: Update alertmanager default configuration

e0e7e11

We now send all the alerts to the alert-logger receiver. Refs: #3180

salt: Expose Loki API to metalk8s-ui

42fe9a6

We also need to enable CORS to allow Cross Origin requests coming from the web UI. Refs: #3180

tests: Add a scenario for alert history feature

3fd62bc

This scenario checks that we can retrieve the cluster alerts from the Loki API. Refs: #3180

Add CHANGELOG entry for alert history feature

2f41e1c

Refs: #3180

build: Fix format:go target

ba95694

We were passing a CmdAction object to target actions, so doit was not doing anything. We now pass the execute method of the object.

build: Add go linting on every go file

f5ede68

lint: Apply go format on all go files

85f7125

docs: Use ubuntu:20.04 to build doc

bf410da

We need a more recent version of plantuml to be able to build alert-history diagram and this version is not shipped in ubuntu:18.04

alexandre-allard dismissed gdemonet’s stale review via b7d65df April 6, 2021 09:44

alexandre-allard force-pushed the feature/alert-history branch from 864d947 to b7d65df Compare April 6, 2021 09:44

gdemonet approved these changes Apr 7, 2021

View reviewed changes

bert-e merged commit b92fb8c into development/2.9 Apr 7, 2021

bert-e deleted the feature/alert-history branch April 7, 2021 15:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alert history #3191

Alert history #3191

alexandre-allard commented Mar 10, 2021

bert-e commented Mar 10, 2021

bert-e commented Mar 10, 2021

bert-e commented Mar 12, 2021

This comment has been minimized.

This comment has been minimized.

bert-e commented Mar 12, 2021

bert-e commented Mar 12, 2021

bert-e commented Mar 17, 2021

bert-e commented Mar 17, 2021

bert-e commented Mar 19, 2021

bert-e commented Mar 19, 2021

gdemonet left a comment

gdemonet Mar 24, 2021

gdemonet Mar 25, 2021

alexandre-allard Mar 25, 2021 •

edited

Loading

alexandre-allard Apr 7, 2021

bert-e commented Mar 25, 2021

bert-e commented Apr 1, 2021

bert-e commented Apr 6, 2021

bert-e commented Apr 6, 2021

gdemonet left a comment

bert-e commented Apr 6, 2021

alexandre-allard commented Apr 6, 2021

bert-e commented Apr 6, 2021

gdemonet left a comment

bert-e commented Apr 7, 2021

bert-e commented Apr 7, 2021

		As for now, there is no retention based on labels, streams, tenant or
		whatever (on-going discussion `GH Loki #162`_).

Alert history #3191

Alert history #3191

Conversation

alexandre-allard commented Mar 10, 2021

bert-e commented Mar 10, 2021

Hello alexandre-allard-scality,

bert-e commented Mar 10, 2021

Waiting for approval

bert-e commented Mar 12, 2021

Branches have diverged

This comment has been minimized.

This comment has been minimized.

bert-e commented Mar 12, 2021

Integration data created

bert-e commented Mar 12, 2021

Waiting for approval

bert-e commented Mar 17, 2021

History mismatch

bert-e commented Mar 17, 2021

Waiting for approval

bert-e commented Mar 19, 2021

Branches have diverged

bert-e commented Mar 19, 2021

Waiting for approval

gdemonet left a comment

Choose a reason for hiding this comment

gdemonet Mar 24, 2021

Choose a reason for hiding this comment

gdemonet Mar 25, 2021

Choose a reason for hiding this comment

alexandre-allard Mar 25, 2021 • edited Loading

Choose a reason for hiding this comment

alexandre-allard Apr 7, 2021

Choose a reason for hiding this comment

bert-e commented Mar 25, 2021

Waiting for approval

bert-e commented Apr 1, 2021

Branches have diverged

bert-e commented Apr 6, 2021

Waiting for approval

bert-e commented Apr 6, 2021

Waiting for approval

gdemonet left a comment

Choose a reason for hiding this comment

bert-e commented Apr 6, 2021

Waiting for approval

alexandre-allard commented Apr 6, 2021

bert-e commented Apr 6, 2021

Waiting for approval

gdemonet left a comment

Choose a reason for hiding this comment

bert-e commented Apr 7, 2021

In the queue

bert-e commented Apr 7, 2021

alexandre-allard Mar 25, 2021 •

edited

Loading