Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Logging dashboards (Loki, Logs, Fluent-bit) #2745

Merged
merged 9 commits into from
Aug 25, 2020
Merged

Add Logging dashboards (Loki, Logs, Fluent-bit) #2745

merged 9 commits into from
Aug 25, 2020

Conversation

TeddyAndrieux
Copy link
Collaborator

@TeddyAndrieux TeddyAndrieux commented Aug 18, 2020 •
edited
Loading

Component:

'salt', 'build', 'logging'

Context:

#2721

Summary:

  • Import dashboard from Grafana Loki repository for Operational, update it to match MetalK8s deployment and deploy it with an SLS
  • Import dashboard from Grafana Loki repository for Logs, update it to match MetalK8s deployment and deploy it with an SLS
  • Import dashboard from Fluent repository for Fluent-bit

Fixes: #2721

@TeddyAndrieux TeddyAndrieux force-pushed the improvement/GH-2721-add-loki-dashboards branch from 10afdfc to fb88032 Compare August 18, 2020 11:13
@alexandre-allard alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from 96c2445 to a0acb6b Compare August 19, 2020 07:44
@TeddyAndrieux TeddyAndrieux force-pushed the improvement/GH-2721-add-loki-dashboards branch from fb88032 to f9aabd9 Compare August 19, 2020 08:12
@alexandre-allard alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from a0acb6b to 0765628 Compare August 20, 2020 07:38
@alexandre-allard
Copy link
Contributor

Pretty cool dashboards, just few things:

  • CPU & RAM usage graphs from the operational dashboard are weird (dots instead of curves).
  • Could it be possible to add the Loki volume in the operational dashboard?

@alexandre-allard alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from 16ea9b1 to be626c7 Compare August 20, 2020 16:53
@TeddyAndrieux TeddyAndrieux force-pushed the improvement/GH-2721-add-loki-dashboards branch 2 times, most recently from 25835e1 to ccd1898 Compare August 24, 2020 07:59
@alexandre-allard alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from be626c7 to c0a0a46 Compare August 24, 2020 08:33
@TeddyAndrieux TeddyAndrieux force-pushed the improvement/GH-2721-add-loki-dashboards branch from ccd1898 to 50d8903 Compare August 24, 2020 09:12
@TeddyAndrieux TeddyAndrieux changed the title Add Logging dashboards (Loki, Logs, (Fluent-bit?)) Add Logging dashboards (Loki, Logs, Fluent-bit) Aug 24, 2020
Base automatically changed from improvement/2701-deploy-fluent-bit-from-charts to development/2.6 August 24, 2020 11:03
@bert-e
Copy link
Contributor

bert-e commented Aug 24, 2020

Hello teddyandrieux,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

@bert-e
Copy link
Contributor

bert-e commented Aug 24, 2020

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • one peer

Peer approvals must include at least 1 approval from the following list:

@TeddyAndrieux TeddyAndrieux marked this pull request as ready for review August 24, 2020 11:04
@TeddyAndrieux TeddyAndrieux requested a review from a team as a code owner August 24, 2020 11:04
@TeddyAndrieux
charts: Import Loki dashboard from Grafana repos
abc2a98 
Take Loki operational dashboard from Grafana GitHub repository
https://github.com/grafana/loki/blob/fdf6208dd479c95777a82f29374acb26c0a0c51a/production/loki-mixin/dashboard-loki-operational.json

Refs: #2721
@TeddyAndrieux
charts: Update Loki dashboard from Grafana for MetalK8s
a52bed0 
We cannot use the Loki dashboard from Grafana repository as is, so some
tuning was needed to use this dashboard:

- We do not have Chunks, Memcached, Promtail, BigTable Backup, Consul
  dashboard, so remove links to these dashboards
- We do not use Memcached, Consul, Big Table, GCS, Dynamo, S3,
  Cassandra, so remove these from the dashboard
- We only have a single Loki Pod for Distributor + Ingester + Querier,
  so regroup panel when distinction do not make sense (e.g.: CPU, Ram
  consumption)
- Since we only have a single Pod for Distributor + Ingester + Querier,
  adapt the query from all panels to query this Pod instead of
  `$namespace/Distributor`, ...
- We only consider a single Loki cluster for MetalK8s deployment so hide
  the `cluster` templating from the dashboard
- We do not use PromTail so the `Bad Words` panel do not make sense,
  replace it with a `Total Logs/Second` panel
- Add links to Fluent-bit and Logs dashboards
- Refactor a bit the dashboard organization so that it looks great with
  the previous changes

Refs: #2721
@TeddyAndrieux
build,salt: Deploy Loki dashboard with Salt
8ba3889 
Create an SLS to deploy the Loki dashboard from charts

Refs: #2721
@TeddyAndrieux
charts: Import Logs dashboard from Grafana Loki repos
850aa32 
Take Loki logs dashboard from Grafana GitHub repository
https://github.com/grafana/loki/blob/a5aa34137a48b69ec2b005719e134fbeb68af6ec/production/loki-mixin/dashboard-loki-logs.json

Refs: #2721
@TeddyAndrieux
charts: Update Logs dashboard from Grafana for MetalK8s
63381fa 
We cannot use the Logs dashboard from Grafana loki repository as is, so
some tunning was needed to use this dashboard:

- We do not have `level` label for the moment, so remove this from all
  query and also the variable
- We have a label for node and ability to filter logs per node look like
  a great feature, so added it
- We do not use promtail so remove the graph related to promtail and
  instead have a graph for Memory usage on the selected containers
- We only consider a single Loki cluster for MetalK8s deployment so hide
  the `cluster` templating from the dashboard
- We also have system logs, so add 2 variables to enable, disable system
  logs and/or pods logs and adapt the query accordingly
  (NOTE: the method used to filter is a bit ugly, if system logs are
  disabled we query for logs with `node` label non-empty, because `node`
  label is empty on system logs, and we do the same with `hostname` for
  pod logs)
- Add links to Loki and Fluent-bit dashboards
- Refactor a bit the dashboard organization so that it looks great with
  the previous changes

Refs: #2721
@TeddyAndrieux
build,salt: Deploy Logs dashboard with Salt
7147bf9 
Create an SLS to deploy the Logs dashboard from charts

Refs: #2721
@TeddyAndrieux
charts: Import fluent-bit dashboard from Fluent repos
2fa7993 
Take fluent-bit dashboard from fluent GitHub repository
https://github.com/fluent/fluent-bit-docs/blob/bc0b2fac9276b9d12dc09ae1061705fa550a270b/monitoring/dashboard.json

Refs: #2721
@TeddyAndrieux
build,salt: Deploy Fluent-bit dashboard with Salt
3eacee2 
Create an SLS to deploy the Fluent-bit dashboard from charts

Refs: #2721
@TeddyAndrieux
charts: Update Fluent-bit dashboard from fluent for MetalK8s
2e18ee6 
Add link to Loki and Logs dashboards and add a templating variable for
`DS_PROMETHEUS`

Refs: #2721
@TeddyAndrieux TeddyAndrieux force-pushed the improvement/GH-2721-add-loki-dashboards branch from 50d8903 to 2e18ee6 Compare August 25, 2020 08:51
alexandre-allard
alexandre-allard approved these changes Aug 25, 2020
View reviewed changes
Copy link
Contributor

@alexandre-allard alexandre-allard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks really great!
Just one thing, there is too much selectors in the log dashboard, but I don't see a way of reducing this number. Maybe splitting system & pod logs ? But one may want to see both on the same page... IDK.
I think we can go with that and if we have feedback from TS, we could improve this later.

@TeddyAndrieux
Copy link
Collaborator Author

/approve

@bert-e
Copy link
Contributor

bert-e commented Aug 25, 2020

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

  • ✔️ development/2.6

The following branches will NOT be impacted:

  • development/1.0
  • development/1.1
  • development/1.2
  • development/1.3
  • development/2.0
  • development/2.1
  • development/2.2
  • development/2.3
  • development/2.4
  • development/2.5

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

  • Any commit you add on the source branch will trigger a new cycle after the
    current queue is merged.
  • Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

@bert-e
Copy link
Contributor

bert-e commented Aug 25, 2020

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/2.6

The following branches have NOT changed:

  • development/1.0
  • development/1.1
  • development/1.2
  • development/1.3
  • development/2.0
  • development/2.1
  • development/2.2
  • development/2.3
  • development/2.4
  • development/2.5

Please check the status of the associated issue GH-2721.

Goodbye teddyandrieux.

@bert-e bert-e merged commit 2e18ee6 into development/2.6 Aug 25, 2020
@bert-e bert-e deleted the improvement/GH-2721-add-loki-dashboards branch August 25, 2020 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexandre-allard alexandre-allard alexandre-allard approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Deploy Grafana dashboards for Loki
3 participants
@TeddyAndrieux @alexandre-allard @bert-e