-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add docs for loki deployment, especially for separate mode #680
Conversation
e6343ef
to
ede06bc
Compare
any more comments? |
why the CI failed on building k3s.. |
@mizeng The k3s error is fixed in master. You can just merge from master and it should resolve it. |
@steven-sheehy merged from master and still failed |
But a different error ;) Any ideas @Kuqd @slim-bean ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi! This looks nice.
Some minor nits around readability and grammar.
@@ -0,0 +1,160 @@ | |||
# Overview | |||
This page lists the 2 common Loki deployments, to help user to better understand Loki architecture. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This page lists the 2 common Loki deployments, to help user to better understand Loki architecture. | |
This page lists the 2 common Loki deployments, to help users better understand the Loki architecture. |
# Overview | ||
This page lists the 2 common Loki deployments, to help user to better understand Loki architecture. | ||
|
||
Loki consists several components as below (Quote from slides [here](https://grafana.com/blog/2018/12/12/loki-prometheus-inspired-open-source-logging-for-cloud-natives/)): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Loki consists several components as below (Quote from slides [here](https://grafana.com/blog/2018/12/12/loki-prometheus-inspired-open-source-logging-for-cloud-natives/)): | |
Loki consists several components as below (taken from [those slides](https://grafana.com/blog/2018/12/12/loki-prometheus-inspired-open-source-logging-for-cloud-natives/)): |
|
||
|
||
# Loki Deployment on Single Process | ||
To those who want to just have a try at Loki, deploying Loki on `Single Process` is the easiest way. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Single Process Loki is not just for trying it out.
You can definitely run Loki in this mode in production and even horizontally scale it, just all components at once.
This might even be the easier approach to get it running in production
To those who want to just have a try at Loki, deploying Loki on `Single Process` is the easiest way. | ||
It combines all above component in a single process. | ||
|
||
[Run Locally Using Docker](https://github.com/grafana/loki/tree/master/production#run-locally-using-docker) and [Run from Source](https://github.com/grafana/loki/tree/master/production#build-and-run-from-source) have already gave the examples. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Run Locally Using Docker](https://github.com/grafana/loki/tree/master/production#run-locally-using-docker) and [Run from Source](https://github.com/grafana/loki/tree/master/production#build-and-run-from-source) have already gave the examples. | |
Refer to [Run Locally Using Docker](https://github.com/grafana/loki/tree/master/production#run-locally-using-docker) and [Run from Source](https://github.com/grafana/loki/tree/master/production#build-and-run-from-source) for examples. |
|
||
# Loki Deployment on Single Process | ||
To those who want to just have a try at Loki, deploying Loki on `Single Process` is the easiest way. | ||
It combines all above component in a single process. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It combines all above component in a single process. | |
It combines all of the above components in a single process. |
2. The Storage: | ||
|
||
You need to think about using some index store and chunk store which can access from all above components. | ||
For example, if you start above components in Kubernetes Pods, you may choose Amazon DynamoDB+S3, or choose boltdb+filesystem in a NFS storage like CephFS. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For example, if you start above components in Kubernetes Pods, you may choose Amazon DynamoDB+S3, or choose boltdb+filesystem in a NFS storage like CephFS. | |
For example, if you run on AWS, you may choose Amazon DynamoDB+S3, or if using Ceph/Rook boltdb+filesystem might be a good choice. |
3. The http_listen_port and grpc_listen_port: | ||
|
||
If you start above components separately in different Kubernetes Pod or in different VM/BM, you don't need to worry about it. | ||
If you start above components separately in your dev machine or one VM/BM, you need assign different port to them. Otherwise you will see errors like "address already in use". |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you start above components separately in your dev machine or one VM/BM, you need assign different port to them. Otherwise you will see errors like "address already in use". |
For example, if you start above components in Kubernetes Pods, you may choose Amazon DynamoDB+S3, or choose boltdb+filesystem in a NFS storage like CephFS. | ||
3. The http_listen_port and grpc_listen_port: | ||
|
||
If you start above components separately in different Kubernetes Pod or in different VM/BM, you don't need to worry about it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you start above components separately in different Kubernetes Pod or in different VM/BM, you don't need to worry about it. | |
If each of those components has its own IP address, you don't need to worry about it. Otherwise, you might see errors like `address already in use`. |
If you start above components separately in your dev machine or one VM/BM, you need assign different port to them. Otherwise you will see errors like "address already in use". | ||
4. The startup command args | ||
|
||
You need to specify the target for different component. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to specify the target for different component. | |
You need to select which component to run. |
4. The startup command args | ||
|
||
You need to specify the target for different component. | ||
For example, to start up Ingester only, using command like `./loki -config.file=/etc/loki/local-config.yaml -target=ingester` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For example, to start up Ingester only, using command like `./loki -config.file=/etc/loki/local-config.yaml -target=ingester` | |
For example, to start the Ingester only, use a command like `./loki -config.file=/etc/loki/local-config.yaml -target=ingester` |
please is it necessary to create a consul cluster ??? otherwise what if we have created a kubernetes service for each component deployment controller with a clusterip type ? |
@mizeng could you please share with me the whole configuration ? if it's possible ? |
Hi! For most up to date full configuration, please refer to our If you prefer not to generate the |
Yes, I am aware of ksonnet being deprecated, which is the reason Grafana built a replacement solution called Tanka, which can be used as a drop-in replacement: https://github.com/grafana/tanka. So yes, while it is named ksonnet, it will stay our state-of-the-art config for Loki! |
Also as soon as we get rid of Consul we will proposed a helm version that you can scale. Work in progress! |
@sh0rez , glad to hear that! So Tanka is fork/variant of ksonnet and will get long-term maintenance from grafana? @cyriltovena , cannot wait to see the scalable helm version :) |
Yes! Tanka is a rewrite from scratch, meant to replace ksonnet internally for us and for everyone else who might need it! |
Great! |
|
||
Below is the sample Loki Config for Separate Mode, please pay attention to the `ingester.lifecycler.ring.kvstore` part. | ||
```yaml | ||
uth_enabled: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo
close this one since there's a better doc commit merged. |
@mizeng your doc helped me tremendously and I wish you would re-open the PR. I think I disagree that a couple jsonnet files are a good substitute for documentation. Especially since it's yet another layer of things to learn or dig through in order to understand how to run the components individually. :) |
My feedback for #643.
Hope could help new beginners to understand how to deploy loki in separate mode.