-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve helm chart documentation for storage in 4.5+ #8524
Comments
This is my current configuration for s3. You can (and should) use the same bucket when using boltdb-shipper or tsdb
insecure:false just forces https afaik. To make sure your objects have encryption, enable it on the bucket |
Why is it that when I ask about the values.yaml file noone actually replies with an example for the values.yaml file? Anyhow, after two days of going bold trying to figure this out, here is how others can use the actual helm chart the way it was set up to be used. Found the value for loki:
# Set to false if you don't intend to set 'X-Scope-OrgID' header for the loki datasource
auth_enabled: true
# You don't need this if you are not using the auth header
querier:
# I am using one tenant id per cluster
# This config is for cluster level logging
# Another centralized logging cluster will use all tenant ids in header and multi `true`
# See https://grafana.com/docs/loki/latest/operations/multi-tenancy/
multi_tenant_queries_enabled: false
# S3 storage configuration
storage:
type: s3
bucketNames:
chunks: "<loki-bucketname>"
ruler: "<loki-bucketname>"
admin: "<loki-bucketname>"
s3:
# Not really sure why this is required at all but it works only if this is provided
s3: "s3://<loki-bucketname>"
# Endpoints: https://docs.aws.amazon.com/general/latest/gr/s3.html
endpoint: "s3.eu-west-1.amazonaws.com"
# AWS region of bucket
region: "eu-west-1"
# Secret access key for a user with bucket permissions
secretAccessKey: ""
# Access key for a user with bucket permissions
accessKeyId: ""
# Set to false, multiple posts about problems with it set to true
s3ForcePathStyle: false
# We want to use the HTTPS only endpoint so set to false
insecure: false
# Our bucket is SSE-3 encrypted
sse_encryption: true
# Timeouts etc
http_config: {} After getting the configuration to actually work, the loki datasource is available via apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasource-loki
labels:
grafana_datasource: "1"
data:
datasource.yaml: |-
apiVersion: 1
datasources:
- name: loki
type: loki
url: "http://loki-gateway"
# You only need this if you have auth enabled
jsonData:
httpHeaderName1: 'X-Scope-OrgID'
secureJsonData:
httpHeaderValue1: '<THE_TENANT_ID_FOR_CLUSTER>' As for grafana/promtail, this values.yaml configuration did the trick config:
clients:
- url: http://loki-gateway/loki/api/v1/push
# You only need this if you have auth enabled
tenant_id: "<THE_TENANT_ID_FOR_CLUSTER>" |
I've come to realize that for some reason I am only able to get it to work using the non-https only endpoint i.e |
Agreed that S3 documentation is lacking and there appears to be a conflict between the |
I agree that the documentation can be confusing. Currently, I don't know what ruler or admin is used for or why we need to set them. I haven't seen anything written into those buckets yet on my deployment. Also, when I deploy the grafana/loki Helm chart, it sets the default number of backend, read, and write pod replicas to 3. |
+1 |
+1 |
1 similar comment
+1 |
@sourcehawk's example was very useful, thank you! Here is what I ended up with as a series of ansible tasks deploying the helm charts (the
I hope this helps some people! ---
- name: Add grafana chart repo
kubernetes.core.helm_repository:
name: grafana
repo_url: "https://grafana.github.io/helm-charts"
- name: Deploy loki
vars:
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 100m
memory: 64Mi
kubernetes.core.helm:
name: loki
release_namespace: grafana-system
create_namespace: yes
chart_ref: grafana/loki
# https://github.com/grafana/loki/blob/main/production/helm/loki/values.yaml
# https://github.com/grafana/loki/issues/8524
values:
loki:
storage:
bucketNames:
chunks: "{{ b2_loki_bucket }}"
ruler: "{{ b2_loki_bucket }}"
admin: "{{ b2_loki_bucket }}"
type: s3
s3:
s3: "s3://{{ b2_loki_bucket }}"
endpoint: s3.eu-central-003.backblazeb2.com
region: eu-central-003
accessKeyId: "{{ b2_loki_key_id }}"
secretAccessKey: "{{ b2_loki_secret }}"
s3ForcePathStyle: true
insecure: false
# https://github.com/grafana/loki/issues/4613#issuecomment-1855200860
limits_config:
split_queries_by_interval: "1h"
query_scheduler:
max_outstanding_requests_per_tenant: 2048
gateway:
nginxConfig:
resolver: coredns.kube-system.svc.cluster.local
resources: "{{ resources }}"
monitoring:
lokiCanary:
resources:
limits:
cpu: 10m
memory: 64Mi
requests: {}
selfMonitoring:
grafanaAgent:
resources: "{{ resources }}"
backend:
persistence:
size: "{{ 10 * disk_multiplier|float }}Gi"
resources: "{{ resources }}"
read:
persistence:
size: "{{ 10 * disk_multiplier|float }}Gi"
resources: "{{ resources }}"
singleBinary:
persistence:
size: "{{ 10 * disk_multiplier|float }}Gi"
resources: "{{ resources }}"
write:
persistence:
size: "{{ 10 * disk_multiplier|float }}Gi"
resources: "{{ resources }}"
- name: Deploy promtail (collects logs and sends them to loki)
kubernetes.core.helm:
name: promtail
release_namespace: grafana-system
create_namespace: yes
chart_ref: grafana/promtail
# https://github.com/grafana/helm-charts/blob/main/charts/promtail/values.yaml
# https://github.com/grafana/loki/issues/8524
values:
config:
clients:
- url: http://loki-gateway/loki/api/v1/push
tenant_id: system
snippets:
# https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/#the-base-relabel_config-block
extraRelabelConfigs:
- regex: "(?P<tenant>bb|aa)-.*"
source_labels:
- "namespace"
action: replace
target_label: tenant
pipelineStages:
- cri: {}
- match:
selector: '{tenant=~".+"}'
stages:
- tenant:
label: "tenant"
- output:
source: message
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 50m
memory: 128Mi
# TODO: ADD CHART VERSIONS TO ALL OF THESE
- name: Deploy grafana
kubernetes.core.helm:
name: grafana
release_namespace: grafana-system
create_namespace: yes
chart_ref: grafana/grafana
# https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml
# https://github.com/grafana/loki/issues/8524
values:
adminUser: "{{ grafana_admin_username }}"
adminPassword: "{{ grafana_admin_password }}"
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 100m
memory: 128Mi
persistence:
size: "{{ 10 * disk_multiplier|float }}Gi"
datasources:
datasources.yaml:
apiVersion: 1
datasources:
- name: "Logs (System)"
type: loki
url: "http://loki-gateway.grafana-system.svc.cluster.local"
jsonData:
httpHeaderName1: 'X-Scope-OrgID'
secureJsonData:
httpHeaderValue1: 'system'
- name: "Logs (AA)"
type: loki
url: "http://loki-gateway.grafana-system.svc.cluster.local"
jsonData:
httpHeaderName1: 'X-Scope-OrgID'
secureJsonData:
httpHeaderValue1: 'aa'
- name: "Logs (BB)"
type: loki
url: "http://loki-gateway.grafana-system.svc.cluster.local"
jsonData:
httpHeaderName1: 'X-Scope-OrgID'
secureJsonData:
httpHeaderValue1: 'bb' |
This example was very helpful. our initial setup was missing the The endpoint had been configured as The format that works now is (
|
@pbsladek to configure the extra path, did you set it for the Update: To set up bucket subpath (path prefix), I had to append it to the loki:
storage:
bucketNames:
chunks: "{{ loki_bucket }}"
ruler: "{{ loki_bucket }}"
admin: "{{ loki_bucket }}"
type: s3
s3:
# s3: "s3://xxxxx-loki"
endpoint: s3.us-east-1.amazonaws.com/path/prefix
region: us-east-1
accessKeyId: "{{ loki_key_id }}"
secretAccessKey: "{{ loki_secret }}"
s3ForcePathStyle: true
insecure: false However, when I did so, I got an error about an invalid I also tried debugging the S3 endpoint / bucket / path combo with s3-proxy. s3-proxy allows to set up webhooks that are triggered on requests. The webhook event contains info about which S3 path was requested. I've set it up to send the webhook event to http-echo service so the S3 path data was printed to the logs. However, it seems s3-proxy doesn't handle multipart uploads, or there was some kind of error along those lines. So in my case I didn't manage to set the bucket subpath, and instead decided to just go with another bucket. |
Hello Folks, Welcome to the party 😄 I am using the latest helm_chart for To me, this looks like, three buckets are required, however admin bucket is optional if you are using Enterprise.
I think the second section, is semy optional. In my case, I am using a VPC s3 endpoint, with network routing.. So I don't touch the endpoint. However, I still have some doubts, about whether this is correct, and which S3 bucket I have to provide in
|
I am a bit lost regarding storage configuration for the latest grafana/loki helm chart. The default values may make it look like a simple task but that is not what I am experiencing. Just trying to configure an S3 bucket using the default values has been a guessing game...
storage.s3.s3
and what should the value of it be?insecure: false
?loki_cluster_seed.json
file but the writer is erroring with InvalidAccessPoint, what configuration is missing? (err="failed to flush chunks: store put chunk: InvalidAccessPoint: The specified accesspoint name or account is not valid
)What should be done?
The text was updated successfully, but these errors were encountered: