Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce overrides-exporter module to Loki #4520

Merged

Conversation

JordanRushing
Copy link
Contributor

@JordanRushing JordanRushing commented Oct 21, 2021

What this PR does / why we need it:
This PR introduces a new module to Loki that will export tenant limits including overrides to further expand visibility into tenant configurations and behavior.

Examples runtime-config with overrides:

overrides:
  "user1":
    ingestion_burst_size_mb: 350000
    ingestion_rate_mb: 350000
    max_query_series: 1000

Running Loki with only the overrides-exporter target specified and a basic schema_config:

$ ./cmd/loki/loki -config.file=demo_schema_config.yaml -target=overrides-exporter -runtime-config.file=example_runtime.yaml -server.http-listen-port=8080

Exported metrics:

$ curl -sq localhost:8080/metrics | grep override                                                                                                                                                                                                                          
# HELP loki_overrides Resource limit overrides applied to tenants                                                                                                                                                                                                          
# TYPE loki_overrides gauge                                                                                                                                                                                                                                                
loki_overrides{limit_name="cardinality_limit",user="user1"} 100000                                                                                                                                                                                                         
loki_overrides{limit_name="creation_grace_period",user="user1"} 6e+11                                                                                                                                                                                                      
loki_overrides{limit_name="ingestion_burst_size_mb",user="user1"} 350000                                                                                                                                                                                                   
loki_overrides{limit_name="ingestion_rate_mb",user="user1"} 350000                                                                                                                                                                                                         
loki_overrides{limit_name="max_cache_freshness_per_query",user="user1"} 6e+10                                                                                                                                                                                              
loki_overrides{limit_name="max_chunks_per_query",user="user1"} 2e+06                                                                                                                                                                                                       
loki_overrides{limit_name="max_concurrent_tail_requests",user="user1"} 10                                                                                                                                                                                                  
loki_overrides{limit_name="max_entries_limit_per_query",user="user1"} 5000                                                                                                                                                                                                 
loki_overrides{limit_name="max_global_streams_per_user",user="user1"} 5000                                                                                                                                                                                                 
loki_overrides{limit_name="max_label_name_length",user="user1"} 1024                                                                                                                                                                                                       
loki_overrides{limit_name="max_label_names_per_series",user="user1"} 30                                                                                                                                                                                                    
loki_overrides{limit_name="max_label_value_length",user="user1"} 2048                                                                                                                                                                                                      
loki_overrides{limit_name="max_line_size",user="user1"} 0                                                                                                                                                                                                                  
loki_overrides{limit_name="max_queriers_per_tenant",user="user1"} 0                                                                                                                                                                                                        
loki_overrides{limit_name="max_query_length",user="user1"} 2.5956e+15                                                                                                                                                                                                      
loki_overrides{limit_name="max_query_lookback",user="user1"} 0                                                                                                                                                                                                             
loki_overrides{limit_name="max_query_parallelism",user="user1"} 32                                                                                                                                                                                                         
loki_overrides{limit_name="max_query_series",user="user1"} 1000                                                                                                                                                                                                            
loki_overrides{limit_name="max_streams_matchers_per_query",user="user1"} 1000                                                                                                                                                                                              
loki_overrides{limit_name="max_streams_per_user",user="user1"} 0                                                                                                                                                                                                           
loki_overrides{limit_name="min_sharding_lookback",user="user1"} 0                                                                                                                                                                                                          
loki_overrides{limit_name="per_stream_rate_limit",user="user1"} 3.145728e+06                                                                                                                                                                                               
loki_overrides{limit_name="per_stream_rate_limit_burst",user="user1"} 1.572864e+07                                                                                                                                                                                         
loki_overrides{limit_name="per_tenant_override_period",user="user1"} 1e+10                                                                                                                                                                                                 
loki_overrides{limit_name="reject_old_samples_max_age",user="user1"} 1.2096e+15                                                                                                                                                                                            
loki_overrides{limit_name="retention_period",user="user1"} 2.6784e+15                                                                                                                                                                                                      
loki_overrides{limit_name="ruler_evaluation_delay_duration",user="user1"} 0                                                                                                                                                                                                
loki_overrides{limit_name="ruler_max_rule_groups_per_tenant",user="user1"} 0                                                                                                                                                                                               
loki_overrides{limit_name="ruler_max_rules_per_rule_group",user="user1"} 0                                                                                                                                                                                                 
loki_overrides{limit_name="ruler_remote_write_queue_batch_send_deadline",user="user1"} 0                                                                                                                                                                                   
loki_overrides{limit_name="ruler_remote_write_queue_capacity",user="user1"} 0                                                                                                                                                                                              
loki_overrides{limit_name="ruler_remote_write_queue_max_backoff",user="user1"} 0                                                                                                                                                                                           
loki_overrides{limit_name="ruler_remote_write_queue_max_samples_per_send",user="user1"} 0                                                                                                                                                                                  
loki_overrides{limit_name="ruler_remote_write_queue_max_shards",user="user1"} 0                                                                                                                                                                                            
loki_overrides{limit_name="ruler_remote_write_queue_min_backoff",user="user1"} 0                                                                                                                                                                                           
loki_overrides{limit_name="ruler_remote_write_queue_min_shards",user="user1"} 0                                                                                                                                                                                            
loki_overrides{limit_name="ruler_remote_write_timeout",user="user1"} 0                                                                                                                                                                                                     
loki_overrides{limit_name="split_queries_by_interval",user="user1"} 0 

Which issue(s) this PR fixes:
Closes #4389

Special notes for your reviewer:
N/A

Checklist

  • Documentation added
  • Tests updated

@JordanRushing JordanRushing changed the title Introduce overrides-exporter module from cortex into loki; add some t… Introduce overrides-exporter module to Loki Oct 21, 2021
@pull-request-size pull-request-size bot added size/L and removed size/M labels Oct 21, 2021
@JordanRushing JordanRushing linked an issue Oct 21, 2021 that may be closed by this pull request
@JordanRushing
Copy link
Contributor Author

@KMiller-Grafana QQ: any opinions on where to place new documentation for the overrides-exporter target for Loki? For reference here is the comparable doc for Cortex.

@JordanRushing JordanRushing self-assigned this Oct 21, 2021
@JordanRushing JordanRushing added this to In progress in Loki Project via automation Oct 21, 2021
@JordanRushing JordanRushing marked this pull request as ready for review October 21, 2021 21:55
@JordanRushing JordanRushing requested a review from a team as a code owner October 21, 2021 21:55
@JordanRushing JordanRushing changed the title Introduce overrides-exporter module to Loki Introduce overrides-exporter module to Loki Oct 21, 2021
Loki Project automation moved this from In progress to Reviewer approved Oct 21, 2021
Copy link
Contributor

@dannykopping dannykopping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! A couple small notes

pkg/validation/exporter.go Outdated Show resolved Hide resolved
pkg/validation/exporter_test.go Show resolved Hide resolved
@JordanRushing JordanRushing force-pushed the jordanrushing/loki/tenant-limit-exporter branch from ba1b254 to 55a37b3 Compare October 22, 2021 03:26
Copy link
Contributor

@dannykopping dannykopping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the reflection approach, personally. If security or cardinality becomes a concern, we could always add a denylist or allowlist mechanism to reduce them. I think the benefit of avoiding drift outweighs the potential risk here.

pkg/validation/exporter.go Show resolved Hide resolved
pkg/validation/exporter_test.go Show resolved Hide resolved
@owen-d owen-d merged commit ca67292 into grafana:main Oct 22, 2021
Loki Project automation moved this from Reviewer approved to Done Oct 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

Successfully merging this pull request may close these issues.

Add loki limits exporter
4 participants