Is it possible to disable crash recovery?

[paragsomani]

What did you do?
I have a prometheus server deployed on openshift. We are monitoring different pods, kublets etc. Ingestion rate is ~10K/second.
Polling/scraping interval: 10s
Memory assigned: 8GiB
CPU: 6 cores
Disk type: NFS/HDD
storage.local.target-heap-size: 5.7GiB
retention period: 1h

With above config, prometheus frequently runs into rushed mode(with urgency score 1) and resulting less scraping of data. If we are trying to stop it(using SIGTERM) it takes too much time(in openshift grace period of shutdown is 15m), and still does not gets shutdown gracefully. This causes crash recovery of data which again causes delay in start for 15-20m.

Can you advice, whether i can disable with crash recovery of data, so that i can work with some loss of data?

What did you expect to see?
Disabling of crash recovery.
What did you see instead? Under which circumstances?

Environment
openshift

• System information:

openshift 3.5

• Prometheus version:
  v 1.8.2

• Alertmanager version:

NA

• Prometheus configuration file:

global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m
scrape_configs:
- job_name: kubernetes-nodes
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: node
    namespaces:
      names: []
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: false
  relabel_configs:
  - separator: ;
    regex: __meta_kubernetes_node_label_(.+)
    replacement: $1
    action: labelmap
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: kubernetes.default.svc:443
    action: replace
  - source_labels: [__meta_kubernetes_node_name]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics
    action: replace
- job_name: kubernetes-pods
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names:
      - XYZ
      - ABC
      - DEF
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: $1
    action: replace
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    separator: ;
    regex: ([^:]+)(?::\d+)?;(\d+)
    target_label: __address__
    replacement: $1:$2
    action: replace
  - separator: ;
    regex: __meta_kubernetes_pod_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: kubernetes_namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: kubernetes_pod_name
    replacement: $1
    action: replace
remote_write:
- url: http://prometheus-sop.ft-aramse:9096
  remote_timeout: 30s
  queue_config:
    capacity: 100000
    max_shards: 1000
    max_samples_per_send: 100
    batch_send_deadline: 5s
    max_retries: 10
    min_backoff: 30ms
    max_backoff: 100ms

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[jamtur01]

@brian-brazil I note there are a lot of these. Would it be worth adding some variant of this to the issue template?

[brian-brazil]

There's only a handful, and it's no longer relevant with 2.0.

[jamtur01]

@brian-brazil I meant some variant of...

"It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided."

[brian-brazil]

Ah, we already have something for that: https://raw.githubusercontent.com/prometheus/prometheus/master/.github/ISSUE_TEMPLATE.md

The rate of such questions has dropped since that went in, but they've never gone completely away.

[jamtur01]

Oh I see - it doesn't show up in the rendered version in the GitHub Web UI.

[brian-brazil]

Yeah, that's how you have it show up when an issue is being created but not in the issue itself.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.