Need more tsdb log

[tinytub]

What did you do?

stop graceful prometheus or killed hard prometheus

What did you expect to see?

Want to see more tsdb starting logs
Whatever stop graceful or killed hard, tsdb start will spend long time (1 minute - 10 minute) or hang at starting tsdb log. and I cannot figure out what cause the long time start.

What did you see instead? Under which circumstances?

time="2017-09-18T14:47:08+08:00" level=info msg="Starting tsdb" source="main.go:224"
time="2017-09-18T14:48:19+08:00" level=info msg="tsdb started" source="main.go:230"

Environment

• System information:

Linux 2.6.32-696.3.2.el6.v6.2.x86_64 x86_64

• Prometheus version:

prometheus, version 2.0.0-beta.4 (branch: HEAD, revision: 1b80f63)
build user: root@c5b3f10ffc51
build date: 20170914-11:22:50
go version: go1.8.3

• Prometheus configuration file:

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
      monitor: 'codelab-monitor'
      slave: 1

rule_files:
  - "alert_rules/*" 
  - "query_rules/*"

scrape_configs:
  - job_name: 'app-exporter'
    scrape_interval:     30s
    static_configs:
#with a lot of static server targets

  - job_name: 'qm'
    scrape_interval: 5s
    file_sd_configs:
      - files: ['/usr/local/prometheus/qmgroups/*.json']


  - job_name: 'kubernetes-controller-manager'
    scheme: http
    static_configs:
      - targets:
          - 'a.net:10252'
    relabel_configs:
      - target_label: hulkid
        replacement: 2936

  - job_name: 'kubernetes-scheduler'
    scheme: http
    static_configs:
      - targets:
          - 'a.net:10251'


  - job_name: 'kubernetes-apiserver'
    scheme: https
    kubernetes_sd_configs:
       - api_server: ['b.net:6443']
        role: endpoints
        tls_config:
          ca_file: /usr/local/prometheus/ssl/ca.pem
          cert_file: /usr/local/prometheus/ssl/kube-admin.pem
          key_file:  /usr/local/prometheus/ssl/kube-admin-key.pem
    tls_config:
      ca_file: /usr/local/prometheus/ssl/ca.pem
      cert_file: /usr/local/prometheus/ssl/kube-admin.pem
      key_file:  /usr/local/prometheus/ssl/kube-admin-key.pem
      insecure_skip_verify: true
    relabel_configs:
    - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
      action: keep
      regex: default;kubernetes;https

alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets:
      - "alertmanager-1:9093"
      - "alertmanager-2:9093"

[gouthamve]

Hi, on startup we read the write ahead log (WAL) and that could several be Gigs in size which could take several mins. But that is essentially the only thing that happens on startup so I am not sure if we should print anything else.

[fabxc]

@tinytub this should've been a deadlock in TSDB (fixed in prometheus-junkyard/tsdb#146). More logging would generally not help as @gouthamve pointed out but #3186 should improve debugging capabilities if such things come up in the future.

[tinytub]

@fabxc thx a lot, hope for next prerelease

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.