Retention/Compacting not working

[3komma14]

What did you do?
Starting to use prometheus and is getting some disc space issues.
Retention is set but, but data directory is growing and growing

Command line

prometheus.exe --web.listen-address="localhost:8081" --storage.tsdb.retention=3h

What did you expect to see?
That the data directory would stop growing. Doesnt matter how low retention periode is used. Would expect a 1hr retention to keep the data directory small.

What did you see instead? Under which circumstances?
For a period the data directory size is stable, then it grows and grow. Errors are also starting to be logged.

After stopping it also wont restart unless you delete the data directory..

Environment

• System information:

OS Name: Microsoft Windows Server 2012 R2 Datacenter
OS Version: 6.3.9600 N/A Build 9600
OS Manufacturer: Microsoft Corporation
OS Configuration: Member Server
OS Build Type: Multiprocessor Free
System Manufacturer: Microsoft Corporation
System Model: Virtual Machine
System Type: x64-based PC
Processor(s): 1 Processor(s) Installed.
[01]: Intel64 Family 6 Model 62 Stepping 4 GenuineIntel ~2195 Mhz
BIOS Version: Microsoft Corporation Hyper-V UEFI Release v1.0, 26.11.2012

• Prometheus version:

prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98
4de5032d1a0)
build user: root@615b82cb36b6
build date: 20171108-07:19:19
go version: go1.9.2

• Alertmanager version:

not used

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
  - "nservicebus.rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'nservicebus'

    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:3030']
      
  - job_name: 'Server'

    scrape_interval: 5s

    static_configs:
      - targets: ['somehost:9182']

• Alertmanager configuration file:

not used

• Logs:

level=info ts=2017-11-14T07:54:17.9729243Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1510581600000 maxt=1510588800000
level=error ts=2017-11-14T07:54:18.6889275Z caller=db.go:260 component=tsdb msg="compaction failed" err="reload blocks: read meta information data\\01BYTGJFGXBJBYQBYBHCVA4VZS: open data\\01BYTGJFGXBJBYQBYBHCVA4VZS\\meta.json: The system cannot find the file specified."
level=error ts=2017-11-14T07:55:18.6974924Z caller=db.go:255 component=tsdb msg="retention cutoff failed" err="read block meta data\\01BYTGJFGXBJBYQBYBHCVA4VZS: open data\\01BYTGJFGXBJBYQBYBHCVA4VZS\\meta.json: The system cannot find the file specified."

Unable to restart

level=info ts=2017-11-14T08:30:10.6508625Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, revision=0a74f98628a0463dddc90528220c94de5032d1a0)"
level=info ts=2017-11-14T08:30:10.6618638Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:19:19)"
level=info ts=2017-11-14T08:30:10.6628631Z caller=main.go:217 host_details=(windows)
level=info ts=2017-11-14T08:30:10.6658619Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2017-11-14T08:30:10.6658619Z caller=web.go:380 component=web msg="Start listening for connections" address=localhost:8081
level=info ts=2017-11-14T08:30:10.6668618Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=error ts=2017-11-14T08:30:10.6918617Z caller=main.go:323 msg="Opening storage failed" err="read meta information data\\01BYTGJFGXBJBYQBYBHCVA4VZS: open data\\01BYTGJFGXBJBYQBYBHCVA4VZS\\meta.json: The system cannot find the file speci
fied."

[3komma14]

Think this issue #3487 is related.

[lukmanulhakimd]

Any update yet? I'm facing the same issue here. I ran the prometheus using flag --storage.tsdb.retention=3h, but prometheus seems to store data for more than 6 hours.

[gouthamve]

This should be fixed by This should be fixed by prometheus/tsdb#213 and #3508

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.