Compact failed error in windows environment.

[arunsraga]

What did you do?
Running the prometheus in windows environment in command line.

What did you expect to see?
Error free info logs

What did you see instead? Under which circumstances?
Getting Error in console as follows.

**

level=error ts=2017-10-06T08:31:35.7785845Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write new tombstones file: rename store2\01BVRCZB664WNZ3XM2YQCKH5A3.tmp\tombstones.tmp store2\01BVRCZB664WNZ3XM2YQCKH5A3.tmp\tombstones: The process cannot access the file because it is being used by another process."

**

Environment

• System information:
  OS: Windows 10 Pro,
  version :1703,
  System type : 64 bit operating system, X64 based processor
  Installed Ram : 8 GB

• Prometheus version:
  version=2.0.0.beta5 and version=2.0.0.rc

• Alertmanager version:
  version=0.8.0

• Prometheus configuration file:

---
  global: 
    scrape_interval: "15s"
    evaluation_interval: "15s"
    external_labels: 
      monitor: "codelab-monitor"
  rule_files: 
    - "first.rules"
  alerting: 
    alertmanagers: 
      - 
        static_configs: 
          - 
            targets: 
              - "localhost:9093"
  scrape_configs: 
    - 
      job_name: "prometheus"
      static_configs: 
        - 
          targets: 
            - "localhost:9090"
    - 
      job_name: "wmi_exporter"
      static_configs: 
        - 
          targets: 
            - "localhost:9182"
    - 
      job_name: "tomcat_exporter"
      static_configs: 
        - 
          targets: 
            - "localhost:8080"
    - 
      job_name: "apache_sys_exporter"
      static_configs: 
        - 
          targets: 
            - "localhost:9117"
    - 
      job_name: "apache_grok_exporter"
      static_configs: 
        - 
          targets: 
            - "localhost:9144"
    - 
      job_name: "black_box_GET"
      scrape_interval: "30s"
      metrics_path: "/probe"
      params: 
        module: 
          - "http_2xx_example"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/api_jobs/59c9ef96f949ff3bb4efce33"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login_Page"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login_Page"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login_Page1"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login_Page1"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page2"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page2"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page3"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page3"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page5"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page5"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page6"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page6"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page7"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page7"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page8"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page8"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page9"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page9"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page10"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page10"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page11"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page11"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page12"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page12"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page13"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page13"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page14"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page14"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Login Page15"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Login Page15"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/login"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Logout"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Logout"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/logout"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Logout1"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Logout1"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/logout1"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Logout2"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Logout2"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/logout"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Logout3"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Logout3"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/logout"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Logout4"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Logout4"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/logout"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Logout5"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Logout5"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/logout"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"
    - 
      job_name: "Test_api"
      scrape_interval: "60s"
      metrics_path: "/probe"
      params: 
        module: 
          - "Test_api"
      static_configs: 
        - 
          targets: 
            - "http://localhost:3050/test_api"
      relabel_configs: 
        - 
          source_labels: 
            - "__address__"
          regex: "(.*)(:80)?"
          target_label: "__param_target"
          replacement: "${1}"
        - 
          source_labels: 
            - "__param_target"
          regex: "(.*)"
          target_label: "instance"
          replacement: "${1}"
        - 
          source_labels: []
          regex: ".*"
          target_label: "__address__"
          replacement: "192.168.2.128:9115"

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

   1) prometheus-2.0.0-beta.2.windows-amd64  
       prometheus.exe --config.file=prometheus.yml
       LOGS
     

>    ts=2017-10-05T13:22:18.6217556Z caller=db.go:283 msg="compaction failed" err="delete compacted head block: remove data\\01BVPAZNHNRQD3NAGHQA1J6MD6\\wal: The process cannot access the file because it is being used by another process."
>         ts=2017-10-05T13:22:18.6217556Z caller=db.go:288 msg="reloading blocks failed" err="read meta information data\\01BVPAZNHNRQD3NAGHQA1J6MD6: open data\\01BVPAZNHNRQD3NAGHQA1J6MD6\\meta.json: The system cannot find the file specified."
>         ts=2017-10-05T13:22:48.6207642Z caller=db.go:283 msg="compaction failed" err="delete compacted head block: remove data\\01BVPAZNHNRQD3NAGHQA1J6MD6\\wal: The process cannot access the file because it is being used by another process.
> 

2) version=2.0.0-beta.2
prometheus.exe --config.file=prometheus.yml --web.enable-lifecycle --storage.tsdb.min-block-duration=15m --storage.tsdb.path="store1/" --storage.tsdb.no-lockfile
INFO[0000] Starting prometheus (version=2.0.0-beta.2, branch=HEAD, revision=a52f082939a566d5269671e98be06fc6bdf61d09)  source="main.go:204"
INFO[0000] Build context (go=go1.8.3, user=root@41a0740ea598, date=20170818-08:25:08)  source="main.go:205"

> 
> ts=2017-10-06T05:08:34.26669Z caller=compact.go:327 msg="compact blocks" blocks=[01BVR0VRF6V6SFH3XDX2PCEA1W]
> ts=2017-10-06T05:08:34.4990538Z caller=db.go:283 msg="compaction failed" err="persist head block: write new tombstones file: rename store1\\01BVR1BKATKAA7SYJSBACX126A.tmp\\tombstones.tmp store1\\01BVR1BKATKAA7SYJSBACX126A.tmp\\tombstones: The process cannot access the file because it is being used by another process."
> ts=2017-10-06T05:09:04.2661736Z caller=compact.go:327 msg="compact blocks" blocks=[01BVR0VRF6V6SFH3XDX2PCEA1W]

3)  version=2.0.0.beta5

prometheus-2.0.0.beta5.exe --config.file=prometheus.yml --web.enable-lifecycle --storage.tsdb.min-block-duration=15m --storage.tsdb.path="store2/" --storage.tsdb.no-lockfile
level=info ts=2017-10-06T06:09:37.1530613Z caller=main.go:214 msg="Starting prometheus" version="(version=2.0.0-beta.5, branch=HEAD, revision=64c7932e851f3276391fc3e216cd30f447d4924e)"
level=info ts=2017-10-06T06:09:37.1530613Z caller=main.go:215 build_context="(go=go1.8.3, user=root@df40cc0bd79b, date=20170921-16:31:00)"
level=info ts=2017-10-06T06:09:37.154078Z caller=main.go:216 host_details=(windows)
level=info ts=2017-10-06T06:09:37.1550627Z caller=web.go:367 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2017-10-06T06:09:37.1560612Z caller=main.go:308 msg="Starting TSDB"
level=info ts=2017-10-06T06:09:37.1560612Z caller=targetmanager.go:68 component="target manager" msg="Starting target manager..."
level=info ts=2017-10-06T06:09:37.1620763Z caller=main.go:320 msg="TSDB started"
level=info ts=2017-10-06T06:09:37.1620763Z caller=main.go:387 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2017-10-06T06:09:37.1830704Z caller=main.go:364 msg="Server is ready to receive requests."

> ts=2017-10-06T06:22:30.4923064Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1507269600000 maxt=1507270500000
> ts=2017-10-06T06:22:30.6733098Z caller=db.go:259 component=tsdb msg="compaction failed" err="persist head block: write new tombstones file: rename store2\\01BVR5JZJWSBJH7RE841957Z4B.tmp\\tombstones.tmp store2\\01BVR5JZJWSBJH7RE841957Z4B.tmp\\tombstones: The process cannot access the file because it is being used by another process."
> ts=2017-10-06T06:22:31.6947519Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1507269600000 maxt=1507270500000

4) version=2.0.0.rc.exe
prometheus-2.0.0.rc.exe --config.file=prometheus.yml --web.enable-lifecycle --storage.tsdb.min-block-duration=15m --storage.tsdb.path="store2/" --storage.tsdb.no-lockfile
level=info ts=2017-10-06T06:25:59.178623Z caller=main.go:213 msg="Starting prometheus" version="(version=2.0.0-rc.0, branch=HEAD, revision=012e52e3f9a0c1741b160498615a29bd5d723028)"
level=info ts=2017-10-06T06:25:59.1796215Z caller=main.go:214 build_context="(go=go1.9.1, user=root@d94ce23e8b9b, date=20171005-14:50:29)"
level=info ts=2017-10-06T06:25:59.1806225Z caller=main.go:215 host_details=(windows)
level=info ts=2017-10-06T06:25:59.1816242Z caller=web.go:378 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2017-10-06T06:25:59.1816242Z caller=main.go:307 msg="Starting TSDB"
level=info ts=2017-10-06T06:25:59.1826233Z caller=targetmanager.go:68 component="target manager" msg="Starting target manager..."
level=info ts=2017-10-06T06:25:59.2816224Z caller=main.go:319 msg="TSDB started"
level=info ts=2017-10-06T06:25:59.2816224Z caller=main.go:386 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2017-10-06T06:25:59.2956323Z caller=main.go:363 msg="Server is ready to receive requests."

> level=info ts=2017-10-06T06:37:30.8277062Z caller=compact.go:360 component=tsdb msg="compact blocks" count=1 mint=1507270500000 maxt=1507271400000
> level=error ts=2017-10-06T06:37:30.9972906Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write new tombstones file: rename store2\\01BVR6EETBRVEK3EAMPNNEBHKQ.tmp\\tombstones.tmp store2\\01BVR6EETBRVEK3EAMPNNEBHKQ.tmp\\tombstones: The process cannot access the file because it is being used by another process."
> level=info ts=2017-10-06T06:37:32.0144174Z caller=compact.go:360 component=tsdb msg="compact blocks" count=1 mint=1507270500000 maxt=1507271400000
> level=error ts=2017-10-06T06:37:32.2444107Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write new tombstones file: rename store2\\01BVR6EFZEKTYEVXSNDPYWRD57.tmp\\tombstones.tmp store2\\01BVR6EFZEKTYEVXSNDPYWRD57.tmp\\tombstones: The process cannot access the file because it is being used by another process."
> level=info ts=2017-10-06T06:37:34.2447337Z caller=compact.go:360 component=tsdb msg="compact blocks" count=1 mint=1507270500000 maxt=1507271400000
> level=error ts=2017-10-06T06:37:34.3970651Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write new tombstones file: rename store2\\01BVR6EJ54866JQXW369R07R8W.tmp\\tombstones.tmp store2\\01BVR6EJ54866JQXW369R07R8W.tmp\\tombstones: The process cannot access the file because it is being used by another process."

I have checked in prometheus user group for same issue, i have not found any thing related to this so i raised same issue in google prometheus user group also.

Thank You
Arun Raga

[fabxc]

Thanks for reporting. According to the error the OS must be thinking multiple processes are using this file, which is rather odd.
Do you have anything else running interacting with the storage directory? What kind of disk/filesystem are you using?

[arunsraga]

Do you have anything else running interacting with the storage directory? What kind of disk/filesystem are you using?

1. I was suspecting the antivirus may be accessing the file system, i excluded the prometheus dir from antivirus scan it doesn't woked
2. I have cross checked the resource monitor, No other process accessing the "prometheus" working directory. Please find the image
   
   .
3. Disk type is HDD and File System is NTFS.

[fabxc]

@arunsraga did disabling the antivirus program work?

[ranbochen]

I have this problem too.
My server is Window2016_X64 with defender shutdown.
no other process access data folder.

[ranbochen]

I have found the cause of the problem: "tsdb" engine did not close TombstoneFile before rename it.
See: https://github.com/prometheus/tsdb/blob/f347eac33d3e0e9a8ab67cd12d7e99b58dd84155/tombstones.go#L43

func test() {
	path := "test_file"
	tmp := path + ".tmp"
	f, err := os.Create(tmp)
	if err != nil {
		panic(err)
	}
	defer f.Close()
	//f.Close() // open this comment will not fail
	err = renameFile(tmp, path)
	if err != nil {
		panic(err)
	}
}

[fabxc]

Awesome, thanks for digging through the code 💯 Would you like to a send a PR with a fix?

@gouthamve

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.