Compaction failed - access to data directory denied on Windows 7

[chester89]

Bug Report

What did you do?
Ran prometheus on a Windows machine with prometheus.exe --config.file=..\config\prometheus.yml --storage.tsdb.retention=16w --storage.tsdb.path=..\data --web.enable-admin-api
Prometheus is fetching data from local 2 instances of wmi_exporter
What did you expect to see?
No errors in the logs

What did you see instead? Under which circumstances?
Error in the logs about failed compaction, see attached log entries. Access to data subdirectory is denied, which is strange, I haven't set any specific access rights for it
Environment

• System information:

  Windows 7 Professional SP1

• Prometheus version:

  version=2.3.2, branch=HEAD, revision=71af5e29e815795e9dd14742ee7725682fa14b7b

• Prometheus configuration file:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.


scrape_configs:

  - job_name: 'win-local'

    scrape_interval: 15s

    static_configs:
      - targets: ['localhost:5015']	  

  - job_name: 'vs'
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:6016']
        labels:
          os: windows
          type: ide

• Logs:

level=error ts=2018-09-20T05:00:05.003853Z caller=db.go:272 component=tsdb msg="compaction failed" err="persist head block: rename block dir: rename E:\\db\\prometheus\\data\\01CQTNJWZ9M9NFRGXZYGN22KTQ.tmp E:\\db\\prometheus\\data\\01CQTNJWZ9M9NFRGXZYGN22KTQ: Access is denied."
level=error ts=2018-09-22T11:00:03.5533739Z caller=db.go:272 component=tsdb msg="compaction failed" err="persist head block: rename block dir: rename E:\\db\\prometheus\\data\\01CR0EZF9VWSCD7NYGKSFERHCJ.tmp E:\\db\\prometheus\\data\\01CR0EZF9VWSCD7NYGKSFERHCJ: Access is denied."

Issue #3253 sounds related, but it's not quite the same error message. I checked process explorer - no process other than Prometheus server is using the data directory. The whole data directory is read-only (not sure if that's intended)

[gouthamve]

Hi, this is 2.3.2 and we've done a bunch of fixes regarding Windows filehandling in 2.4.2. Could you upgrade and let us know if the problem persists?

[chester89]

sure, I'll update today and let you know

[chester89]

@gouthamve so far no errors. I'll let it run for a week just to be sure.
I also noticed snapshot API no longer works without a skip_head parameter (but it did in 2.3.2)

[gouthamve]

@chester89 Cool, I'll close this for now. Can re-open if the issue persists. Could you open a new issue for the snapshot API issue?

[chester89]

Sure, a bit later in the day

[chester89]

@gouthamve just filed #4671

[chester89]

@gouthamve I'm seeing the same errors:

level=error ts=2018-10-01T01:00:03.3966692Z caller=db.go:305 component=tsdb msg="compaction failed" err="persist head block: rename block dir: rename E:\\db\\prometheus\\data\\01CRPJ79X00AD1AF4FTSC0V9X2.tmp E:\\db\\prometheus\\data\\01CRPJ79X00AD1AF4FTSC0V9X2: Access is denied."
level=error ts=2018-10-01T01:00:04.4546692Z caller=db.go:305 component=tsdb msg="compaction failed" err="persist head block: rename block dir: rename E:\\db\\prometheus\\data\\01CRPJ7AXCBK532G6DQRA8NKGH.tmp E:\\db\\prometheus\\data\\01CRPJ7AXCBK532G6DQRA8NKGH: Access is denied."

[chester89]

@gouthamve I think you should reopen the issue

[chester89]

@gouthamve still the same problem on 2.7.1 running on Windows 10