Failed to downgrade from `2.1.0` to `2.0.0`

[FUSAKLA]

What did you do?
I tried to downgrade back to 2.0.0 from 2.1.0

What did you expect to see?
No problem with rollback because there was no mention about storage incompatibility in 2.1.0 release notes.

What did you see instead? Under which circumstances?
Downgrade failed.

Environment

Running in kubernetes with data mounted on host machine.

Docker official upstream images:

prom/prometheus:v2.1.0

prom/prometheus:v2.0.0

---

• System information:

  Linux 4.13.0-25-generic x86_64

• Prometheus version:

prometheus, version 2.1.0 (branch: HEAD, revision: 85f23d82a045d103ea7f3c89a91fba4a93e6367a)
  build user:       root@6e784304d3ff
  build date:       20180119-12:01:23
  go version:       go1.9.2

prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2

• Logs:

level=info ts=2018-01-24T17:32:57.106618248Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2018-01-24T17:32:57.106646142Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=error ts=2018-01-24T17:32:57.187507378Z caller=main.go:323 msg="Opening storage failed" err="read meta information /prometheus/01C4MHVPYRN7THY4KS3KAZP1A5: unexpected meta file version 2"

the metafile.json contains:

{
	"ulid": "01C4MHVPYRN7THY4KS3KAZP1A5",
	"minTime": 1516802400000,
	"maxTime": 1516809600000,
	"stats": {
		"numSamples": 13784198,
		"numSeries": 42773,
		"numChunks": 117088
	},
	"compaction": {
		"level": 1,
		"sources": [
			"01C4MHVPYRN7THY4KS3KAZP1A5"
		]
	},
	"version": 2
}

[brancz]

This is because of a storage version bump, which obviously v2.0.0 is not aware of.

/cc @gouthamve any idea if this could be solvable? In general I'd say snapshot & backup in case you need to rollback, but maybe you can think of something better.

[gouthamve]

Well I don't have a good answer for that. So we bumped version to make sure we support index files larger than 4GiB which 2.0.0 cannot support. The only way I see now is to delete the version 2 blocks and then start prometheus 2.0.0 but that will have some dataloss.

[FUSAKLA]

Maybe it should be empathized in the release notes i think this is pretty important note.

[gouthamve]

I totally agree and it has been an oversight from my side. I updated the notes now to reflect the same. Sorry for that.

[FUSAKLA]

I hope no one bumped into this in production

Thanks for the update I think we can close this if there is no better solution but deletion of data mentioned above.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.