fix wal panic when page flush fails.

[krasi-georgiev]

fixes: prometheus/prometheus#3283

New records should be added to the page only when the last flush
succeeded. Otherwise the page would be full and panics when trying to
add a new record.

to replicate the bug:

dd if=/dev/zero of=tmp/data.img bs=1M count=20 // create a 20mb disk image
mkfs.ext4 /tmp/data.img
mkdir /tmp/prometheus/
sudo mount -t ext4 -o loop /tmp/data.img /tmp/prometheus/
GO111MODULE=on go run cmd/prometheus/main.go   --config.file=.local/simpleMany.yaml --storage.tsdb.path=/tmp/prometheus   --storage.tsdb.wal-segment-size=1MB

before the fix this will panic when the disk image is full because it is trying to add more records to the page when it is already full.

At the moment can't think of simple test to test for this, but will add it to https://github.com/prometheus/tsdb/issues/579 if we figure out how to inject file system faults.

Signed-off-by: Krasi Georgiev kgeorgie@redhat.com

[bwplotka]

I think it makes sense, but not fully approving, as it's the first time I see this code. Some suggestions though. Let me know if I need to really really dive into this flow.

[bwplotka]

Line 437: What if the clear is true, and we do Write and n is not actually p.alloc? I think we would then drop some bytes, right? Now is the question if the implementation of segment.Write does partial writes in any case (n). Worth to check?

[bwplotka]

I know it's not your bug/issue but: https://www.oreilly.com/library/view/97-things-every/9780596809515/ch08.html

[krasi-georgiev]

I think we would then drop some bytes, right?

Not sure what you mean by this. Can you add more details.

n, err := w.segment.Write(p.buf[p.flushed:p.alloc])
	if err != nil {
		return err
	}
	p.flushed += n

When Write returns an error tsdb will exit(leaving the WAL in corrupted state) and will run a repair on the next start up.

[krasi-georgiev]

ping @bwplotka

[gouthamve]

LGTM!

[krasi-georgiev]

merging as is.
@bwplotka if you still think there is a problem please let me know and I will open another PR.