tail/tail.go openSegment makes assumptions about order of results from ReadDir

[jmacd]

The body of this function reads:

func openSegment(dir string, n int) (io.ReadCloser, error) {
	files, err := fileutil.ReadDir(dir)
	if err != nil {
		return nil, err
	}
	for _, fn := range files {
		k, err := strconv.Atoi(fn)
		if err != nil || k < n {
			continue
		}
		if k > n {
			return nil, errors.Errorf("next segment %d too high, expected %d", n, k)
		}
		return wal.OpenReadSegment(filepath.Join(dir, fn))
	}
	return nil, tsdb.ErrNotFound
}

The purpose of this function is to find a filename with the matching number of unspecified width. The fix is to treat k != n as a non-error (i.e., continue).

[jmacd]

We've seen error messages in 0.9.0 debug logs, e.g.,

err="corruption after 706478080 bytes: read first header byte: open next segment: next segment 47852 too high, expected 47893"

[jmacd]

Note there is a request to make this repository an official fork of the upstream https://github.com/Stackdriver/stackdriver-prometheus-sidecar. It would make it easier to share this sort of fix. (Stackdriver note: There are a couple of related fixes in the same tail/tail.go file also worth pulling upstream.)