[Lightning]: Panic on invalid Mertic countter operation

[hi-rustin]

Bug Report

Version: 6.1.0

Lightning panic on https://github.com/pingcap/tidb/blob/master/br/pkg/lightning/restore/restore.go#L2333

I think it's because float64(currOffset - startOffset) < 0.

panic: counter cannot decrease in value

goroutine 202374 [running]:
github.com/prometheus/client_golang/prometheus.(*counter).Add(0x0?, 0xc069674c80?)
/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/counter.go:109 +0xf4
github.com/pingcap/tidb/br/pkg/lightning/restore.(*chunkRestore).deliverLoop(0xc0618805a0,
{0x445dbd8, 0xc0017c6040}

, 0xc06c80a3c0, 0xc0012af740, 0x3, 0xc002a8ff18?, 0xc002a8ffd0?, 0xc0016e5080)
/mnt/tidb/sql/br/pkg/lightning/restore/restore.go:2255 +0x7a3
github.com/pingcap/tidb/br/pkg/lightning/restore.(*chunkRestore).restore.func1()
/mnt/tidb/sql/br/pkg/lightning/restore/restore.go:2509 +0xab
created by github.com/pingcap/tidb/br/pkg/lightning/restore.(*chunkRestore).restore
/mnt/tidb/sql/br/pkg/lightning/restore/restore.go:2507 +0x31b

[D3Hunter]

value of currOffset comes from parser.pos which increase monotonically. the init value of parser.pos comes from chunk.Chunk.Offset. so it should happen that currOffset - startOffset < 0

need to reproduce it.

[D3Hunter]

user changed lightning code, cannot find code related to hash b3ee7aef716e39dd5c6744e4dac0fbbb3529aca7

not sure the root cause is from 6.1 code base or from user's customization

[D3Hunter]

update: the error happens when importing fixed-width text format file, not csv but pretend it as csv. in the datafile there're binary data which contains csv terminator which causes error.

cannot reproduce it now.

[lance6716]

Reproduced in a ONCALL ticket

[lance6716]

I printed newOffset and cr.chunk.Chunk.EndOffset

tidb/lightning/pkg/importer/chunk_process.go

Line 487 in 07508df

if kvSize >= minDeliverBytes || len(kvPacket) >= maxKvPairsCnt || newOffset == cr.chunk.Chunk.EndOffset {

there's an off-by-1 error, the chunk reader does not stop

[2024/05/21 15:55:36.744 +08:00] [INFO] [chunk_process.go:485] ["lance test3"] [table=`test`.`gla_glis_quarter`] [engineNumber=0] [fileIndex=7] [path=test.gla_glis_quarter.csv:469762831] [offset=536871695] [EndOffset=536871696]
...
[2024/05/21 15:55:36.744 +08:00] [INFO] [chunk_process.go:485] ["lance test3"] [table=`test`.`gla_glis_quarter`] [engineNumber=0] [fileIndex=7] [path=test.gla_glis_quarter.csv:469762831] [offset=536871902] [EndOffset=536871696]

[lance6716]

root cause:

For data with lint terminator \r\n and don't set terminator in configuration and strict-format = true, we try to seek every 256MiB and read the next line terminator as the chunk end position

tidb/pkg/lightning/mydump/region.go

Lines 455 to 458 in aeafb18

	if err = parser.SetPos(endOffset, 0); err != nil {
	return nil, nil, err
	}
	_, pos, err := parser.ReadUntilTerminator()

But unfortunately we seek to \r and ReadUntilTerminator then found \n. So end position of a chunk is set to \n. But elsewhere we use \r as the position. This causes the off-by-1 error

workaround is, set terminator to '\r\n' in configuration. ReadUntilTerminator will work fine because it knows \n is not terminator.

[seiya-annie]

/found customer