We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The current implementation of reading rows from RowContainer is not fast enough. The following function is used to dump a chunk:
RowContainer
// GetRowAndAppendToChunk gets a Row from the ListInDisk by RowPtr. Return the Row and the Ref Chunk. func (l *ListInDisk) GetRowAndAppendToChunk(ptr RowPtr, chk *Chunk) (row Row, _ *Chunk, err error) { off, err := l.getOffset(ptr.ChkIdx, ptr.RowIdx) if err != nil { return } r := l.dataFile.getSectionReader(off) format := rowInDisk{numCol: len(l.fieldTypes)} _, err = format.ReadFrom(r) if err != nil { return row, nil, err } row, chk = format.toRow(l.fieldTypes, chk) return row, chk, err }
It would be better to pipeline the format.ReadFrom (which is blocked by disk) and the format.toRow (which is blocked by CPU).
format.ReadFrom
format.toRow
Also, with the assumption that a whole chunk is stored together tightly, we can get the first row offset and use it for the whole chunk.
The text was updated successfully, but these errors were encountered:
(*ListInDisk).GetChunk
util/chunk: optimize (*ListInDisk).GetChunk and add a fast row cont…
ab4c06a
…ainer reader (#45130) close #45125
aa4084e
…ainer reader (#45130) (#45203) close #45125
a431496
…ainer reader (#45130) (#45204) close #45125
Successfully merging a pull request may close this issue.
Enhancement
The current implementation of reading rows from
RowContainer
is not fast enough. The following function is used to dump a chunk:It would be better to pipeline the
format.ReadFrom
(which is blocked by disk) and theformat.toRow
(which is blocked by CPU).Also, with the assumption that a whole chunk is stored together tightly, we can get the first row offset and use it for the whole chunk.
The text was updated successfully, but these errors were encountered: