hrpc.NumberOfRows is not working

[LG991103]

i set the hrpc.NumberOfRows to 2,but i got a length of 1600

[LG991103]

am i using the scan methond wrong way?

[dethi]

// NumberOfRows is an option for scan requests.
// Specifies how many rows are fetched with each request to regionserver.
// Should be > 0, avoid extremely low values such as 1 because a request
// to regionserver will be made for every row.

NumberOfRows doesn't limit how many rows are returned by the scan in total. It limit how many rows are returned by each scans request send to HBase.

[LG991103]

I'm sorry i don't understand what you said. could you  take an example, please.

…

---Original--- From: "Thibault ***@***.***> Date: Thu, Jul 14, 2022 23:27 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [tsuna/gohbase] hrpc.NumberOfRows is not working (Issue #190) // NumberOfRows is an option for scan requests. // Specifies how many rows are fetched with each request to regionserver. // Should be > 0, avoid extremely low values such as 1 because a request // to regionserver will be made for every row. NumberOfRows doesn't limit how many rows are returned by the scan in total. It limit how many rows are returned by each scans request send to HBase. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[dethi]

A scan make multiples requests to regionservers. NumberOfRows only limit the number of rows returned PER requests, not for the totality of the scan.

This will do what you want, get 2 results and stop:

hrpc.NewScanRangeStr(ctx, table, startRow, stopRow, hrpc.NumberOfRows(2), hrpc.CloseScanner())

CloseScanner will close the scanner immediately after the first results are returned.

[dethi]

Another option, especially if you want to follow the same pattern for a larger number of rows or if the rows are spread on multiple regionservers, is to exit the for loop when you have reached the size you wanted and make sure to call scan.Close() to free the scanner on the regionservers.

[LG991103]

got it thx very much

…

---Original--- From: "Thibault ***@***.***> Date: Fri, Jul 15, 2022 02:55 AM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [tsuna/gohbase] hrpc.NumberOfRows is not working (Issue #190) A scan make multiples requests to regionservers. NumberOfRows only limit the number of rows returned PER requests, not for the totality of the scan. What you want to use is hrpc.NewScanRangeStr(ctx, table, startRow, stopRow, hrpc.NumberOfRows(2), hrpc.CloseScanner()). CloseScanner will close the scanner immediately after the first request is returned. This will do what you want: get 2 results and stop. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jseow5177]

Hello, may I ask how can I pull x number of rows from a start key? I am trying to achieve paginated scans. I tried to leave the end key empty, but it still always fetches more than the specified limit.

[LG991103]

A scan make multiples requests to regionservers. NumberOfRows only limit the number of rows returned PER requests, not for the totality of the scan. What you want to use is hrpc.NewScanRangeStr(ctx, table, startRow, stopRow, hrpc.NumberOfRows(2), hrpc.CloseScanner()). CloseScanner will close the scanner immediately after the first request is returned. This will do what you want: get 2 results and stop.

…

---Original--- From: "Jonathan ***@***.***> Date: Sat, Jul 16, 2022 10:18 AM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [tsuna/gohbase] hrpc.NumberOfRows is not working (Issue #190) Hello, may I ask how can I pull x number of rows from a start key? I am trying to achieve paginated scans. I tried to leave the end key empty, but it still always fetches more than the specified limit. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jseow5177]

Hmmm, yeah I get that part. But when I did this,

req, err := hrpc.NewScanRangeStr(ctx, tableName, cursor, "", hrpc.NumberOfRows(uint32(limit)+1), hrpc.CloseScanner())

It still returns more rows that limit + 1.

[tomasbanet]

I made it work like this:

// scanLimit reads at most n number of cells from scanner and returns the result.
// It is the caller's responsibility to close the scanner once all reading has been finished.
func scanLimit(scanner hrpc.Scanner, n int) ([]*hrpc.Cell, error) {
	var cells []*hrpc.Cell
	for len(cells) < n {
		result, err := scanner.Next()
		if err == io.EOF {
			break
		} else if err != nil {
			return nil, err
		}
		for _, cell := range result.Cells {
			// result.Cells can contain anywhere from 1 to many cells
			if len(cells) < n {
				cells = append(cells, cell)
			}
		}
	}
	return cells, nil
}

Caller:

scan, err := hrpc.NewScanRangeStr(ctx, table, k, "",
	hrpc.NumberOfRows(10),
)
if err != nil {
	return err
}
scanner := runner.client.Scan(scan)
cells, err := scanLimit(scanner, 10)
if err != nil {
	return err
}
scanner.Close()

Note that despite setting hrpc.NumberOfRows(10), scanner.Next() can return anywhere from 1 cell at a time to over a million cells at once, which is quite strange.

[dethi]

Note that despite setting hrpc.NumberOfRows(10), scanner.Next() can return anywhere from 1 cell at a time to over a million cells at once, which is quite strange.

1 Cell = 1 Column. If you ask for 1 row but that row has 10000000 columns, you will get 10000000 cells. You can change that by asking only for a specific columns using the hrpc.Families(...) option