Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get_range/get_range_hash with out of bounds throws file_not_found or EOF exceptions instead of out of range #2007

Closed
abereziny opened this issue Nov 1, 2022 · 2 comments · Fixed by #2011
Assignees
Labels
neofs-storage Storage node application issues U2 Seriously planned

Comments

@abereziny
Copy link

abereziny commented Nov 1, 2022

Sometimes get_range and get_range_hash requests with out of bounds --range parameter returns "object not found" exception instead of "out of range"

Expected Behavior

neofs-cli --config wallet_config.yml object hash --wallet 'fb0a1cf9-62d8-462d-a73e-f5718599b15d.json' --cid '4zMtoHgPnFe2fhfj2BqWyJus9N49Zta3hHZLYgtJuZY9' --oid 'A4j6bdbQeLX6gzjZFdVRcLiWvS8fNAKbPe3vofCLUkhf' --range '1001:10'
Error:
return code: 1
output: rpc error: read payload hashes via client: status: code = 2053 message = out of range

Current Behavior

File not found:

neofs-cli --config wallet_config.yml object hash --wallet 'fb0a1cf9-62d8-462d-a73e-f5718599b15d.json' --cid '4zMtoHgPnFe2fhfj2BqWyJus9N49Zta3hHZLYgtJuZY9' --oid 'A4j6bdbQeLX6gzjZFdVRcLiWvS8fNAKbPe3vofCLUkhf' --range '1001:10'
Error:
return code: 1
output: rpc error: read payload hashes via client: status: code = 2049 message = object not found

EOF:

neofs-cli --config wallet_config.yml object hash --wallet '385f73fa-5d6a-41e2-b356-203e5e1e9150.json' --cid '5azT2GeRKLzGrnVmQ4z3vxSKayeQLbgmQM9aKV6r57CR' --oid '2K3qkKk8k2tNvisz8qbS8sgQ7bhpXk9hmLJPaDKRedKe' --range '10:18446744073709551615'
RETCODE: 1

STDOUT:
rpc error: read payload hashes via client: write request: rpc error: code = Unavailable desc = error reading from server: EOF
STDERR:

Failures:
EOF:
https://obj-jenkins.spb.yadro.com/view/Ready%20to%20use/job/sbercloud_test/147/allure/#suites/1aa4749d1babf458baceeb848afce143/9bbe35f376020bb7/

FILE_NOT_FOUND:
https://obj-jenkins.spb.yadro.com/view/Ready%20to%20use/job/sbercloud_test/144/allure/#suites/1aa4749d1babf458baceeb848afce143/388134d232b14b9a/
Passes:
https://obj-jenkins.spb.yadro.com/view/Ready%20to%20use/job/sbercloud_test/142/allure/#suites/1aa4749d1babf458baceeb848afce143/e7686ec36972e23e/

Steps to Reproduce (for bugs)

  1. Put file
  2. Get_range with out_of_bounds --range param

Your Environment

Sber-cloud: v0.1.1-nb-20221024.1

@fyrchik
Copy link
Contributor

fyrchik commented Nov 1, 2022

Nov 01 11:18:45 az neofs-node[2488]: panic: runtime error: makeslice: len out of range
Nov 01 11:18:45 az neofs-node[2488]: goroutine 36592 [running]:
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/internal/client.PayloadRange({{{{0x1276e40, 0xc001592ec0}, {0x126f240, 0xc0020c8930}, 0xc00049a0c0, 0x0, 0x0, 0x1, {0x1ab3ea0, 0x0, ...}}}, ...})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/internal/client/client.go:352 +0x291
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*clientWrapper).getObject(0xc00043a720, 0xc000fe44e0, {{0xc00043a690, 0x1, 0x1}, {0x0, 0x0, 0x0}, {0xc00046fa40, 0x21, ...}})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/util.go:143 +0x318
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).processNode(0xc000fe44e0, {0x47172f?, 0xc00002dd10?}, {{0xc00043a690, 0x1, 0x1}, {0x0, 0x0, 0x0}, {0xc00046fa40, ...}})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/remote.go:21 +0x13e
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).processCurrentEpoch(0xc000fe44e0)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/container.go:86 +0x3ca
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).executeOnContainer(0xc000fe44e0)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/container.go:29 +0x1ca
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).analyzeStatus(0xc000fe44e0, 0x1)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:113 +0x1a5
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).execute(0xc000fe44e0)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:92 +0x57
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).get(0xc00000ee38, {0x126f240?, 0xc0020c8930}, {{0x126aa00, 0xc0006a65c0}, 0xc0022b2410, {{0x44, 0x23, 0xdf, 0x2e, ...}, ...}, ...}, ...)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:81 +0x17e
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).getChild(0xc0023680d0, {0x87, 0xa5, 0xa8, 0xf1, 0xfa, 0xc5, 0x4b, 0xe4, 0x70, ...}, ...)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/exec.go:207 +0x26b
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).overtakePayloadDirectly(0xc0023680d0, {0xc001ea01e0, 0x1, 0x0?}, {0xc001830c10, 0x1, 0x8?}, 0x0)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/assemble.go:143 +0x12d
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).overtakePayloadInReverse(0xc0023680d0, {0x1d, 0xc0, 0xa6, 0xa, 0x34, 0x13, 0x81, 0x19, 0xf5, ...})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/assemble.go:174 +0x167
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).assemble(0xc0023680d0)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/assemble.go:46 +0x28a
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).analyzeStatus(0xc0023680d0, 0x1)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:104 +0x20c
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).execute(0xc0023680d0)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:92 +0x57
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).get(0xc00000ee38, {0x126f240?, 0xc0020c8930}, {{0x126aa28, 0xc0006b8780}, 0xc0022b2410, {{0x44, 0x23, 0xdf, 0x2e, ...}, ...}, ...}, ...)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:81 +0x17e
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).getRange(0x20?, {0x126f240, 0xc0020c8930}, {{{0x126aa28, 0xc0006b8780}, 0xc0022b2410, {{0x44, 0x23, 0xdf, 0x2e, ...}, ...}, ...}, ...}, ...)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:22 +0x179
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).GetRange(...)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:18
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/get/v2.(*Service).GetRange(0xc00000ee40, 0xf61020?, {0x126a8e8, 0xc0006b8750})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/v2/service.go:67 +0xee
Nov 01 11:18:45 az neofs-node[2488]: main.(*objectSvc).GetRange(0xa0?, 0xc000300400?, {0x126a8e8?, 0xc0006b8750?})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/object.go:93 +0x27
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object.TransportSplitter.GetRange(...)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/transport_splitter.go:140
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object/acl/v2.Service.GetRange({0xc00182f1a0, {0xc00046a0d0, {0x12639a0, 0xc001800f40}, {0x126e558, 0xc0017d1488}}}, 0xc000ab8138, {0x126a780?, 0xc000ab8228})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/acl/v2/service.go:385 +0x687
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object.(*Common).GetRange(0xc001800f60, 0xc000ab81c8?, {0x126a780, 0xc000ab8228})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/common.go:82 +0x8d
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object.(*ResponseService).GetRange(0xc0017d1ea8, 0xc000da8920?, {0x126a7a8?, 0xc000ab81b0})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/response.go:137 +0x115
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object.(*SignService).GetRange.func3(0xc000da8920)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/sign.go:187 +0xce
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/util.(*SignService).HandleServerStreamRequest(0xc00000ee60, {0x103a120?, 0xc000ab8138}, 0xc000ab8198, 0x112bc28, 0xc000b4b9b0)
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/util/sign.go:149 +0x132
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object.(*SignService).GetRange(0xc001800f80, 0xc000ab8138, {0x126a2a8?, 0xc0032bc1f0})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/sign.go:179 +0xe6
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/services/object.MetricCollector.GetRange({{0x12741a0?, 0xc001800f80?}, {0x12782a8?, 0xc0000001e0?}}, 0x10?, {0x126a2a8, 0xc0032bc1f0})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/services/object/metrics.go:127 +0x12e
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-node/pkg/network/transport/object/grpc.(*Server).GetRange(0xc001814f60, 0x104d920?, {0x1273e30?, 0xc0032bc1d0})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-node/pkg/network/transport/object/grpc/range.go:26 +0xb4
Nov 01 11:18:45 az neofs-node[2488]: github.com/nspcc-dev/neofs-api-go/v2/object/grpc._ObjectService_GetRange_Handler({0xffee00?, 0xc001814f60}, {0x1272ee8, 0xc002376270})
Nov 01 11:18:45 az neofs-node[2488]:         github.com/nspcc-dev/neofs-api-go/v2@v2.14.0/object/grpc/service_grpc.pb.go:744 +0xd0
Nov 01 11:18:45 az neofs-node[2488]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc001774000, {0x1275518, 0xc000636340}, 0xc000314120, 0xc001827b90, 0x1a6ff20, 0x0)
Nov 01 11:18:45 az neofs-node[2488]:         google.golang.org/grpc@v1.48.0/server.go:1558 +0xf46
Nov 01 11:18:45 az neofs-node[2488]: google.golang.org/grpc.(*Server).handleStream(0xc001774000, {0x1275518, 0xc000636340}, 0xc000314120, 0x0)
Nov 01 11:18:45 az neofs-node[2488]:         google.golang.org/grpc@v1.48.0/server.go:1640 +0x9d6
Nov 01 11:18:45 az neofs-node[2488]: google.golang.org/grpc.(*Server).serveStreams.func1.2()
Nov 01 11:18:45 az neofs-node[2488]:         google.golang.org/grpc@v1.48.0/server.go:932 +0x98
Nov 01 11:18:45 az neofs-node[2488]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Nov 01 11:18:45 az neofs-node[2488]:         google.golang.org/grpc@v1.48.0/server.go:930 +0x28a
Nov 01 11:18:45 az systemd[1]: neofs-storage.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 01 11:18:45 az systemd[1]: neofs-storage.service: Failed with result 'exit-code'.
Nov 01 11:18:45 az systemd[1]: neofs-storage.service: Consumed 42.399s CPU time.

@fyrchik
Copy link
Contributor

fyrchik commented Nov 7, 2022

Closed via #2011 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
neofs-storage Storage node application issues U2 Seriously planned
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants