Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

'Object not found' error right after PUT big object #1878

Closed
anikeev-yadro opened this issue Oct 11, 2022 · 4 comments
Closed

'Object not found' error right after PUT big object #1878

anikeev-yadro opened this issue Oct 11, 2022 · 4 comments
Assignees
@carpawell
Labels
bug Something isn't working U0 Needs to be resolved immediately

Comments

@anikeev-yadro
Copy link

Expected Behavior

GET object successfully right after PUT

Current Behavior

We have 'Object not found' error right adter PUT when we try to GET it.

Steps to Reproduce (for bugs)

  1. PUT object with object_size: 104857600
COMMAND: 'neofs-cli --config /tests/wallet_config.yml object put --rpc-endpoint '172.26.163.107:8080' --wallet '/tests/TemporaryDir/95c6eef3-ed0d-4b3d-bf64-311f96ef3f25.json' --cid '9WbxmWhVrHKCM3FA32DuExvv1YnimXykDQsK8AoTG7dw' --file '/tests/TemporaryDir/d98f1c69-6387-4eaa-9483-511080f724e1' --attributes 'key1=2,common_key=common_value' --no-progress'
OUTPUT:
 [/tests/TemporaryDir/d98f1c69-6387-4eaa-9483-511080f724e1] Object successfully stored
  OID: AGaJv32jxYn72EvLhjgQUNwHnUfpcsr1fXgTtboY4E8V
  CID: 9WbxmWhVrHKCM3FA32DuExvv1YnimXykDQsK8AoTG7dw

RC: 0
Start / End / Elapsed	 05:57:27.254856 / 05:57:42.983090 / 0:00:15.728234
  1. Try to GET object:
COMMAND: 'neofs-cli --config /tests/wallet_config.yml object get --rpc-endpoint '172.26.161.88:8080' --wallet '/tests/TemporaryDir/95c6eef3-ed0d-4b3d-bf64-311f96ef3f25.json' --cid '9WbxmWhVrHKCM3FA32DuExvv1YnimXykDQsK8AoTG7dw' --oid 'AGaJv32jxYn72EvLhjgQUNwHnUfpcsr1fXgTtboY4E8V' --file 'TemporaryDir/f83a9afe-efda-4f59-a2b8-5dacafdd151a' --no-progress'
OUTPUT:
 rpc error: copy payload: status: code = 2049 message = object not found
RC: 1
Start / End / Elapsed	 05:57:58.445248 / 05:57:59.079210 / 0:00:00.633962

Logs:

Oct 11 05:57:56 vedi neofs-node[7226]: 2022-10-11T05:57:56.264Z        debug        get/get.go:87        serving request...        {"component": "Object.Get service", "request": "GET", "address": "9WbxmWhVrHKCM3FA32DuExvv1YnimXykDQsK8AoTG7dw/H1DSJByifEjZddEbdi2vqZiXXZ8j8SxxPoiFFNnTcJBo", "raw": false, "local": false, "with session": true, "with bearer": false}
Oct 11 05:57:56 vedi neofs-node[7226]: 2022-10-11T05:57:56.290Z        debug        get/get.go:99        operation finished successfully        {"component": "Object.Get service", "request": "GET", "address": "9WbxmWhVrHKCM3FA32DuExvv1YnimXykDQsK8AoTG7dw/H1DSJByifEjZddEbdi2vqZiXXZ8j8SxxPoiFFNnTcJBo", "raw": false, "local": false, "with session": true, "with bearer": false}
Oct 11 05:57:56 vedi neofs-node[7226]: 2022-10-11T05:57:56.291Z        debug        get/exec.go:212        parent address in child object differs        {"component": "Object.Get service", "request": "GET", "address": "9WbxmWhVrHKCM3FA32DuExvv1YnimXykDQsK8AoTG7dw/AGaJv32jxYn72EvLhjgQUNwHnUfpcsr1fXgTtboY4E8V", "raw": false, "local": false, "with session": true, "with bearer": false}
Oct 11 05:57:56 vedi neofs-node[7226]: 2022-10-11T05:57:56.387Z        debug        get/get.go:87        serving request...        {"component": "Object.Get service", "request": "GET", "address": "9WbxmWhVrHKCM3FA32DuExvv1YnimXykDQsK8AoTG7dw/88RMo4yoJp49tkcTBBWVAGihNVjRkbXnZgiQWd6fybK6", "raw": false, "local": false, "with session": true, "with bearer": false}
Oct 11 05:57:56 vedi neofs-node[7226]: 2022-10-11T05:57:56.411Z        debug        get/get.go:99        operation finished successfully        {"component": "Object.Get service", "request": "GET", "address": "9WbxmWhVrHKCM3FA32DuExvv1YnimXykDQsK8AoTG7dw/88RMo4yoJp49tkcTBBWVAGihNVjRkbXnZgiQWd6fybK6", "raw": false, "local": false, "with session": true, "with bearer": false}
Oct 11 05:57:56 vedi neofs-node[7226]: 2022-10-11T05:57:56.412Z        debug        get/exec.go:212        parent address in child object differs        {"component": "Object.Get service", "request": "GET", "address": "9WbxmWhVrHKCM3FA32DuExvv1YnimXykDQsK8AoTG7dw/AGaJv32jxYn72EvLhjgQUNwHnUfpcsr1fXgTtboY4E8V", "raw": false, "local": false, "with session": true, "with bearer": false}

aa516c7db2d83630.zip

Versions:

neo-go 0.99.3-2-gaa57779d
neofs-authmate v0.24.0-9-gd6d6a09f
neofs-adm 0.32.0-91-g807c0a13
neofs-cli 0.32.0-91-g807c0a13
neofs-http-gw 0.24.0-12-g45c036f5
neofs-ir 0.32.0-91-g807c0a13
neofs-lens 0.32.0-91-g807c0a13
neofs-node 0.32.0-91-g807c0a13
neofs-s3-authmate 0.24.0-9-gd6d6a09f
neofs-s3-gw 0.24.0-9-gd6d6a09f

Your Environment

Server setup and configuration:
cloud, 4 VMs, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux
cthulhu-rider pushed a commit to cthulhu-rider/neofs-node that referenced this issue Oct 11, 2022
[nspcc-dev#1878] services/object: Fix child check in GET
57a8703 
In previous implementation `ObjectService.Get` RPC handler failed with
`parent address in child object differs` while assembling the "big"
object. This was caused by the child check which required parent
reference to be set in all child objects. The check was impracticable
because not all elements of the split-chain have a link to the parent.

Make `execCtx.isChild` to return `true` if parameterized object has no
parent header in its own header.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
cthulhu-rider pushed a commit to cthulhu-rider/neofs-node that referenced this issue Oct 11, 2022
[nspcc-dev#1878] services/object: Fix child check in GET
486a043 
In previous implementation `ObjectService.Get` RPC handler failed with
`parent address in child object differs` while assembling the "big"
object. This was caused by the child check which required parent
reference to be set in all child objects. The check was impracticable
because not all elements of the split-chain have a link to the parent.

Make `execCtx.isChild` to return `true` if parameterized object has no
parent header in its own header.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
cthulhu-rider pushed a commit to cthulhu-rider/neofs-node that referenced this issue Oct 11, 2022
[nspcc-dev#1878] services/object: Fix child check in GET
6dea9c2 
In previous implementation `ObjectService.Get` RPC handler failed with
`parent address in child object differs` while assembling the "big"
object. This was caused by the child check which required parent
reference to be set in all child objects. The check was impracticable
because not all elements of the split-chain have a link to the parent.

Make `execCtx.isChild` to return `true` if parameterized object has no
parent header in its own header.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
@cthulhu-rider cthulhu-rider mentioned this issue Oct 11, 2022
Merged
@cthulhu-rider cthulhu-rider removed their assignment Oct 12, 2022
cthulhu-rider pushed a commit to cthulhu-rider/neofs-node that referenced this issue Oct 12, 2022
[nspcc-dev#1878] services/object: Fix child check in GET
fb8ac48 
In previous implementation `ObjectService.Get` RPC handler failed with
`parent address in child object differs` while assembling the "big"
object. This was caused by the child check which required parent
reference to be set in all child objects. The check was impracticable
because not all elements of the split-chain have a link to the parent.

Make `execCtx.isChild` to return `true` if parameterized object has no
parent header in its own header.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
@fyrchik
Copy link
Contributor

fyrchik commented Oct 12, 2022

Closed via #1879 and #1884.

aprasolova pushed a commit to aprasolova/neofs-node that referenced this issue Oct 19, 2022
@aprasolova
[nspcc-dev#1878] services/object: Fix child check in GET
028b878 
In previous implementation `ObjectService.Get` RPC handler failed with
`parent address in child object differs` while assembling the "big"
object. This was caused by the child check which required parent
reference to be set in all child objects. The check was impracticable
because not all elements of the split-chain have a link to the parent.

Make `execCtx.isChild` to return `true` if parameterized object has no
parent header in its own header.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
@dansingjulia
Copy link

reproduced on last changes.

2022-11-08T02:54:05.655Z    info    api/router.go:162    call method    {"status": 200, "request_id": "50374cef-8e7d-4ec3-8875-c8377181d635", "method": "PutObject", "bucket": "fa226b77-3055-4b48-9c7b-919afc79888b", "object": "39be4ae6-902c-4e9d-ba46-4a00c916f52c", "description": "OK"}
2022-11-08T02:54:09.057Z    info    api/router.go:162    call method    {"status": 200, "request_id": "ce5d249e-0ab8-446f-bf18-ddfce8b77f3c", "method": "PutObject", "bucket": "fa226b77-3055-4b48-9c7b-919afc79888b", "object": "193ead7f-f08e-4a52-9919-ea401c95849b", "description": "OK"}
2022-11-08T02:54:12.360Z    error    handler/util.go:29    call method    {"status": 500, "request_id": "c6a5669e-a4af-4c6f-be57-670ebeb23336", "method": "CopyObject", "bucket": "434822bc-bdf0-4fa1-aebe-ab2ebc505611", "object": "/home/ci/jenkins_home/workspace/Bundle_neofs_build_NB/neofs-testcases/c11ae580-8afb-4652-8b53-963af7c5440a", "description": "couldn't copy object", "error": "save object via connection pool: init writing on API client: read payload: copy object payload written: '0': status: code = 2049 message = object not found", "src_bucket_name": "fa226b77-3055-4b48-9c7b-919afc79888b", "src_object_name": "193ead7f-f08e-4a52-9919-ea401c95849b"}
2022-11-08T02:54:15.794Z    info    handler/copy.go:241    object is copied    {"bucket": "434822bc-bdf0-4fa1-aebe-ab2ebc505611", "object": "/home/ci/jenkins_home/workspace/Bundle_neofs_build_NB/neofs-testcases/c11ae580-8afb-4652-8b53-963af7c5440a", "object_id": "Hk9Ena4D4dk58hEteruhi7wDivDQXfCJQSEyhYujxgfe"}
2022-11-08T02:54:15.794Z    info    api/router.go:162    call method    {"status": 200, "request_id": "cb6dbec2-7b08-4a84-babe-8643c50fe4ad", "method": "CopyObject", "bucket": "434822bc-bdf0-4fa1-aebe-ab2ebc505611", "object": "/home/ci/jenkins_home/workspace/Bundle_neofs_build_NB/neofs-testcases/c11ae580-8afb-4652-8b53-963af7c5440a", "description": "OK"}
2022-11-08T02:54:20.883Z    debug    layer/layer.go:345    bucket not found    {"error": "dns: couldn't resolve container '757151d4-d5dc-407e-99e1-c046c2e7e34c' as '757151d4-d5dc-407e-99e1-c046c2e7e34c.container': lookup 757151d4-d5dc-407e-99e1-c046c2e7e34c.container on 127.0.0.11:53: server misbehaving"}

s3 cannto get object from node after put operation with error: object not found

@dansingjulia
Copy link

dansingjulia commented Nov 9, 2022 •
edited
Loading

logs

@abereziny abereziny mentioned this issue Nov 9, 2022
Merged
@alexchetaev alexchetaev added 2022Q4 and removed 2022Q3 labels Nov 11, 2022
@fyrchik fyrchik assigned carpawell and unassigned cthulhu-rider Nov 12, 2022
@fyrchik
Copy link
Contributor

fyrchik commented Nov 12, 2022

Closed via #2045 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@carpawell carpawell

Labels
bug Something isn't working U0 Needs to be resolved immediately
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@fyrchik @carpawell @alexchetaev @cthulhu-rider @dansingjulia @anikeev-yadro