Complex object sometimes can't be fetched from certain nodes when bearer token is attached

[alexvanin]

I am trying to upload and download object with bearer token. I use wallets from neofs-dev-env:

• wallets/wallet.json as container owner,
• services/s3_gate/wallet.json as request sender.

Expected Behavior

All storage nodes in the system return complex object.

Current Behavior

Some container nodes return object not found error.

neofs-cli object get --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE \
  --oid BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B \
  -w services/s3_gate/wallet.json \
  --bearer ./b.signed.json \
  -r s03.neofs.devenv:8080 --file ./a
Enter password > 
 1000000 / 10485760 [==========>-----------------------------------------------------]   9.54% 0s
rpc error: copy payload: status: code = 2049 message = object not found

When bearer token is not attached, object is fetched (from public container).

neofs-cli object get --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE \
  --oid BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B \
  -w services/s3_gate/wallet.json \
  -r s03.neofs.devenv:8080 --file ./a
Enter password > 
 10485760 / 10485760 [===============================================================] 100.00% 0s
[./a] Object successfully saved
ID: BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B
CID: BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE
Owner: NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt
...

Steps to Reproduce (for bugs)

1. Start dev-env. I also reduce max object size to 1MB so it works faster.

make up/basic && make update.max_object_size val=1000000 && make prepare.ir

2. Create container where container size is bigger than number of replicas.

I used REP 2 IN X CBF 1 SELECT 3 FROM * AS X so I had:

• one node out of container
• three nodes in container
• two replicas for each object in these three nodes

neofs-cli container create -p "REP 2 IN X CBF 1 SELECT 3 FROM * AS X" \
  --basic-acl eacl-public-read-write \
  -r s01.neofs.devenv:8080 \
  -w wallets/wallet.json

3. Sign bearer token for all containers. Bearer token for services/s3_gate/wallet.json is attached to this issue.

neofs-cli util sign bearer-token --from b.json --to b.signed.json --json -w wallets/wallet.json

4. Upload complex object into container

neofs-cli object put --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE \
  --file ./10m \
  -w services/s3_gate/wallet.json \
  -r s01.neofs.devenv:8080 \
  --bearer ./b.signed.json

5. Try to get complex object from all available nodes using bearer token

neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json -r s01.neofs.devenv:8080 --file ./a 
Enter password > 
 0 / 10485760 [----------------------------------------------------------------]   0.00% 0s
rpc error: copy payload: status: code = 2049 message = object not found

neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json  -r s02.neofs.devenv:8080 --file ./a
Enter password > 
 10485760 / 10485760 [===============================================================] 100.00% 0s
[./a] Object successfully saved
ID: BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B
CID: BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE
Owner: NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt

neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json -r s03.neofs.devenv:8080 --file ./a 
Enter password > 
 1000000 / 10485760 [==========>-----------------------------------------------------]   9.54% 0s
rpc error: copy payload: status: code = 2049 message = object not found

neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json -r s04.neofs.devenv:8080 --file ./a 
Enter password > 
 10485760 / 10485760 [===============================================================] 100.00% 0s
[./a] Object successfully saved
ID: BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B
CID: BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE
Owner: NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt

Context

GET requests are failed on container nodes which contain some split information.

# S01 FAIL
neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s01.neofs.devenv:8080 --ttl 1
Found 1 objects.
BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B

# S02 OK [ OUT OF CONTAINER ]
$ neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s02.neofs.devenv:8080 --ttl 1
Found 0 objects.

# S03 FAIL
neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s03.neofs.devenv:8080 --ttl 1
Found 1 objects.
BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B

# S04 OK
$ neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s04.neofs.devenv:8080 --ttl 1
Found 0 objects.

These nodes try to send inter-container requests to fetch the one of the child objects from container nodes, but they fail.

2022-11-10T15:58:07.349Z        debug   get/remote.go:14        processing node...      {"component": "Object.Get service", "request": "GET", "address": "BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE/H2zVhS6sUSdLyoaNUuAVM1fdxw4dDidfwbJFCKvjUEKx", "raw": false, "local": false, "with session": false, "with bearer": true}
2022-11-10T15:58:07.351Z        debug   get/remote.go:34        remote call failed      {"component": "Object.Get service", "request": "GET", "address": "BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE/H2zVhS6sUSdLyoaNUuAVM1fdxw4dDidfwbJFCKvjUEKx", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "read object header: status: code = 2048 message = access to object operation denied"}

See more logs in attachment below.

Node sends new requests to container nodes and attaches bearer token from original request. Bearer token integrity check fails there.

neofs-node/pkg/services/object/acl/acl.go

Lines 252 to 255 in d8d3588

	if !token.AssertUser(usrSender) {
	// TODO: #767 in this case we can issue all owner keys from neofs.id and check once again
	return errBearerInvalidOwner
	}

Possible Solution

Do not attach bearer token for inter-container communication. It should work fine considering that only container nodes are assembling the objects right now (see #838)

Your Environment

• Version used: neofs-node v0.34.0-28-g01a226b3 (support branch)

b.json
s03-get-logs.txt

[carpawell]

@fyrchik, @acid-ant, @cthulhu-rider, depending on the "speed of the fix", solution for #838, and some other preferences i see two main possibilities:

1. Drop bearer token when spawning a new GET/HEAD request if a node IS sure it is a container node (in fact, i am surprised why the token is even attached);
2. Add a session token for every child object (or for a full container) to any GET request (strange but i do not see any other opportunity to be able to get all the parts of a big object with a bearer token).

[fyrchik]

Closed via #2045