Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

policer: Unexpectedly many requests for the same object per clock #1411

Closed
cthulhu-rider opened this issue May 20, 2022 · 0 comments · Fixed by #1507
Closed

policer: Unexpectedly many requests for the same object per clock #1411

cthulhu-rider opened this issue May 20, 2022 · 0 comments · Fixed by #1507
Assignees
@carpawell
Labels
bug Something isn't working neofs-storage Storage node application issues U3 Regular

Comments

@cthulhu-rider
Copy link
Contributor

Need to be explored. Eyes caught on a large number of messages:

2022-05-20T10:53:30.044Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s03.neofs.devenv/tcp/8080]: read object header from NeoFS: write request: rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2022-05-20T10:53:30.053Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s01.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:30.094Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s04.neofs.devenv/tcp/8082/tls /dns4/s04.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:31.044Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s03.neofs.devenv/tcp/8080]: read object header from NeoFS: write request: rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2022-05-20T10:53:31.053Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s01.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:31.059Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s04.neofs.devenv/tcp/8082/tls /dns4/s04.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:32.045Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s03.neofs.devenv/tcp/8080]: read object header from NeoFS: write request: rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2022-05-20T10:53:32.053Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s01.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:32.058Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s04.neofs.devenv/tcp/8082/tls /dns4/s04.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:33.046Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s03.neofs.devenv/tcp/8080]: read object header from NeoFS: write request: rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2022-05-20T10:53:33.054Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s01.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:33.061Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s04.neofs.devenv/tcp/8082/tls /dns4/s04.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:34.046Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s03.neofs.devenv/tcp/8080]: read object header from NeoFS: write request: rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2022-05-20T10:53:34.053Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s01.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:34.061Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s04.neofs.devenv/tcp/8082/tls /dns4/s04.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:35.047Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s03.neofs.devenv/tcp/8080]: read object header from NeoFS: write request: rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2022-05-20T10:53:35.055Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s01.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}
2022-05-20T10:53:35.062Z	error	policer/check.go:114	receive object header to check policy compliance	{"component": "Object Policer", "object": "HTP3MZLWsq5uzrZfh1BwGhQoC6NZW8EG12rCChoueKVN/EH3MwCPXGBpLFMR3nYzXMAqcjLTGa4FR9izcftdnuRAq", "error": "(*headsvc.RemoteHeader) could not head object in [/dns4/s04.neofs.devenv/tcp/8082/tls /dns4/s04.neofs.devenv/tcp/8080]: read object header from NeoFS: unexpected header type <nil>"}

Note: don't pay attention to exact errors, that was a debug run.

Reason?:

  • policer task selection
  • placement
  • ???
@cthulhu-rider cthulhu-rider added triage neofs-storage Storage node application issues U3 Regular labels May 20, 2022
@cthulhu-rider cthulhu-rider changed the title replication:Uunexpectedly many requests for the same object per clock replication: Unexpectedly many requests for the same object per clock May 20, 2022
@cthulhu-rider cthulhu-rider changed the title replication: Unexpectedly many requests for the same object per clock policer: Unexpectedly many requests for the same object per clock May 20, 2022
@cthulhu-rider cthulhu-rider added the bug Something isn't working label May 20, 2022
@carpawell carpawell self-assigned this Jun 8, 2022
@carpawell carpawell mentioned this issue Jun 9, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@carpawell carpawell

Labels
bug Something isn't working neofs-storage Storage node application issues U3 Regular
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix/Do not handle object concurrently by the policer
2 participants
@carpawell @cthulhu-rider