Handle GRPC timeout error #883

Ivo-Yankov · 2023-02-07T16:10:26Z

Description:
This PR aims to resolve several inconsistent errors that occur at times of increased load.

Adds a new env var for configuring sdkClient._maxExecutionTime.
Fixes a rare bug in eth_getTransansactionByHash and eth_getTransactionReceipt.
Improves error handling.

Related issue(s):

Fixes #874

Notes for reviewer:

Checklist

Documented (Code comments, README, etc.)
Tested (unit, integration, etc.)

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

codecov-commenter · 2023-02-07T16:14:14Z

Codecov Report

Base: 74.16% // Head: 75.09% // Increases project coverage by +0.92% 🎉

Coverage data is based on head (1c3aa1a) compared to base (1f9b860).
Patch coverage: 68.62% of modified lines in pull request are covered.

❗ Current head 1c3aa1a differs from pull request most recent head 8d4d502. Consider uploading reports for the commit 8d4d502 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #883      +/-   ##
==========================================
+ Coverage   74.16%   75.09%   +0.92%     
==========================================
  Files          29       16      -13     
  Lines        1827     1349     -478     
  Branches      339      254      -85     
==========================================
- Hits         1355     1013     -342     
+ Misses        374      269     -105     
+ Partials       98       67      -31

Impacted Files	Coverage Δ
packages/relay/src/lib/errors/SDKClientError.ts	`62.50% <0.00%> (-4.17%)`	⬇️
packages/relay/src/lib/clients/sdkClient.ts	`20.70% <22.22%> (+0.08%)`	⬆️
packages/relay/src/lib/eth.ts	`84.74% <71.42%> (+1.90%)`	⬆️
packages/relay/src/lib/clients/mirrorNodeClient.ts	`94.42% <100.00%> (+0.09%)`	⬆️
packages/relay/src/lib/errors/JsonRpcError.ts	`72.72% <100.00%> (+2.72%)`	⬆️
packages/relay/src/lib/hbarlimiter/index.ts	`100.00% <100.00%> (ø)`
packages/server/src/rateLimit/index.ts
packages/server/src/server.ts
packages/server/src/validator/constants.ts
...kages/server/src/koaJsonRpc/lib/RpcInvalidError.ts
... and 9 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

# Conflicts: # .env.example

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

Nana-EC

LG, just some clean up and comment suggestions

helm-chart/templates/configmap.yaml

packages/relay/src/lib/clients/mirrorNodeClient.ts

packages/relay/src/lib/clients/sdkClient.ts

packages/server/tests/localAcceptance.env

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

Nana-EC

LG

Nana-EC · 2023-02-14T13:44:02Z

packages/server/tests/localAcceptance.env

-TIER_1_RATE_LIMIT = 100
-TIER_2_RATE_LIMIT = 800
-TIER_3_RATE_LIMIT = 1600
+DEFAULT_RATE_LIMIT = 400


Why did we double everything here?
Maybe update the PR comments

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

sonarcloud · 2023-02-14T13:50:13Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

Nana-EC

LG

* feat: add grpc timeout config and error handling Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: add new predefined errors Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: add env var to helmchart configs Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: increase rate limits Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: increase rate limits Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * fix: add workaround for rare bug Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: revert rate limit increase Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * nit: address comments Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * nit: address comment from a previous PR Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * nit: typo Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: revert rate limits for acceptance tests Signed-off-by: Ivo Yankov <ivo@devlabs.bg> --------- Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

* feat: add grpc timeout config and error handling Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: add new predefined errors Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: add env var to helmchart configs Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: increase rate limits Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: increase rate limits Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * fix: add workaround for rare bug Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: revert rate limit increase Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * nit: address comments Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * nit: address comment from a previous PR Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * nit: typo Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: revert rate limits for acceptance tests Signed-off-by: Ivo Yankov <ivo@devlabs.bg> --------- Signed-off-by: Ivo Yankov <ivo@devlabs.bg> Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Cherry pick PRs that went into main branch since cutting a 0.18.0-alpha1. PRs map to individual commits - [PR 882 - eth_feeHistory invalid cached response](#882) - [PR 887 - Needed changes for metrics improvements](#887) - [PR 883 - Handle GRPC timeout error](#883) - PR 898 - update-logs-and-metrics Added the requestId to the server context](#898) --------- Signed-off-by: ebadiere <ebadiere@gmail.com> Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com> --------- Signed-off-by: nikolay <n.atanasow94@gmail.com> Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com> Signed-off-by: Alfredo Gutierrez <alfredo@swirldslabs.com> Signed-off-by: Ivo Yankov <ivo@devlabs.bg> Signed-off-by: ebadiere <ebadiere@gmail.com> Co-authored-by: Nikolay Atanasow <n.atanasow94@gmail.com> Co-authored-by: Alfredo Gutierrez <alfredo@swirldslabs.com> Co-authored-by: Ivo Yankov <ivo@devlabs.bg> Co-authored-by: Eric Badiere <ebadiere@gmail.com>

* feat: add grpc timeout config and error handling Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: add new predefined errors Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: add env var to helmchart configs Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: increase rate limits Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: increase rate limits Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * fix: add workaround for rare bug Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: revert rate limit increase Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * nit: address comments Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * nit: address comment from a previous PR Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * nit: typo Signed-off-by: Ivo Yankov <ivo@devlabs.bg> * chore: revert rate limits for acceptance tests Signed-off-by: Ivo Yankov <ivo@devlabs.bg> --------- Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

Ivo-Yankov added 2 commits February 7, 2023 15:40

feat: add grpc timeout config and error handling

424fb71

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

chore: add new predefined errors

57c7000

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

Ivo-Yankov added 5 commits February 8, 2023 11:28

Merge branch 'main' into 874-handle-grpc-timeout-error

39e9d02

# Conflicts: # .env.example

chore: add env var to helmchart configs

c66b3a6

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

chore: increase rate limits

91756cc

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

chore: increase rate limits

82c0aab

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

fix: add workaround for rare bug

ae6eff4

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

Ivo-Yankov marked this pull request as ready for review February 8, 2023 12:33

Ivo-Yankov self-assigned this Feb 8, 2023

Ivo-Yankov linked an issue Feb 8, 2023 that may be closed by this pull request

intermittent unreliable contract deploys #874

Closed

Ivo-Yankov requested review from Nana-EC, georgi-l95, lukelee-sl and natanasow February 8, 2023 12:38

Ivo-Yankov added enhancement New feature or request P1 dev tools Features enabling dev tool integration limechain labels Feb 8, 2023

chore: revert rate limit increase

14a641d

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

Nana-EC requested review from AlfredoG87 and ebadiere February 10, 2023 19:39

Nana-EC reviewed Feb 11, 2023

View reviewed changes

Ivo-Yankov added 3 commits February 14, 2023 11:51

nit: address comments

d04d249

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

nit: address comment from a previous PR

8d4d502

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

nit: typo

9ed7f3a

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

Nana-EC previously approved these changes Feb 14, 2023

View reviewed changes

chore: revert rate limits for acceptance tests

a815b1f

Signed-off-by: Ivo Yankov <ivo@devlabs.bg>

Ivo-Yankov dismissed Nana-EC’s stale review via a815b1f February 14, 2023 13:48

Nana-EC approved these changes Feb 14, 2023

View reviewed changes

Nana-EC added this to the 0.18.0 milestone Feb 14, 2023

Ivo-Yankov merged commit 41debf8 into main Feb 14, 2023

Ivo-Yankov deleted the 874-handle-grpc-timeout-error branch February 14, 2023 14:18

Nana-EC mentioned this pull request Feb 15, 2023

Cherry pick 0.18 commits since 0.18.0-alpha1 #901

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle GRPC timeout error #883

Handle GRPC timeout error #883

Ivo-Yankov commented Feb 7, 2023 •

edited by Nana-EC

Loading

codecov-commenter commented Feb 7, 2023 •

edited

Loading

Nana-EC left a comment

Nana-EC left a comment

Nana-EC Feb 14, 2023

sonarcloud bot commented Feb 14, 2023

Nana-EC left a comment

Handle GRPC timeout error #883

Handle GRPC timeout error #883

Conversation

Ivo-Yankov commented Feb 7, 2023 • edited by Nana-EC Loading

codecov-commenter commented Feb 7, 2023 • edited Loading

Codecov Report

Nana-EC left a comment

Choose a reason for hiding this comment

Nana-EC left a comment

Choose a reason for hiding this comment

Nana-EC Feb 14, 2023

Choose a reason for hiding this comment

sonarcloud bot commented Feb 14, 2023

Nana-EC left a comment

Choose a reason for hiding this comment

Ivo-Yankov commented Feb 7, 2023 •

edited by Nana-EC

Loading

codecov-commenter commented Feb 7, 2023 •

edited

Loading