[Backport 0.23] fix(gateway): map exception to correct error code

[korthout]

Description

backports #5228

• Previous BrokerException was directly returned to the client, it is now mapped to the correct StatusException and code.
• The error message was rewritten to make more clear what has happened.

Related issues

fixes #5187

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The impact of the changes is verified by a benchmark

Documentation:

• The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
• New content is added to the release announcement

[korthout]

@saig0 Can you have a look? It's a direct cherry pick

[saig0]

Thanks 🍥

Please test the fix manually since there are no automated tests :)

[korthout]

Reproduced problem by:

1. checkout stable/0.23
2. rm -rf ./dist/target/zeebe-broker/data
3. build broker and run
4. deploy a simple workflow (start -> service task (jobtype: test) -> end)
5. stop broker
6. change broker to always reply RESOURCE_EXHAUSTED (CommandApiRequestHandler#L138=> if(true))
7. restart broker
8. zbctl --insecure create worker test --handler=cat

2020/09/25 10:18:54 Failed to activate jobs for worker zbctl rpc error: code = Unknown desc =
2020/09/25 10:18:54 Failed to activate jobs for worker zbctl rpc error: code = Unknown desc =
2020/09/25 10:18:54 Failed to activate jobs for worker zbctl rpc error: code = Unknown desc =
2020/09/25 10:18:54 Failed to activate jobs for worker zbctl rpc error: code = Unknown desc =
2020/09/25 10:18:54 Failed to activate jobs for worker zbctl rpc error: code = Unknown desc =

Manual test of fix by:

1. stop broken broker
2. checkout zell-5187-grpc-resource-exhausted-0.23, bu again change that broker always replies RESOURCE_EXHAUSTED
3. restart broker
4. zbctl --insecure create worker test --handler=cat

2020/09/25 10:20:46 Failed to activate jobs for worker zbctl rpc error: code = ResourceExhausted desc = Expected to activate jobs of type 'test', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
2020/09/25 10:20:46 Failed to activate jobs for worker zbctl rpc error: code = ResourceExhausted desc = Expected to activate jobs of type 'test', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
2020/09/25 10:20:46 Failed to activate jobs for worker zbctl rpc error: code = ResourceExhausted desc = Expected to activate jobs of type 'test', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
2020/09/25 10:20:46 Failed to activate jobs for worker zbctl rpc error: code = ResourceExhausted desc = Expected to activate jobs of type 'test', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
2020/09/25 10:20:47 Failed to activate jobs for worker zbctl rpc error: code = ResourceExhausted desc = Expected to activate jobs of type 'test', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.

[korthout]

bors r+

[zeebe-bors]

Build succeeded:

• continuous-integration/jenkins/branch