Skip to content

WFE: Add custom balancer implementation which routes nonce redemption RPCs by prefix#6618

Merged
beautifulentropy merged 39 commits intomainfrom
nonce-service-discovery
Feb 3, 2023
Merged

WFE: Add custom balancer implementation which routes nonce redemption RPCs by prefix#6618
beautifulentropy merged 39 commits intomainfrom
nonce-service-discovery

Conversation

@beautifulentropy
Copy link
Copy Markdown
Member

@beautifulentropy beautifulentropy commented Jan 27, 2023

Assign nonce prefixes for each nonce-service by taking the first eight characters of the the base64url encoded HMAC-SHA256 hash of the RPC listening address using a provided key. The provided key must be same across all boulder-wfe and nonce-service instances.

  • Add a custom grpc-go load balancer implementation (nonce) which can route nonce redemption RPC messages by matching the prefix to the derived prefix of the nonce-service instance which created it.
  • Modify the RPC client constructor to allow the operator to override the default load balancer implementation (round_robin).
  • Modify the srv RPC resolver to accept a comma separated list of targets to be resolved.
  • Remove unused nonce-service -prefix flag.

Fixes #6404

@beautifulentropy beautifulentropy force-pushed the nonce-service-discovery branch 2 times, most recently from 144b6ae to df2894a Compare January 27, 2023 01:11
@beautifulentropy beautifulentropy changed the title WFE: Use custom loadbalancer to route nonce redemption RPCs WFE: Use custom load balancer to route nonce redemption RPCs Feb 2, 2023
@beautifulentropy beautifulentropy changed the title WFE: Use custom load balancer to route nonce redemption RPCs WFE: Add custom balancer implementation which routes nonce redemption RPCs by prefix Feb 2, 2023
Copy link
Copy Markdown
Contributor

@jsha jsha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good!

@beautifulentropy beautifulentropy requested a review from jsha February 3, 2023 19:15
jsha
jsha previously approved these changes Feb 3, 2023
Copy link
Copy Markdown
Contributor

@jsha jsha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the changes! Look great.

Copy link
Copy Markdown
Contributor

@aarongable aarongable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with just a few comment nits!

@beautifulentropy
Copy link
Copy Markdown
Member Author

@jsha can you re-ack when you get a chance?

@beautifulentropy beautifulentropy merged commit d73125d into main Feb 3, 2023
@beautifulentropy beautifulentropy deleted the nonce-service-discovery branch February 3, 2023 22:52
beautifulentropy added a commit that referenced this pull request Jul 28, 2023
…efixes (#7004)

Fix an issue related to the custom gRPC Picker implementation introduced
in #6618. When a nonce contained a prefix not associated with a known
backend, the Picker would continuously rebuild, re-resolve DNS, and
eventually throw a 500 "Server Error" at RPC timeout. The Picker now
promptly returns a 400 "Bad Nonce" error as expected, in response the
requesting client should retry their request with a fresh nonce.

Additionally:
- WFE unit tests use derived nonces when `"BOULDER_CONFIG_DIR" ==
"test/config-next"`.
- `Balancer.Build()` in "noncebalancer" forces a rebuild until non-zero
backends are available. This matches the
[balancer/roundrobin](https://github.com/grpc/grpc-go/blob/d524b409462c601ef3f05a7e1fba19755a337c77/balancer/roundrobin/roundrobin.go#L49-L53)
implementation.
- Nonces with no matching backend increment "jose_errors" with label
`"type": "JWSInvalidNonce"` and "nonce_no_backend_found".
- Nonces of incorrect length are now rejected at the WFE and increment
"jose_errors" with label `"type": "JWSMalformedNonce"` instead of
`"type": "JWSInvalidNonce"`.
- Nonces not encoded as base64url are now rejected at the WFE and
increment "jose_errors" with label `"type": "JWSMalformedNonce"` instead
of `"type": "JWSInvalidNonce"`.

Fixes #6969
Part of #6974
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support communication of prefix between WFE and Nonce

4 participants