Webhook Sentry

Webhook Sentry is a proxy that helps you send webhooks to your customers securely.

Why?

Security

Sending webhooks appears simple on the surface -- they're just HTTP requests after all. But sending them securely is hard. If your application sends webhooks, does your implementation

1. Prevent SSRF attacks?
2. Protect against DNS rebinding attacks?
3. Support mutual TLS?
4. Validate SSL certificate chains correctly?
5. Use an updated CA certificate bundle?
6. Specify reasonable idle socket and connection timeouts?

By proxying webhooks through Webhook Sentry, you get all of these for free.

Auditability

Sending webhooks involves making connections to untrusted and possibly malicious servers on the public internet. Maintaining an audit trail is essential for forensics and compliance. Limiting the set of instances that send such requests to a single proxy layer makes auditing simpler and more manageable.

Static Egress IPs

Many customers require webhook requests to be sent from a list or range of static IPs in order to configure their firewalls. In a cloud environment with autoscaling, you may not want to allocate static IPs to your application instances. In other situations, like serverless applications, it may be impossible to assign static IPs. With a centralized egress proxy layer, you only need to assign static IPs to your proxy instances.

Getting Started

Webhook Sentry runs on port 9090 by default. You can configure the address and port in the listeners section of the config.

The simplest way to run Webhook Sentry is to use the latest binary:

1. Download the latest release for your platform
2. Run the downloaded binary:

whsentry

We also have a docker image:

docker run -p 9090:9090 juggernaut/webhook-sentry:latest

You can also pin a tagged release:

docker run -p 9090:9090 juggernaut/webhook-sentry:v1.0.8

If you need to override settings, you can mount a configuration file, pass in command line flags or set environment variables. See configuration for details.

If you need prometheus metrics for the service, allow access on port 2112 with something like -p 2112:2112.

Usage

HTTP target

curl -x http://localhost:9090 http://www.google.com

HTTPS target

HTTP clients create a CONNECT tunnel when a proxy is configured and the target is a https URL. This does not give us the benefits of initiating TLS from the proxy. To get around this behavior, Webhook Sentry supports a unique way of proxying to HTTPS targets. Pass a X-WhSentry-TLS header and change the protocol to http:

curl -v -x http://localhost:9090 --header 'X-WhSentry-TLS: true' http://www.google.com

Although CONNECT is supported, I strongly recommend using the header approach to take advantage of the TLS capabilities of Webhook Sentry, like mutual TLS and robust certificate validation.

Mutual TLS

Specify clientCertFile and clientKeyFile in the configuration to enable mutual TLS:

clientCertFile: /path/to/client.pem
clientKeyFile: /path/to/key.pem

Prometheus Metrics

Point your collector to :2112 for metrics.

E.g if the proxy is running on localhost, to verify metrics are correctly exposed:

curl http://localhost:2112/metrics

AWS EKS Configuration for Static IP egress

To deploy Webhook Sentry with a static egress IP addresses in AWS EKS, you'll need a node group with an Elastic IP address:

• Create a new NAT Gateway.
• Create an Elastic IP address and assign it to your NAT Gateway. This will be your egress IP.
• Create a subnet dividing your network space.
• Create a route table linking the subnet to the NAT Gateway.
• Create a custom node group for the Webhook Sentry pods and assign it to the new subnet.
• Finally, assign your Webhook Sentry pods to that node group, and ensure your existing pods do not get assigned to it.

For redundancy, you may want to create multiple NAT Gatways, subnets, and egress IP addressses.

Protections

SSRF attack protection

Webhook Sentry blocks access to private/internal IPs to prevent SSRF attacks:

$ curl -i -x http://localhost:9090 http://127.0.0.1:3000

HTTP/1.1 403 Forbidden
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
X-Whsentry-Reason: IP 127.0.0.1 is blocked
X-Whsentry-Reasoncode: 1000
Date: Fri, 18 Sep 2020 07:15:20 GMT
Content-Length: 24

IP 127.0.0.1 is blocked

Unlike naive implementations, it also correctly checks the IP after DNS resolution. This example makes use of the 1u.ms service which can serve up DNS records using any IP we want:

$ curl -i -x http://localhost:9090 http://make-127-0-0-1-rr.1u.ms

HTTP/1.1 403 Forbidden
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
X-Whsentry-Reason: IP 127.0.0.1 is blocked
X-Whsentry-Reasoncode: 1000
Date: Fri, 18 Sep 2020 07:21:58 GMT
Content-Length: 24

IP 127.0.0.1 is blocked

DNS rebinding attack prevention

A malicious attacker can set up their DNS such that it first resolves to a valid public IP adddress, but subsequent resolutions point to private/internal IP addresses. This can be used to exploit webhook implementations that validate the resolved IP using getaddrinfo() or equivalent, then pass the original URL to a HTTP client library which resolves the host a second time. Again, let's use 1u.ms to first return a valid public IP and then the loopback IP:

$ curl -i -x http://localhost:9090 http://make-3-221-81-55-rebind-127-0-0-1-rr.1u.ms/get

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 324
Content-Type: application/json
Date: Wed, 30 Sep 2020 07:38:47 GMT
Server: gunicorn/19.9.0

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "make-3-221-81-55-rebind-127-0-0-1-rr.1u.ms",
    "User-Agent": "Webhook Sentry/0.1",
    "X-Amzn-Trace-Id": "Root=1-5f743607-afdf257ca619f90a14fc92b8"
  },
  "origin": "73.189.176.226",
  "url": "http://make-3-221-81-55-rebind-127-0-0-1-rr.1u.ms/get"
}

Mozilla CA certificate bundle

Webhook Sentry uses the latest Mozilla CA certificate bundle instead of relying on CA certificates bundled with the OS. This avoids the problem of out-of-date root CA certificates on older OS versions. See this blog post for why this is important. Notably, Stripe's webhooks were affected by this issue and took hours to fix.

On startup, Webhook Sentry checks if there is a newer version of the Mozilla CA certificate bundle than on disk, and if so, downloads it.

Additionally, by virtue of being written in Go, Webhook Sentry does not rely on OpenSSL or GnuTLS for certificate validation.

Configuration

webhook-sentry uses viper for configuration. You can use a yaml file, environment variables or command line flags to provide configuration parameters. By default, webhook-sentry looks for a file named config.yaml in the current working directory. You can specify a different file using the --config <filename flag.

The parameters documented below are in YAML, but most can also be provided as environment variables or command line flags:

• listener: An HTTP/HTTPS endpoint the proxy listens on. For HTTPS endpoints, also specify certFile and keyFile.

Example:

listener:
  type: https
  address: 127.0.0.1:9091
  certFile: /path/to/cert
  keyFile: /path/to/key

• connectTimeout: Timeout for the TCP connection to the destination host.

Default: 10s

• connectionLifetime: Maximum time a connection to the destination can be alive.

Default: 60s

• readTimeout: Maximum time a connection to the destination can remain idle.

Default: 10s

• maxResponseBodySize: Maximum size of the HTTP response body in bytes. If Content-Length is specified in the response and it is greater than this value, the connection is torn down and the response is discarded. The client receives a 502.

Default: 1048576

• clientCertFile: Path to the client certificate to present to the destination (if enabling mutual TLS)

• clientKeyFile: Path to the private key of the client certificate (if enabling mutual TLS)

• accessLog: Specifies type and file of the proxy access log. type can be either text or json. By default, text is output to stdout.

Example

accessLog:
  type: json
  file: /path/to/access.log

• proxyLog: Specifies type and file of the proxy application log. This log includes warnings and info messages related to handling proxy requests. By default, text is output to stdout.

• metrics.address: Listening address of the Prometheus metrics endpoint.

Default: :2112

Limitations

• No IPv6 support
• No TLSv1.3 support
• No Proxy authentication
• Proxy does not check client certificates (not to be confused with proxy presenting client certificate to the remote host)