a small request demuxer for webhooks without environment affinity
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config
lib
poc
spec
.gitignore
.rspec
.ruby-gemset
.ruby-version
Gemfile
Gemfile.lock
Procfile
app.rb
config.ru
readme.md

readme.md

Webhook-Proxy

webhook-proxy sits between one or more applications and external services, and provides a stable URL to recieve webhook callbacks for those applications. It works by inspecting the contents of incoming webhook requests to map them to a backend via an identifier, which is configurable per service. It can also dynamically create identifier-to-backend mappings by inspecting the requests/responses from the application to the external service.

See ./poc for a naive example.

how is it supposed to work?

  • given that:

    • external service is configured to send webhooks to your proxy at /h/{service}
    • proxy service configuration includes a matcher to find an identifier in the request, eg.:
      • twilio matcher: req:form:To, which would match the phone number being called
      • crowdflower matcher: req:json:payload.job_id, which would match the job id
    • proxy service configuration includes a unique mapping of identifier to backend: eg:
      • twilio: Send calls to +13035557788 to my staging environment, and +13035558899 to production.
      • crowdflower: Send results of jobs 123456 to stage, and 456789 to production.
  • then, requests recieved from external service at /h/{service}

    • parse out the identifier from the request, and figure out which backend should handle the request
    • stream the request to the backend and the backend's response as the response to the service

    The application backends can also proxy requests to external services through webhook-proxy, which gives the proxy a chance to dynamically create mappings from an identifier (which is parsed from the request/response to the service) to the enviroment. This let you build environment affinity into your applications, eg., creating a crowdflower unit from stage and getting the unit_complete callback on stage, without having to send to a different job or mess with the hook configuration in the job configuration.

  • given that:

    • proxy service configuration includes a url to pass requests on to
    • proxy service configuration includes a matcher to find an identifier in either the request or response not both. eg:
      • crowdflower matcher: res:json:unit_id, which would send the callback for only this created unit to the desired backend
  • then, requests recieved from internal applications at /s/{service}(.{backend})?(/{path})?

    • send request to external service.
    • parse out the identifier from the request or response if configured.
      • if a new identifier is found, add a mapping from that identifier to the backend specified in the request
    • stream response back to application.

matchers

A matcher has to be able to match against any part of a request or response where appropriate: it's defined as:

(req/res):(location/format):path

  • req/res determines if the matcher should look in the request or response for the identifier. res only makes sense when dynamically creating mappings.
  • location/format determines where in the req/response for the data:
    • form: looks in form data
    • json: assumes the body is json, and path is a dotted path the correct data
    • header: looks in the headers
  • path is dependent on the location/format:
    • inside form, path is the name of the field
    • inside header, path is the name of the header
    • inside json, path is a dotted path to the object