HTTP service that accepts a collection of "components", fans-out requests and returns aggregated content
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
aggregator
config
requester
tasks
.gitignore
Dockerfile
LICENSE
README.md
bootstrap.sh
curl-format.txt
glide.lock
glide.yaml
main.go
ssh.exp

README.md

Go-Requester

HTTP service that accepts a collection of "components"
then fans-out requests and returns aggregated content

Summary

  • Components should be defined in a YAML page configuration file
  • Components are requested concurrently via goroutines
  • Components can be marked as "mandatory" (if they fail, the request summary is set to "failure")

Example Page Config

components:
  - id: google
    url: http://google.com
  - id: integralist
    url: http://integralist.co.uk
    mandatory: true
  - id: not-found
    url: http://httpstat.us/404
  - id: timeout
    url: http://httpstat.us/408
  - id: server-error
    url: http://httpstat.us/500
  - id: service-unavailable
    url: http://httpstat.us/503

Note:
example config provided as part of this repo
./config/page.yaml

Example JSON Output

{
  "summary": "success",
  "components": [
    {
      "id": "google",
      "status": 200,
      "body": "<!doctype html> ... </html>",
      "mandatory": false
    },
    {
      "id": "integralist",
      "status": 200,
      "body": "<!doctype html> ... </html>",
      "mandatory": true
    },
    {
      "id": "slooow",
      "status": 408,
      "body": "Get http://localhost:3000/pugs: net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
      "mandatory": false
    }
    {
      "id": "not-found",
      "status": 404,
      "body": "404 Not Found",
      "mandatory": false
    }
  ]
}

Note: the toplevel summary key's value will be failure if any mandatory components fail

Installing Vendored Dependencies

You'll need Glide installed:

go get github.com/Masterminds/glide

If using Go 1.6 you're OK, otherwise you'll need to set the vendor experiment:

export GO15VENDOREXPERIMENT=1

To install the dependencies found in the Glide.lock file:

glide install

Run Tests

go test $(glide novendor)

Build Application

go build

Once the application is built and installed into your binary path, you can run it:

go-requester <path/to/your/page/config>

Running Locally

There are two ways to run this application locally:

  1. Host Machine
  2. Docker

Host Machine

go run main.go ./config/page.yaml

Alternative: godo server --watch

Once the application is running, view: http://localhost:8080/

If you want to see how latency/slow responses effect the application, then also try running: https://github.com/Integralist/go-slow-http-server which was specifically designed to be used alongside go-requester for testing purposes.

Docker

docker build -t gorequester .
docker run --rm --name gr -v "$PWD":/go/src/app -p 5000:8080 gorequester

Once the application is running, view: http://<docker-machine-ip>:5000/

To debug the container:

docker exec -it <container_id> /bin/bash
docker run -it --entrypoint /bin/bash -v "$PWD":/go/src/app -p 5000:8080 gorequester

If you specify a private repo as a dependency, then you'll need to pass in your SSH credentials:

docker run \
  -it \
  -v "$PWD":/go/src/app \
  -v "$HOME/.ssh/github_bbc_rsa":/.ssh/github_rsa \
  -p 5000:8080 \
  gorequester ssh_setup

Notice we've passed ssh_setup. Our bootstrap script (which runs inside the Docker container) will identify when this argument is provided and subsequently load up the SSH agent within the container.

Because we've added the -it flag we're able to manually type in our SSH private key's passphrase (you do use a passphrase don't you?).

Expect

Having to type in your SSH passphrase can be annoying an restrict your ability to automate. Although I personally wouldn't advise it, you can make a change to the bootstrap script and have it use the ssh.exp script included in this repository.

Using this Expect script will mean you don't have to manually enter the passphrase for your SSH private key. The MASSIVE downside to this process is that you'll need to edit the ssh.exp script to include your passphrase. This means your secret passphrase has now been codified into the file.

If your laptop is compromised or you commit the change accidentally, then you're in serious trouble. So really... don't use it unless you're really sure you know what you're doing.

You'll also need to comment out the line ssh-add /.ssh/github_rsa from the bootstrap script and replace it with /ssh.exp.

Curl Timings

We can verify the performance of this application using curl timings, like so:

curl -w "@curl-format.txt" -o /dev/null -s http://localhost:8080/

Note: I've included a curl-format.txt file within the repo

TODO

  • Refactor code so that certain aspects are loaded from other packages
  • Add logic for loading page config remotely
  • Dynamically change port number when run as binary
  • Tests!

Licence

The MIT License (MIT)

Copyright (c) 2015 Mark McDonnell