Automatically identify potential story leads
JavaScript CSS Other
Fetching latest commit…
Cannot retrieve the latest commit at this time.


Automatically identify potential story leads.

Newsagent lets you easily create bots to periodically gather data that you are interested in, and then alert you to anything that changes.


Firstly you'll need to clone this repository. You will also need Docker and Docker Machine.

Next we need to create configuration files based on existing example files. The first is for the Runner component which is responsible for each agent running periodically and sending emails when differences are detected. To do this first copy the example file:

$ cp runner/config.example.json runner/config.json

Then edit the runner/config.json file, and set the email portion -- this should be in Nodemailer format. You can use Gmail or other services but the easiest choice is to get a free account at an email delivery service such as Mailgun.

The second configuration file is for the Registry, which creates and manages the agents. Copy the example file for this:

$ cp registry/config.example.json registry/config.json

The last configuration file is for the interface:

$ cp interface/config.example.json interface/config.json

In these cases nothing needs to be added to the file.

On your local machine

This requires Virtualbox. Create a new virtual machine, and open the necessary ports:

$ docker-machine create -d virtualbox newsagent
$ eval "$(docker-machine env newsagent)"
$ VBoxManage controlvm newsagent natpf1 'registry,tcp,,4001,,4001'
$ VBoxManage controlvm newsagent natpf1 'interface,tcp,,4000,,4000'

You can now build and run Newsagent:

$ docker image build -t newsagent .
$ docker container run -dv /var/run/docker.sock:/var/run/docker.sock --name newsagent -p 4001:4001 -p 4000:4000 newsagent

Newsagent should now be available at http://localhost:4000/.

On an AWS machine

This requires the AWS CLI to be installed and configured. Create a new AWS EC2 machine, and open the necessary ports:

$ docker-machine create -d amazonec2 \
    --amazonec2-region 'eu-west-1' \
    --amazonec2-instance-type 't2.medium' \
$ eval "$(docker-machine env newsagent)"
$ NEWSAGENT_GROUP=$(aws ec2 describe-instances --output text --query 'Reservations[].Instances[?Tags[?Value==`newsagent`]] | [][].SecurityGroups[0].GroupId')
$ aws ec2 authorize-security-group-ingress --group-id $NEWSAGENT_GROUP --protocol tcp --port 4001 --cidr
$ aws ec2 authorize-security-group-ingress --group-id $NEWSAGENT_GROUP --protocol tcp --port 4000 --cidr

This uses a t2.medium machine, which is pretty much the minimum. If you are expecting to have many long-running agents you may need a more powerful/expensive machine.

You can now build and run Newsagent:

$ docker image build -t newsagent .
$ docker container run -dv /var/run/docker.sock:/var/run/docker.sock --name newsagent -p 4001:4001 -p 4000:4000 newsagent

Newsagent should now be available at your AWS machine on port 4000.


To update to a new version, first select our Newsagent stack:

$ eval "$(docker-machine env newsagent)"

Then wipe out all existing containers and images:

$ docker container stop $(docker container ls -q)
$ docker system prune -f

Now just follow the respective instructions above to build and run the new version.


To stop Newsagent run:

$ docker container stop newsagent
$ docker container rm newsagent
$ docker image rm newsagent

To shut down the machine it ran on:

$ docker-machine stop newsagent
$ docker-machine rm newsagent


If you are developing Newsagent it is much easier to run it outside of Docker.

You will need Node, Docker, and Docker Machine.

Use Docker Machine to create a new machine with the Virtualbox driver as above, only use a different name -- 'default' is recommended. You will need to open port 4000 and 4001 as above too.

As in the deployment instructions, you will need a config.json file for the Runner and the Registry. However, the Registry configuration needs to point to your local Docker machine. To do this, edit the dockerHosts portion to include something like this:

    "host": "",
    "port": "2376",
    "certPath": "<PATH TO YOUR HOME DIRECTORY>/.docker/machine/machines/<NAME OF YOUR DOCKER MACHINE>"

You can find the host and port of your Docker Machine by running docker-machine ls.

Next, install the dependencies for Runner and the Registry:

$ cd runner && npm install
$ cd registry && npm install

From the registry directory, you can then run:

$ node Start

And from the interface directory the pages can just be served statically, such as with Python:

$ python -m SimpleHTTPServer 4000

You then should be able to access the Newsagent interface on port 4000, and the Registry API on port 4001.

Similar tools