A small demo showing off an FME standalone engine running in a docker container accepting jobs
PowerShell Shell Python
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
workspaces
FMEEngineClient.py
README.md
docker-compose.yml
start_containers.ps1
start_containers.sh
start_containers_noweb.ps1
start_containers_noweb.sh
stop_containers.ps1
stop_containers.sh
submit_job.ps1
submit_job.sh
submit_lots_of_jobs.ps1
submit_lots_of_jobs.sh

README.md

FME Standalone Engine

Overview

This is a demo showing an FME Engine running in a docker container listening on a socket for jobs. The container mounts a directory containing a single workspace called AirQualityReport.fmw. This workspace has a published parameter Location which is the name of a city. It outputs an html page with the latest data on the air quality for that city as well as a google map containing data points on air quality throughout the city.

The FME Engine listens on a port awaiting jobs. A python script is used to send jobs to that engine which will then run the translation using the latest data and output an updated html page.

The html page is then served by an apache server that is also running in a docker container.

Prerequisites

  • Docker 1.12
  • Python

This requires docker 1.12 as it uses the new service mode. This currently has release candidates but is not officially released. This will also run on the Docker for Mac or Docker for WIndows Beta as they use Docker 1.12.

Usage

Start the Docker containers. On Linux and mac, you can run the start_containers.sh script. On Windows you can run start_containers.ps1. This will launch two docker services. One is the FME Engine listening for jobs on port 7777, and one is the apache server serving html files on port 80.

Submit a job to the FME engine. You can submit a job by running submit_job.sh on Linux and Mac or submit_job.ps1 on Windows. This will run the python script FMEEngineClient.py to submit a job to the FME Engine listening on localhost on port 7777 for the city of Vancouver. After a few seconds you should see a message returned with "Translation Successful". The FME Engine will have read the latest data from a web service and generated an HTML page and placed it in the results folder which is also the directiry being served by apache.

You can then open a browser to your machine on port 80. You should see an index listing with a single html file called AirQualityReportVancouver.html that was generated by the engine. Click on it to view the report.

Scaling

Now imagine we want to generate a report for several cities at the same time. We will need more than one engine to take care of this unless we want to wait a while.

Scale the FME Engines by running the command docker service scale fme_engine=10. This will launch more FME Engine containers waiting for jobs. They are all listening on the same port, but Docker will take care of routing requests to the local machine on that port to each of the containers using a round robin scheduling strategy.

Run the script submit_lots_of_jobs.sh (or submit_lots_of_jobs.ps1 on windows). This will submit around 30 jobs all at once to that port. The engine containers will run the jobs in parallel and will each generate a report.

Refresh the page being served on your localhost to see the additional reports that were generated. Click on them to view.

These containers are all running on your single host, but it isn't that much of a stretch to imagine these containers spread over multiple hosts, able to be scaled when required.

Shutting everything down.

After you are done you can stop the services by running stop_containers.sh or stop_containers.ps1.