A catalog for OpenAerialMap imagery
Clone or download
smit1678 Merge pull request #142 from sharkinsspatial/duplicateVertice
Handle all duplicate vertex cases for MongoDB.
Latest commit ed8af6e Aug 21, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.build_scripts Fix merge script to use correct async calls for sub docs in schema. Nov 2, 2017
bin Implement transcoding with monq (queue) Nov 27, 2017
controllers @turf/bbox-polygon's exports changed May 2, 2018
models getToken endpoint to allow users to create durable JWT tokens. Jul 10, 2018
plugins Update Facebook authentication to fetch profile picture directly. Jan 30, 2018
routes Merge branch 'develop' into duplicateVertice Aug 21, 2018
services Merge branch 'develop' into duplicateVertice Aug 21, 2018
test Merge branch 'develop' into duplicateVertice Aug 21, 2018
.dockerignore Integration test setup for uploading/processing Jun 9, 2017
.env.sample Drop MAX_WORKERS Nov 27, 2017
.eslintrc Integration test setup for uploading/processing Jun 9, 2017
.gitignore Merge entire oam-uploader-api-repo with history Jun 2, 2017
.nvmrc Sync updated/deleted meta to their S3 counterparts Aug 18, 2017
.travis.yml Add _meta.json URI during imagery conversion Aug 23, 2017
Brewfile Process with oam-dynamic-tiler-tools Jan 22, 2017
CONTRIBUTING.md initial commit Aug 6, 2015
Dockerfile Refactor Dockerfile + extend marblecutter-tools Nov 27, 2017
LICENSE Merge entire oam-uploader-api-repo with history Jun 2, 2017
Procfile Implement transcoding with monq (queue) Nov 27, 2017
README.md Update README with JWT signing instructions. Jan 18, 2018
catalog-worker.js Use OIN_REGISTER_URL as source of buckets to index Oct 2, 2017
config.js Include window.opener message to pass jwt to drone deploy. Dec 20, 2017
docker-compose.yml Docker Compose docs Nov 27, 2017
index.js Integration test setup for uploading/processing Jun 9, 2017
newrelic.js Tidy up deployment process Jun 5, 2017
npm-shrinkwrap.json Enable tile filling for mosaics May 29, 2018
package.json Merge branch 'develop' into duplicateVertice Aug 21, 2018
yarn.lock Merge branch 'develop' into duplicateVertice Aug 21, 2018

README.md

OAM Catalog API Build Status

A catalog and processor for OpenAerialMap imagery. The application indexes all metadata available within Open Imagery Network and creates an API to search and find imagery. It also takes imagery from users and converts to a format and places it in the Open Imagery Network. The API powers the frontend search tool, OAM Imagery Browser. Read the ecosystem documentation for more information about OpenAerialMap.

Full details of the API endpoints is available at: https://hotosm.github.io/oam-catalog

Installation

First you will need to have MongoDB and Node.js on your machine.

Then you will need to install the applications node module dependencies:

npm install

Usage

The API can be started with: node index.js

And the background OIN indexer can be run with: node catalog-worker.js

Development Using Docker

To start a self-contained development instance using Docker Compose, run:

docker-compose up

This will download Docker image dependencies and install code for this project into a set of containers.

Once it has started, connect to http://localhost:4000 to access the API.

The MongoDB command line interface can be run within its container like so: docker-compose exec mongo mongo

The following environment variables should be set (probably in .env; see sample.env for defaults and additional information):

# AWS credentials for writing to S3
AWS_ACCESS_KEY_ID=<redacted>
AWS_SECRET_ACCESS_KEY=<redacted>
AWS_REGION=us-east-1

# Social login
FACEBOOK_APP_ID=<redacted>
FACEBOOK_APP_SECRET=<redacted>
GOOGLE_CLIENT_ID=<redacted>
GOOGLE_CLIENT_SECRET=<redacted>

# S3 resources
OIN_BUCKET=<redacted>
OIN_BUCKET_PREFIX=<optional>
UPLOAD_BUCKET=<redacted>

# JWT Secret Key
JWT_SECRET_KEY=<redacted>

If Docker does not bind / forward ports to localhost, set HOST_TLD to the Docker hostname and use that to access the API.

To specify a bucket for indexing, modify test/fixtures/oin-buckets.json.

Instructions for generating the JWT signing key can be found here. If you find that additional environment variables are needed, please submit a pull request!

Testing

There are 3 test suites:

Unit-like tests, under test/specs
These should be isolated and fast, with as much mocking/stubbing as possible, suitable for TDD. Run with: mocha test/specs

Integration tests, under test/integration
These test the actual interaction of the API against real imagery uploads to Amazon S3. You will need AWS credendtials in your .env and, in order to use the imagery processing binaries, a running instance of the Docker image for this repo. There is a customised Docker Compose config at test/docker-compose.yml which already has all the necessary and useful changes to run on a local developer machine (such as mounting the live codebase into the running container). It can be run with docker-compose -f test/docker-compose.yml up. The actual tests can be run with mocha test/integration.

End-to-end browser tests, see https://github.com/hotosm/oam-browser
The frontend code is pinned against a specific version of this API, so it is necessary to ensure that this pinning is still reliable and also, if the version here is bumped, to note if that new version is compatible (if not then the frontend will need updating). These tests require the frontend code itself, generally you will not need to run them locally, they will be run by Travis on every commit to Github.

Transcoding using AWS Batch

AWS Batch can be used for transcoding; this enables use of elastic resources to process large quantities of imagery without requiring the API to run on an instance scaled for imagery ingestion. To enable it, set the following environment variables:

USE_BATCH=true
# Job Definition name
AWS_BATCH_JD_NAME=oam-transcode
# Job Queue name
AWS_BATCH_JQ_NAME=oam-transcoding

Configuring Batch is out of scope for this document, although a sample job definition looks like this:

{
    "jobDefinitionName": "oam-transcode",
    "type": "container",
    "parameters": {},
    "retryStrategy": {
        "attempts": 2
    },
    "containerProperties": {
        "image": "quay.io/mojodna/marblecutter-tools",
        "vcpus": 1,
        "memory": 3000,
        "command": [
            "process.sh",
            "Ref::input",
            "Ref::output",
            "Ref::callback_url"
        ],
        "jobRoleArn": "arn:aws:iam::<redacted>:role/oam-transcoder",
        "volumes": [],
        "environment": [
            {
                "name": "EFS_HOST",
                "value": "<redacted>.efs.us-east-1.amazonaws.com"
            }
        ],
        "mountPoints": [],
        "privileged": true,
        "ulimits": []
    }
}

In this sample, an Amazon Elastic File System (EFS) volume is mapped into the container through the EFS_HOST environment variable. This allows Batch jobs to handle imagery that outstrips available temporary storage on underlying instances (22GB at this writing, shared between all running tasks). If you expect to transcode smaller imagery (or don't need to support concurrent large uploads), this can be omitted.

The oam-transcoder role needs to have been created ahead of time with appropriate access to both the upload (UPLOAD_BUCKET) and storage (OIN_BUCKET) buckets.

The user / role used when running the API itself (typically an instance role if running on AWS) requires permission to submit Batch jobs, specified as:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "batch:SubmitJob"
            ],
            "Resource": "*"
        }
    ]
}

Contributing

Contributions are very welcome. Please see CONTRIBUTING.md.

License

OAM Browser is licensed under BSD 3-Clause License, see the LICENSE file for more details.