REST API for the SIL App Publishing Service
PHP HTML Groovy CSS Shell Batchfile Other
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
application
build
.dockerignore
.gitattributes
.gitignore
Dockerfile
LICENSE.txt
Makefile
README.md
Vagrantfile
aws.env.encrypted
codeship-services.yml
codeship-steps.yml
common.env
docker-compose.yml
dockercfg.encrypted
local.env.dist

README.md

appbuilder-buildengine-api

This is the Web Service interface for the SIL App Publishing Service.

Build Status

Develop: Codeship Status for sillsdev/appbuilder-buildengine-api

Master: Codeship Status for sillsdev/appbuilder-buildengine-api

API Specs

https://docs.google.com/a/sil.org/document/d/1FtHLCEOvOuSnBC1ryeEUH3Vx2Y5XqF_WhpkPpjWteqs/edit?usp=sharing

Doorman: Service Provisioning

Doorman is a web application developed by GTIS that automates the workflow of approval and provisioning of services for SIL and affiliate organizations (requires Insite authentication). Examples of the services are: Smartsheet, Trello, HipChat, Jira, and CrashPlan.

For the SIL App Publishing Service, Doorman will provide the following services:

  • Readiness Check
  • Collect Publishing Project Definition
    • Project Name
    • Language Code
    • App Publishing Key
    • App Store (Currently: Wycliffe USA)
    • Future: Account for expenses and charges (if we cannot cover them)
  • Notifies Service Admin for Project Creation Approval
  • Create Git Repo in AWS CodeCommit & assigns access rights to user using App Publishing Key
  • Requests New Project in BuildEngine (includes App Publishing URL)
  • Notifies user of App Publishing URL and request them to Send/Receive project
  • Requests New Build in BuildEngine -> Build Artifact in S3
  • Builds App for Initial Store Entry
  • Notifies User for Review
  • Notifies Service Admin for Review, Approval, and creation of initial App Store Entry (must be done by human; Play Store limites number/day)
  • Request BuildEnging to Publish App to Google Play Store
  • Manages Service/Update of Project

Dependencies

In production, there is a dependency on Amazon Web Services for the following services:

Deployment

The staging version is available at https://doorman-sab.gtis.guru.

App Publishing Service

The App Publishing Service is a collection of applications that are deployed as Docker containers. Docker containers are based on Linux Containers which is a virtualization strategy that reuses Linux Kernel of the host and isolates applications in a mini Linux system. Therefore, the host for these containers must be a Linux system (which can be run in a Virtual Machine on non-Linux systems for development).

There is a set of containers:

  • BuildEngine - RESTful Web Service used by Doorman
    • web - REST API implementation that takes requests and updates database
    • cron - processes the updates to the database and interfaces with CodeBuild; updates database with status from CodeBuild

Deployment

In production, the containers are deployed to a docker container service, like Amazon ECS. For local development, these containers can be deployed to:

  • Docker CE: Linux or Mac
  • Vagrant: Windows or Linux

Note: Docker Toolbox exists for Windows, however it an issue with interactive terminal (docker -it container command). This limitation makes it difficult to debug issues during development and is not recommended.

For SIL development or staging, please contact @chris_hubbard for access to the SIL LSDev account.

Development setup instructions

APP_ENV is a variable that differentiates development, staging, and production environments. For development environments, you should use the form "development-USERNAME-MACHINE" (e.g. development-chrish-win). Whenever you see APP_ENV in the directions, please substitute with your chosen environment string. APP_ENV should be limited to alphanumeric characters, '.', '_', and '-' (CodeCommit limit).

AWS_USER_ID is a 12 digit number that you can get from AWS Support Center Whenever you see AWS_USER_ID in the directions, please substitute with the value from your account.

Clone Source Repositories

You will need to authenticate to GitHub to be able to clone the repositories. You can either use username/password with the HTTPS urls:

git clone https://github.com/sillsdev/appbuilder-buildengine-api
git clone https://github.com/sillsdev/docker-appbuilder-agent

Or you can create an SSH Key, store the private and public key in ~/.ssh, and associate the public key with your GitHub account.

mkdir -p ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa
# login to https://github.com/settings/keys/ and "New SSH Key" using ~/.ssh/id_rsa.pub
git clone git@github.com:sillsdev/appbuilder-buildengine-api
git clone git@github.com:sillsdev/docker-appbuilder-agent

If you want to create a separate SSH Key just for GitHub, you can create the SSH Key in a subdirectory of ~/.ssh and then add an entry to ~/.ssh/config (which normally doesn't exist--create an empty file) to specify the location of the key.

Host github.com
	IdentityFile ~/.ssh/github/id_rsa

Back

Create BuildEngine SSH Key

BuildEngine uses git to synchronize the git Repository with ssh keys for authentication.

  1. Create ~/.ssh/buildengine_api directory
  2. Create ssh key in ~/.ssh/buildengine_api directory (use an empty key phrase)
mkdir -p ~/.ssh/buildengine_api
chmod 700 ~/.ssh/buildengine_api
ssh-keygen -t rsa -b 4096 -f ~/.ssh/buildengine_api/id_rsa
openssl rsa -in ~/.ssh/buildengine_api/id_rsa -pubout > ~/.ssh/buildengine_api/id_rsa_pub.pem

This will also save the public key to ~/.ssh/buildengine_api/id_rsa.pub

Back

Associate BuildEngine SSH Key

BuildEngine uses SSH authentication to access the Git repository which store Jenkins job configuration. You need to associate the SSH Public Key with a user in IAM.

  1. In AWS, go to IAM service and create or select a user which will be the "Build Engine" user for your deployment
    • Note: You will need the Access Key Id and Secret Access Key for this user later.
  2. Select the Security Credentials tab
  3. Click on "Upload SSH public key" button
  4. Open ~/.ssh/buildengine_api/id_rsa.pub with a text editor and copy all of the text
  5. Paste the text into the "Upload SSH public key" page.
  6. Click on the "Upload SSH public Key" button
  7. You should see a new entry under "SSH keys for AWS CodeCommit". Save the value of "SSH Key ID" for the new uploaded key to ~/.ssh/buildengine_api/ssh_key_id.txt

Back

Create S3 Folders

BuildEngine and AppBuilder use s3-expand to extract credentials needed to access resources.

  • Create the following folder in AWS S3
sil-appbuilder-secrets/APP_ENV/buildengine_api/ssh
  • Save buildengine_api private and public key in S3
sil-appbuilder-secrets/APP_ENV/buildengine_api/ssh/id_rsa
sil-appbuilder-secrets/APP_ENV/buildengine_api/ssh/id_rsa.pub

CodeBuild is used to initiate jobs that can build the application and/or publish them to the Google Play Store.

The credentials being saved here are to support a system that will build the application using AppBuilder:

  • Create the following folders in AWS S3
sil-appbuilder-secrets/APP_ENV/jenkins/build/appbuilder_ssh
sil-appbuilder-secrets/APP_ENV/jenkins/build/google_play_store
  • Save appbuilder private and public key in S3
sil-appbuilder-secrets/APP_ENV/jenkins/build/appbuilder_ssh/id_rsa
sil-appbuilder-secrets/APP_ENV/jenkins/build/appbuilder_ssh/id_rsa.pub

AppBuilder needs additional credentials for signing APK files. These

  • Create the following folder in S3
sil-appbuilder-secrets/APP_ENV/jenkins/build/google_play_store/wycliffeusa
  • Create a wycliffeusa.keystore (using Scripture App Builder from Tools->Create New KeyStore menu)
  • When creating a keystore, there are 3 other pieces of information needed to access the keystore later.
    • Key Store Password
    • Key Alias Name
    • Key Alias Password
  • Create text files (that will be uploaded to S3) with the values entered for these 3 pieces of information.
    • ksp.txt - Key Store Password
    • ka.txt - Key Alias
    • kap.txt - Key Alias Password
  • Upload wycliffeusa.keystore, ksp.txt, ka.txt, and kap.txt to sil-appbuilder-secrets/APP_ENV/jenkins/build/google_play_store/wycliffeusa

If the credentials being saved here are to support a system that will publish the application to Google Play Store:

  • Create the following folder in S3
sil-appbuilder-secrets/APP_ENV/jenkins/publish/google_play_store/wycliffeusa
  • Upload playstore_api.json to sil-appbuilder-secrets/APP_ENV/jenkins/publish/google_play_store/wycliffeusa

Back

Give Permissions

There are 3 different users involved in accessing S3 resources

  • Build Engine
    • S3: Access secrets
    • S3: Put build artifacts
    • IAM: Create User and Groups, Manage SSH Key
    • CodeCommit: Create Repository
  • End User
    • CodeCommit: Read/Write Project data

Create the following policies:

  • S3 App Builder Secrets - extract ssh keys to access Git repository for Job DSL configuration
    • In AWS IAM Policies, Create Policy
    • Select "Create Your Own Policy"
    • Set the Policy Name to "s3-appbuilder-secrets-APPENV"
    • Paste in this text and then click on "Create Policy"
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::sil-appbuilder-secrets"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::sil-appbuilder-secrets/*"
            ]
        }
    ]
}
  • S3 App Builder Artifacts - write and delete build artifiacts accessed by end user via Doorman
    • Note: These are a too permissive. TODO: Work on minimal set.
    • In AWS IAM Policies, Create Policy
    • Select "Create Your Own Policy"
    • Set the Policy Name to "s3-appbuilder-artifacts-APPENV"
    • Paste in this text and then click on "Create Policy"
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "*"
            ],
            "Resource": [
                "arn:aws:s3:::sil-appbuilder-artifacts"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "*"
            ],
            "Resource": [
                "arn:aws:s3:::sil-appbuilder-artifacts/*"
            ]
        }
    ]
}
  • CodeCommit Repository for project data
    • In AWS IAM Policies, Create Policy
    • Select "Create Your Own Policy"
    • Set the Policy Name to "codecommit-projects-APPENV"
    • Paste in this text and then click on "Create Policy"
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "codecommit:GetBranch",
                "codecommit:GitPull",
                "codecommit:GitPush",
                "codecommit:ListBranches"
            ],
            "Resource": [
                "arn:aws:codecommit:us-east-1:AWS_USER_ID:*"
            ]
        }
    ]
}
  • Project creations and building
    • In AWS IAM Policies, Create Policy
    • Select "Create Your Own Policy"
    • Set the Policy Name to "projects-creation-and-building-APPENV"
    • Paste in this text and then click on "Create Policy"
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:CreateGroup",
                "iam:AddUserToGroup",
                "iam:ListSSHPublicKeys",
                "iam:GetSSHPublicKey",
                "iam:UploadSSHPublicKey",
                "iam:GetUser",
                "iam:CreateUser",
                "iam:GetGroup",
                "iam:PutGroupPolicy",
                "iam:GetRole",
                "iam:CreateRole",
                "iam:AttachRolePolicy",
                "iam:PassRole",
                "codebuild:CreateProject",
                "codebuild:BatchGetProjects",
                "codebuild:BatchGetBuilds",
                "codebuild:StartBuild"
            ],
            "Resource": ""
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "codecommit:GetRepository",
                "codecommit:CreateRepository",
                "codecommit:DeleteRepository"
            ],   
            "Resource": "arn:aws:codecommit::*:*"
        }
    ]
}
  • CodeBuild Base Policy for Building
    • In AWS IAM Policies, Create Policy
    • Select "Create Your Own Policy"
    • Set the Policy Name to "codebuild-basepolicy-build_app-APPENV"
    • Paste in this text and then click on "Create Policy"
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Resource": [
        "arn:aws:logs:us-east-1:AWS_USER_ID:log-group:/aws/codebuild/build_app",
        "arn:aws:logs:us-east-1:AWS_USER_ID:log-group:/aws/codebuild/build_app:*"
      ],
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ]
    },
    {
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::codepipeline-us-east-1-*"
      ],
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:GetObjectVersion"
      ]
    }
  ]
}
  • CodeBuild Base Policy for Publishing
    • In AWS IAM Policies, Create Policy
    • Select "Create Your Own Policy"
    • Set the Policy Name to "codebuild-basepolicy-publish_app-APPENV"
    • Paste in this text and then click on "Create Policy"
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Resource": [
        "arn:aws:logs:us-east-1:AWS_USER_ID:log-group:/aws/codebuild/publish_app",
        "arn:aws:logs:us-east-1:AWS_USER_ID:log-group:/aws/codebuild/publish_app:*"
      ],
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ]
    },
    {
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::codepipeline-us-east-1-*"
      ],
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:GetObjectVersion"
      ]
    }
  ]
}

Attach the following policies to the "Build Engine" user:

  • s3-appbuilder-secrets-APP_ENV
  • s3-appbuilder-artifacts-APP_ENV
  • projects-creation-and-building-APP_ENV

Back

Build Engine Configuration

In the directory where you cloned appbuilder-buildengine-api, do the following:

  • copy local.env.dist to local.env and update the variables. Replace SOME_KEY with your chosen environment string.
  • set EXPAND_S3_KEY and EXPAND_S3_SECRET to the Access Key Id and Secret Access Key of the Build Engine User.
  • set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to the Access Key Id and Secret Access Key of the Build Engine User. itory.
  • set AWS_USER_ID to the value from [AWS Support Center](https://console.aws.amazon.c
  • set BUILD_ENGINE_GIT_SSH_USER to the value of ~/.ssh/buildengine_api/ssh_key_id.txt
  • set APPBUILDER_GIT_SSH_USER to the value stored in ~/.ssh/appbuilder/ssh_key_id.txt
  • set BUILD_ENGINE_REPO_URL to the ssh url saved from creating the Build Engine reposom/support/home)
  • set API_ACCESS_TOKEN to some unique string to your environment.
    • This will be used during testing for HTTP Bearer Token Authentication.
    • See RFC6750 for more details.

Back

Deploy Containers

  • If deploying to native Docker on Linux or Docker Toolbox on Mac, set environment variable DOCKER_UIDGID
echo 'export DOCKER_UIDGID="$(id -u):$(id -g)"' >> ~/.profile
source ~/.profile

Docker CE: Linux

docker-compose up -d

Vagrant: Windows or Linux

vagrant up
vagrant ssh

Docker CE: Mac

docker-compose up -d

Back

Testing API

Doorman interacts with BuildEngine using a RESTful Web Interface. This can be simlated with a Google Chrome Extension called Advanded REST Client.

Give User Write Permissions to CodeCommit

For development deployment, we will need a project to test with. Earlier we created a codecommit-project-APP_ENV policy that gives the appbuilder user read/write permissions to any repository with the name prefix of projects-APP_ENV-.

  • Add the following text to ~/.ssh/config to use the appbuilder ssh key to commit the project.
Host git-codecommit.*.amazonaws.com
    User USERNAME
    IdentityFile ~/.ssh/appbuilder/id_rsa
  • Change USERNAME to be the value of ~/.ssh/appbuilder/ssh_key_id.txt

Create AppBuilder CodeCommit Repository

We use CodeCommit Git repositories for storing the data for the project. The Doorman service provisioning application automates the creation of the repository and setting the security. We need to simulate that here.

  1. Go to AWS CodeCommit
  2. Click "Create new repository"
  3. Set the repository name to projects-APP_ENV-PROJECT_NAME
  4. Click on the "Clone URL" dropdown and select SSH. Copy the URL for the next step.

Commit Project

  • Copy a project to use for testing
    • In a ~/App\ Builder/Scripture\ Apps/App\ Projects/, copy some project to projects-APP_ENV-PROJECT_NAME
  • In a command window, do the following:
    • cd ~/App\ Builder/Scripture\ Apps/App\ Projects/projects-APP_ENV-PROJECT_NAME
    • git init
    • git add .
    • git commit -m "Initial Revision"
    • git remote add origin URL
    • git push origin master

Add Job to BuildEngine

The base URL of BuildEngine depends on how you deployed BuildEngine

  • Start Google Chrome and Switch to "Apps" in Google Chrome and select ARC
  • set URL (first field) to http://BUILDENGINE_HOST/job (BUILDENGINE_HOST depends on how BuildEngine was deployed)
    • vagrant (Windows or Linux): 192.168.70.121
    • native docker (Linux): localhost
    • docker-machine (Mac): run docker-engine ip default to determine IP
      • most likely: 192.168.99.100
  • set VERB to POST
  • set Headers to (replace YOUR_API_TOKEN)
Accept: application/json
Authorization: Bearer YOUR_API_TOKEN
Content-Type: application/json
  • set Payload to (replace YOUR_REQUEST_ID with a unique value like the return value of date +%s)
{
    "request_id" : "YOUR_REQUEST_ID",
    "git_url" : "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/projects-APP_ENV-PROJECT_NAME",
    "app_id" : "scriptureappbuilder",
    "publisher_id" : "wycliffeusa"
}
  • Click Send

[^1]: We use for the Git Repository since AWS only charges for $1/month/active-user. For many projects, the user requesting the project will be active for a short period of time and then not modify the project repository for a long time. It would be possible to use another Git service (like GitHub) to host the project repository, but will require to make a code change in application/common/model/Job.php to allow other repository urls).