Main repository for Data.gov's stack deployment
Clone or download
Latest commit 0badc31 Dec 6, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Merge pull request #485 from GSA/add-unattended-upgrade-role Dec 7, 2018
ansible Merge pull request #485 from GSA/add-unattended-upgrade-role Dec 7, 2018
docker Revert docker-build work Nov 6, 2018
docs Update ROADMAP Aug 14, 2018
files Adding clarification and new harvest source Oct 24, 2018
packer Add tags to packer built amis Oct 20, 2017
.editorconfig Moving encrypted files to inventories, Lint fixes for CircleCi (#207) Jul 18, 2017
.gitignore Add ckan-native-login role and molecule tests Oct 3, 2018
.kitchen.vagrant.yml Getting started on kitchen-docker for ansible (#202) Jul 18, 2017
CONTRIBUTING.md Propose Git Flow with relationship to deployment Nov 10, 2018
DEPENDENCIES.md Update DEPENDENCIES.md Nov 29, 2017
Gemfile fix: Gemfile.lock & Gemfile to reduce vulnerabilities Sep 30, 2018
Gemfile.lock fix: Gemfile.lock & Gemfile to reduce vulnerabilities Sep 30, 2018
Jenkinsfile Add the Jenkinsfile Oct 20, 2017
LICENSE.md Create LICENSE.md Oct 17, 2018
Makefile Merge pull request #485 from GSA/add-unattended-upgrade-role Dec 7, 2018
README.md Update README Nov 6, 2018
Vagrantfile simplify ansible hosts file Aug 2, 2016
ansible.cfg Reverting default module_name to command for crons Jul 19, 2017
container.yml Moving encrypted files to inventories, Lint fixes for CircleCi (#207) Jul 18, 2017
inventory re-work wordpress playbook Nov 9, 2016
meta.yml Moving encrypted files to inventories, Lint fixes for CircleCi (#207) Jul 18, 2017
requirements-dev.txt Lock pytest to 3.9 due to bug Nov 20, 2018
requirements.txt Lock pytest to 3.9 due to bug Nov 20, 2018
secrets.yml.save Merge fluentd and Wordpress work into repo Sep 12, 2016
staging.yml Forgetting to git add on folder rename Jul 22, 2016
test.yml add unattended-upgrades to circleci kitchen tests Dec 6, 2018
travis.yml Getting started on kitchen-docker for ansible (#202) Jul 18, 2017

README.md

Data.gov Deploy

CircleCI

This main repository for Data.gov's stack deployment onto AWS Infrastructure. The responsitory is broken into the following roles all created/provisioned using Ansible:

Included in this Repository:

  • Software
    • Data.gov (Wordpress)
    • Catalog.data.gov (CKAN 2.3)
    • Inventory.data.gov (CKAN 2.5)
    • Labs.data.gov/CRM (Open311 CRM)
    • Labs.data.gov/Dashboard (Project Open Data Dashboard)
  • Security
    • Baseline OS Hardening
    • GSA IT Security Agents
    • Fluentd (Logging)
    • New Relic (Infrastructure Monitoring)
    • New Relic (Application Performance Monitoring)
    • Trendmicro (OSSEC-HIDS)
    • OSQuery (TBD)

Project Status

See our Roadmap.

Provision Infrastructure

Moved to datagov-infrastructure

Requirements for Software Provisioning

  • Ansible > 1.10
  • SSH access (via keypair) to remote instances
  • ansible-secret.txt: export ANSIBLE_VAULT_PASSWORD_FILE=~/ansible-secret.txt
  • run all provisioning/app deployment commands from repo's ansible folder
  • to update ansible/roles/vendor roles run there: ansible-galaxy install -r requirements.yml
  • {{ inventory }} can be:
    • inventories/staging/hosts
    • inventories/production/hosts
    • inventories/local/hosts

Provision apps

cd ansible

ansible-playbook --help

See example(s) below

Wordpress:

provision vm & deploy app: ansible-playbook datagov-web.yml -i {{ inventory }} --tags="provision" --limit wordpress-web

deploy app: ansible-playbook datagov-web.yml -i {{ inventory }} --tags="deploy" --limit wordpress-web

deploy rollback: ansible-playbook datagov-web.yml -i {{ inventory }} --tags="deploy-rollback" --limit wordpress-web

  • You can override branch to be deployed via -e project_git_version=develop

    e.g. ansible-playbook datagov-web.yml -i inventories/staging/hosts --tags=deploy --limit wordpress-web -e project_git_version=develop

Dashboard

provision vm & deploy app: ansible-playbook dashboard-web.yml -i {{ inventory }} --tags="provision" --limit dashboard-web

deploy app: ansible-playbook dashboard-web.yml -i {{ inventory }} --tags="deploy"

deploy rollback: ansible-playbook dashboard-web.yml -i {{ inventory }} --tags="deploy-rollback"

CRM

provision vm & deploy app: ansible-playbook crm-web.yml -i {{ inventory }} --tags="provision" --limit crm-web

deploy app: ansible-playbook crm-web.yml -i {{ inventory }} --tags="deploy"

deploy rollback: ansible-playbook crm-web.yml -i {{ inventory }} --tags="deploy-rollback"

Catalog:

provision vm - web: ansible-playbook catalog.yml -i {{ inventory }} --tags="frontend,ami-fix,bsp" --skip-tags="solr,db,cron" --limit catalog-web

provision vm - harvester: ansible-playbook catalog.yml -i {{ inventory }} --tags="harvester,ami-fix,bsp" --skip-tags="apache,solr,db,saml2" --limit catalog-harvester

provision vm - solr: ansible-playbook catalog.yml -i {{ inventory }} --tags="solr,ami-fix,bsp" --limit solr

Inventory

provision vm && deploy app: ansible-playbook inventory.yml -i {{ inventory }} --skip-tags="solr,db,deploy-rollback" --limit inventory-web

provision vm - solr: ansible-playbook inventory.yml -i {{ inventory }} --tags="solr,ami-fix,bsp" --limit solr

Jekyll

provision vm && deploy app: ansible-playbook jekyll.yml -i {{ inventory }} --limit jekyll-web

ElasticSearch

provision vm && deploy app: ansible-playbook elasticsearch.yml -i {{ inventory }}

Kibana

provision vm && deploy app: ansible-playbook kibana.yml -i {{ inventory }}

EFK nginx

provision vm && deploy app: ansible-playbook efk_nginx.yml -i {{ inventory }}

Common:

install the trendmicro agent: ansible-playbook trendmicro.yml -i {{ inventory }}

Add SecOps user: ansible-playbook secops.yml -i {{ inventory }}

Upgrade ubuntu VMs:

ansible all -m shell -a "apt-get update && apt-get dist-upgrade" --sudo

ansible all -m shell -a "service tomcat6 restart" --sudo

ansible all -m shell -a "service ntp restart" --sudo

ansible all -m shell -a "/usr/bin/killall dhclient && dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0" --sudo

Troubleshooting:

dpkg errors:

sed -i '/postdrop/d' /var/lib/dpkg/statoverride

sed -i '/ssl-cert/d' /var/lib/dpkg/statoverride

ntpd issues: apt-get remove ntp && apt-get purge ntp && apt-get autoclean && apt-get autoremove

Unable to resolve host IP: echo 127.0.0.1 $(hostname) >> /etc/hosts

Development

Install the dependencies (from a python virtualenv).

$ make setup

Run the playbooks locally.

$ make test

You can set the concurrency parameter with make's -j parameter.

$ make -j4 test

This runs all the suites, both molecule and kitchen tests. See below for more on how to work with individual suites. Both suites rely on docker for running tests within containers.

Lint your work.

$ make lint

Testing with molecule

Molecule is the preferred test suite for testing roles. Playbooks can be tested by including them in the molecule playbook.

Molecule is modular, so you must cd to the directory of the role you are testing.

$ cd roles/software/ckan/native-login
$ molecule test

During development, you'll want to run only the converge playbook to avoid creating/destroying the container every time.

$ molecule converge

If you have multiple scenarios, you can specify them individually.

$ moelcule test -s <scenario>

Testing with kitchen

We use Kitchen for testing playbooks, although we are moving suites to molecule.

Run a single suite.

$ cd ansible
$ bundle exec kitchen test catalog

Log into the instance to debug.

$ cd ansible
$ bundle exec kitchen login catalog

Re-run the playbook from a particular step.

$ ANSIBLE_EXTRA_FLAGS='--start-at-task="software/ckan/apache : make sure postgresql packages are installed"' bundle exec kitchen converge catalog

Refer to kitchen commands for more information.