Main repository for Data.gov's stack deployment
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.circleci Use kitchen for local development Aug 16, 2018
ansible Update paths relative to .kitchen.yml Sep 20, 2018
docker Getting started on kitchen-docker for ansible (#202) Jul 18, 2017
docs Update ROADMAP Aug 14, 2018
files Add playbook for one-off patch Aug 7, 2018
packer Add tags to packer built amis Oct 20, 2017
terraform Add clean task to remove plan Sep 4, 2018
.editorconfig Moving encrypted files to inventories, Lint fixes for CircleCi (#207) Jul 18, 2017
.gitignore Terraform for sandbox development/testing Sep 1, 2018
.kitchen.vagrant.yml Getting started on kitchen-docker for ansible (#202) Jul 18, 2017
DEPENDENCIES.md Update DEPENDENCIES.md Nov 29, 2017
Gemfile Getting started on kitchen-docker for ansible (#202) Jul 18, 2017
Gemfile.lock Bump dependencies Aug 15, 2018
Jenkinsfile Add the Jenkinsfile Oct 20, 2017
Makefile Move kitchen to ansible directory Sep 20, 2018
README.md Update README with development commands Aug 16, 2018
Vagrantfile simplify ansible hosts file Aug 2, 2016
ansible.cfg Reverting default module_name to command for crons Jul 19, 2017
container.yml Moving encrypted files to inventories, Lint fixes for CircleCi (#207) Jul 18, 2017
inventory re-work wordpress playbook Nov 9, 2016
meta.yml Moving encrypted files to inventories, Lint fixes for CircleCi (#207) Jul 18, 2017
requirements.txt Update README with development commands Aug 16, 2018
secrets.yml.save Merge fluentd and Wordpress work into repo Sep 12, 2016
staging.yml Forgetting to git add on folder rename Jul 22, 2016
test.yml Getting started on kitchen-docker for ansible (#202) Jul 18, 2017
travis.yml Getting started on kitchen-docker for ansible (#202) Jul 18, 2017

README.md

Data.gov Deploy

CircleCI

This main repository for Data.gov's stack deployment onto AWS Infrastructure. The responsitory is broken into the following roles all created/provisioned using Ansible:

Included in this Repository:

  • Software
    • Data.gov (Wordpress)
    • Catalog.data.gov (CKAN 2.3)
    • Inventory.data.gov (CKAN 2.5)
    • Labs.data.gov/CRM (Open311 CRM)
    • Labs.data.gov/Dashboard (Project Open Data Dashboard)
  • Security
    • Baseline OS Hardening
    • GSA IT Security Agents
    • Fluentd (Logging)
    • New Relic (Infrastructure Monitoring)
    • New Relic (Application Performance Monitoring)
    • Trendmicro (OSSEC-HIDS)
    • OSQuery (TBD)

Project Status

See our Roadmap.

Provision Infrastructure

Moved to datagov-infrastructure

Requirements for Software Provisioning

  • Ansible > 1.10
  • SSH access (via keypair) to remote instances
  • ansible-secret.txt: export ANSIBLE_VAULT_PASSWORD_FILE=~/ansible-secret.txt
  • run all provisioning/app deployment commands from repo's ansible folder
  • to update ansible/roles/vendor roles run there: ansible-galaxy install -r requirements.yml
  • {{ inventory }} can be:
    • inventories/staging/hosts
    • inventories/production/hosts
    • inventories/local/hosts

Provision apps

cd ansible

ansible-playbook --help

See example(s) below

Wordpress:

provision vm & deploy app: ansible-playbook datagov-web.yml -i {{ inventory }} --tags="provision" --limit wordpress-web

deploy app: ansible-playbook datagov-web.yml -i {{ inventory }} --tags="deploy" --limit wordpress-web

deploy rollback: ansible-playbook datagov-web.yml -i {{ inventory }} --tags="deploy-rollback" --limit wordpress-web

  • You can override branch to be deployed via -e project_git_version=develop

    e.g. ansible-playbook datagov-web.yml -i inventories/staging/hosts --tags=deploy --limit wordpress-web -e project_git_version=develop

Dashboard

provision vm & deploy app: ansible-playbook dashboard-web.yml -i {{ inventory }} --tags="provision" --limit dashboard-web

deploy app: ansible-playbook dashboard-web.yml -i {{ inventory }} --tags="deploy"

deploy rollback: ansible-playbook dashboard-web.yml -i {{ inventory }} --tags="deploy-rollback"

CRM

provision vm & deploy app: ansible-playbook crm-web.yml -i {{ inventory }} --tags="provision" --limit crm-web

deploy app: ansible-playbook crm-web.yml -i {{ inventory }} --tags="deploy"

deploy rollback: ansible-playbook crm-web.yml -i {{ inventory }} --tags="deploy-rollback"

Catalog:

provision vm - web: ansible-playbook catalog.yml -i {{ inventory }} --tags="frontend,ami-fix,bsp" --skip-tags="solr,db,cron" --limit catalog-web

provision vm - harvester: ansible-playbook catalog.yml -i {{ inventory }} --tags="harvester,ami-fix,bsp" --skip-tags="apache,solr,db,saml2" --limit catalog-harvester

provision vm - solr: ansible-playbook catalog.yml -i {{ inventory }} --tags="solr,ami-fix,bsp" --limit solr

Inventory

provision vm && deploy app: ansible-playbook inventory.yml -i {{ inventory }} --skip-tags="solr,db,deploy-rollback" --limit inventory-web

provision vm - solr: ansible-playbook inventory.yml -i {{ inventory }} --tags="solr,ami-fix,bsp" --limit solr

Jekyll

provision vm && deploy app: ansible-playbook jekyll.yml -i {{ inventory }} --limit jekyll-web

ElasticSearch

provision vm && deploy app: ansible-playbook elasticsearch.yml -i {{ inventory }}

Kibana

provision vm && deploy app: ansible-playbook kibana.yml -i {{ inventory }}

EFK nginx

provision vm && deploy app: ansible-playbook efk_nginx.yml -i {{ inventory }}

Common:

install the trendmicro agent: ansible-playbook trendmicro.yml -i {{ inventory }}

Add SecOps user: ansible-playbook secops.yml -i {{ inventory }}

Upgrade ubuntu VMs:

ansible all -m shell -a "apt-get update && apt-get dist-upgrade" --sudo

ansible all -m shell -a "service tomcat6 restart" --sudo

ansible all -m shell -a "service ntp restart" --sudo

ansible all -m shell -a "/usr/bin/killall dhclient && dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0" --sudo

Troubleshooting:

dpkg errors:

sed -i '/postdrop/d' /var/lib/dpkg/statoverride

sed -i '/ssl-cert/d' /var/lib/dpkg/statoverride

ntpd issues: apt-get remove ntp && apt-get purge ntp && apt-get autoclean && apt-get autoremove

Unable to resolve host IP: echo 127.0.0.1 $(hostname) >> /etc/hosts

Development

Install the dependencies (from a python virtualenv).

$ make setup

Run the playbooks locally.

$ make test

Run a single suite.

$ bundle exec kitchen test catalog

Log into the instance to debug.

$ bundle exec kitchen login catalog

Re-run the playbook from a particular step.

$ ANSIBLE_EXTRA_FLAGS='--start-at-task="software/ckan/apache : make sure postgresql packages are installed"' bundle exec kitchen converge catalog

Lint your work.

$ make lint

Refer to kitchen commands for more information.