Setup and docs improvements

Puppetfile

@@ -32,6 +32,7 @@ mod "puppetlabs/aws", :latest
 mod "jdowning/rbenv", :latest
 mod "trlinkin/noop", :latest
 mod "puppetlabs/catalog_preview", :latest
+mod "puppet/archive", :latest
 
 # Used by profile::firewall::simple_nat
 # mod "example42/sysctl", :latest
@@ -44,64 +45,9 @@ mod "puppetlabs/catalog_preview", :latest
 # Used by profile::mongo
 # mod 'puppetlabs/mongodb', :latest
 
-# User by profile::sudo::sudoers
+# Used by profile::sudo::sudoers
 # mod 'saz/sudo', :3.1.0
 
-# Older 2.x example42 component modules
-#mod'example42/activemq',
-#  :git => 'https://github.com/example42/puppet-activemq'
-#mod'example42/apt',
-#  :git => 'https://github.com/example42/puppet-apt'
-#mod'example42/autofs',
-#  :git => 'https://github.com/example42/puppet-autofs'
-#mod'example42/git',
-#  :git => 'https://github.com/example42/puppet-git'
-#mod'example42/hosts',
-#  :git => 'https://github.com/example42/puppet-hosts'
-#mod'example42/iptables',
-#  :git => 'https://github.com/example42/puppet-iptables'
-#mod'example42/java',
-#  :git => 'https://github.com/example42/puppet-java'
-#mod'example42/jboss',
-#  :git => 'https://github.com/example42/puppet-jboss'
-#mod'example42/limits',
-#  :git => 'https://github.com/example42/puppet-limits'
-#mod'example42/logrotate',
-#  :git => 'https://github.com/example42/puppet-logrotate'
-#mod'example42/mariadb',
-#  :git => 'https://github.com/example42/puppet-mariadb'
-#mod'example42/mysql',
-#  :git => 'https://github.com/example42/puppet-mysql'
-#mod'example42/network',
-#  :git => 'https://github.com/example42/puppet-network'
-#mod'example42/nrpe',
-#  :git => 'https://github.com/example42/puppet-nrpe'
-#mod'example42/perl',
-#  :git => 'https://github.com/example42/puppet-perl'
-#mod'example42/php',
-#  :git => 'https://github.com/example42/puppet-php'
-#mod'example42/postfix',
-#  :git => 'https://github.com/example42/puppet-postfix'
-#mod'example42/postgresql',
-#  :git => 'https://github.com/example42/puppet-postgresql'
-#mod'example42/profile',
-#  :git => 'https://github.com/example42/puppet-profile'
-#mod'example42/rclocal',
-#  :git => 'https://github.com/example42/puppet-rclocal'
-#mod'example42/resolver',
-#  :git => 'https://github.com/example42/puppet-resolver'
-#mod'example42/sudo',
-#  :git => 'https://github.com/example42/puppet-sudo'
-#mod'example42/svn',
-#  :git => 'https://github.com/example42/puppet-svn'
-#mod'example42/sysctl',
-#  :git => 'https://github.com/example42/puppet-sysctl'
-#mod'example42/timezone',
-#  :git => 'https://github.com/example42/puppet-timezone'
-#mod'example42/tomcat',
-#  :git => 'https://github.com/example42/puppet-tomcat'
-#mod'example42/user',
-#  :git => 'https://github.com/example42/puppet-user'
-#mod'example42/yum',
-#  :git => 'https://github.com/example42/puppet-yum'
-#
+# Used by profile::vagrant
+mod 'unibet/vagrant', :latest
+

README.md

@@ -1,21 +1,12 @@
 # example42 Puppet control-repo
 
-A state of the art, feature rich, Puppet 4 control-repo where you can:
+A state of the art, feature rich, Puppet 4 control-repo. Use it to:
 
-  - Explore the layout of a modern, general purpose, control-repo 
+  - Explore the layout of a modern, opinionated, general purpose, control-repo
   - Create and use Puppet 4 optimised modules with updated design patterns
   - Test your code in the Docker and Vagrant environments
-  - Fork a clone as starting point for your Puppet infrastructure
-
-Released under the terms of Apache2 licence.
-
-Copyright example42 GmbH (and specific commits authors)
-
-Code: [https://github.com/example42/control-repo](https://github.com/example42/control-repo)
-
-Official website: [http://www.example42.com](http://www.example42.com)
-
-Official Support forum: [Google Groups](https://groups.google.com/forum/#!forum/example42-puppet-modules)
+  - Fork and use as starting point for your Puppet infrastructure
+  - Manage the deployment workflow of your Puppet code
 
 
 ## Installation
@@ -25,14 +16,28 @@ Download this repository:
     git clone https://github.com/example42/control-repo
     cd control-repo
 
-To setup or verify the presence of the needed prerequisites just execute the script:
+To install the minimum prequequisites (hiera-eyaml, deep_merge, r10k gems) and populate the external modules directory via r10k, just run: 
 
     bin/puppet_setup.sh
 
-If you have Fabric installed local you can use Fabric for initial setup (and many other operations):
+If you lso want to install the recommended (Fabric, Vagrant, Docker) prerequisites, run:
+
+    bin/setup.sh
+
+You will be asked to confirm or skip each component installation.
+
+NOTE: setup.sh currently does not fully work on Mac and Windows.
+
+For unattended setups (typically in CI pipelines) you can skip confirmation requests with:
+
+    bin/setup.sh auto
+
+
+The setup script installs Fabric, which can be used for several tasks related to control-repo workflow management. To show the available tasks:
+
+    fab -l
+
 
-    fab puppet.setup
-
 ## Documentation
 
 The control-repo is full of more or less hidden stuff, which ease a lot Puppet code development, testing and deployment.
@@ -87,16 +92,10 @@ Some modules (the ones, of generation 2.x, which use the ```params_lookup``` fun
 
 ### Control repo
 
-The modules used (and some example42 other)  in this control repo are defined in the ```Puppetfile``` and can be installed manually via git, from the Forge or via r10k.
-
-To install r10k:
+To be able to use the control-repo with Puppet some gems are needed and defined in the ```Puppetfile``` have to be deployed.
 
-    gem install r10k
+The hiera-eyaml, r10k and deep_merge gems can be installed by the setup script which also runs r10k to populate the modules directory.
 
-To install Fabric you can use the local package providers or pip (you might need to install also the ecdsa package):
-
-    pip install fabric
-    pip install ecdsa
 
 ### Vagrant
 
@@ -106,6 +105,8 @@ For a correct setup of the Vagrant environment you need some extra plugins:
     vagrant plugin install vagrant-vbguest
     vagrant plugin install vagrant-hostmanager
 
+These plugins, as Vagrant itself, can be installed by the setup script.
+
 ### Docker
 
 Docker operations via Fabric or using the command line require Docker to be locally installed.
@@ -114,6 +115,7 @@ If you use Mac or Windows you need the newer native client, things won't work wh
 
 You'll need to run ```docker login``` before trying any operation that involves pushing your images to Docker registry.
 
+Also Docker can be installed by the setup script.
 
 ## Using and understanding this control-repo
 

aws.py

@@ -10,16 +10,16 @@
 
 @task
 def setup():
-  """Install locally the aws cli environment"""
+  """[local] Install locally the aws cli environment"""
   local( main_dir + "/bin/aws_setup.sh" )
 
 @task
 def apply(role='aws',options=''):
-  """Run puppet apply locally using the specified role (default: aws)"""
+  """[local] Run puppet apply locally using the specified role (default: aws)"""
   local( "export FACTER_role=" + str(role) + " && " + main_dir + "/bin/papply_local.sh " + str(options) )
 
 @task
 def status(region=default_region):
-  """Show AWS resources on one or all regions"""
+  """[local] Show AWS resources on one or all regions"""
   local( main_dir + "/bin/aws_status.sh " + str(region) )
 

bin/aws_setup.sh

@@ -2,10 +2,10 @@
 repo_dir=$(git rev-parse --show-toplevel)
 . "${repo_dir}/bin/functions"
 
+puppet_options="--modulepath ${repo_dir}/site:${repo_dir}/modules:/etc/puppet/modules --environmentpath ${repo_dir} --hiera_config ${repo_dir}/bin/hiera-local.yaml"
+
 echo_title "Going to install awscli and dependencies"
-puppet resource package awscli provider=pip ensure=present
-puppet resource package aws-sdk-core provider=gem ensure=present
-puppet resource package retries provider=gem ensure=present
+puppet apply $puppet_options -e 'include profile::aws'
 
 echo_title "Running aws configure"
 aws configure

bin/functions

@@ -33,15 +33,13 @@ echo_subtitle () {
 }
 
 ask_interactive () {
-    if [ "$unattended" != "yes" ] ; then
         echo "${SETCOLOR_ASK}${1}${SETCOLOR_NORMAL}"
-        echo "Press 'x' to exit or just return to go on"
+        echo "Press 'x' to skip or just return to go on"
         read press
         case $press in 
-            x) exit 2 ;;
+            x) return 2 ;;
             *) return
         esac
-    fi
 }
 
 # Shows or executes a command

bin/puppet_install4.sh

@@ -1,5 +1,12 @@
 #!/usr/bin/env bash
-breed=$1
+
+if $1;
+  breed=$1
+else
+  distro = findos distro
+  version = findos version
+  breed="$distro$version"
+fi
 
 SETCOLOR_NORMAL=$(tput sgr0)
 SETCOLOR_TITLE=$(tput setaf 6)
@@ -12,6 +19,10 @@ echo_title () {
 
 echo_title "Checking Puppet installation"
 
+findos(what=distro) {
+
+}
+
 setup_amazon() {
   echo_title "Cleaning up existing ruby and puppet installations"
   yum erase -y ruby puppet

bin/puppet_setup.sh

@@ -4,12 +4,13 @@ script_dir="$(dirname $0)"
 # repo_dir=$(git rev-parse --show-toplevel)
 . "${script_dir}/functions"
 
-if [ "$1" = "unattended" ]; then
-  echo_subtitle "Unattended mode"
-else 
-  ask_interactive "We are going to install the deep_merge, hiera-eyaml and r10k gems and, via r10k, the modules listed in Puppetfile under the modules directory."
+if [ "$1" = "auto" ]; then
+  auto=true
 fi
 
+[ "$auto" = "true" ] || ask_interactive "Going to install the needed gems and, via r10k, the modules listed in Puppetfile."
+[ "$?" = 0 ] || exit 1
+
 echo_title "Installing gems"
 puppet resource package rubygems ensure=present
 which gem || echo "You need gem support! Install rubygems to continue successfully" 
@@ -37,6 +38,6 @@ echo
 progs="puppet vagrant docker fab"
 for prog in $progs; do
   echo_subtitle "Checking ${prog} availability"
-  which $prog || echo "${prog} not found. Installation is recommended."
+  which $prog || echo_failure "${prog} not found. Installation is recommended."
   echo
 done

bin/setup.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+repo_dir="$(dirname $0)/.."
+script_dir="$(dirname $0)"
+. "${script_dir}/functions"
+
+[ "$1" = "auto" ] && auto=true || auto=false
+
+puppet_options="--modulepath ${repo_dir}/site:${repo_dir}/modules:/etc/puppet/modules --environmentpath ${repo_dir} --hiera_config ${repo_dir}/bin/hiera-local.yaml"
+
+setup_puppet() {
+  echo_title "Setting up Puppet environment"
+  if [ "$auto" = "true" ]; then
+    "${script_dir}/puppet_setup.sh" auto
+  else
+    "${script_dir}/puppet_setup.sh"
+  fi
+}
+
+install_fabric() {
+  [ "$auto" = "true" ] || ask_interactive "Going to install Fabric via Puppet"
+  [ "$?" = 0 ] || return
+  echo_title "Installing Fabric"
+  echo_subtitle "Executing: puppet apply -e 'include ::profile::python::fabric'"
+  puppet apply $puppet_options -e 'include ::profile::python::fabric'
+}
+
+install_vagrant() {  
+  [ "$auto" = "true" ] || ask_interactive "Going to install Vagrant via Puppet"
+  [ "$?" = 0 ] || return
+  echo_title "Installing Vagrant"
+  echo_subtitle "Executing: puppet apply -e 'include ::profile::vagrant'"
+  puppet apply $puppet_options -e 'include ::profile::vagrant'
+  echo_subtitle "Executing: puppet apply -e 'include ::profile::vagrant::plugins'"
+  puppet apply $puppet_options -e 'include ::profile::vagrant::plugins'
+}
+
+install_docker() { 
+  [ "$auto" = "true" ] || ask_interactive "Going to install Docker via Puppet"
+  [ "$?" = 0 ] || return
+  echo_title "Installing Docker"
+  echo_subtitle "Executing: puppet apply -e 'include ::docker'"
+  puppet apply $puppet_options -e 'include ::docker'
+}
+
+setup_puppet
+install_fabric
+install_vagrant
+install_docker
+

docker.py

@@ -4,31 +4,31 @@
 
 @task
 def test_role(puppetrole='log', image='ubuntu-14.04'):
-  """Test a role on the specified OS on a Docker image"""
+  """[local] Test a role on the specified OS on a Docker image"""
   local( 'cd ' + main_dir + '; docker/test.sh ' + str(puppetrole) + ' ' + str(image) )
 
 @task
 def build_role(puppetrole='docker_test', image='ubuntu1404'):
-  """WIP Dockerize a role on the specified image OS (data in hieradata/role/$puppetrole.yaml)"""
+  """[local] WIP Dockerize a role on the specified image OS (data in hieradata/role/$puppetrole.yaml)"""
   local( 'cd ' + main_dir + '/docker ; ./dockerize_role.sh ' + str(puppetrole) + ' ' + str(image) )
 
 @task
 def build_role_multios(puppetrole):
-  """WIP Dockerize a role on multiple OS images (data in hieradata/role/$puppetrole.yaml)"""
+  """[local] WIP Dockerize a role on multiple OS images (data in hieradata/role/$puppetrole.yaml)"""
   local( 'cd ' + main_dir + '/docker ; ./dockerize_role_all_os.sh ' + str(puppetrole) )
 
 @task
 def build_multios():
-  """Build multiple OS Docker images with TP (data in hieradata/role/docker_multios_build.yaml)"""
+  """[local] Build multiple OS Docker images with TP (data in hieradata/role/docker_multios_build.yaml)"""
   local( 'cd ' + main_dir + '/docker ; ./generate_all.sh' )
 
 @task
 def purge_images():
-  """Remove ALL local docker images (CAUTION)"""
+  """[local] Remove ALL local docker images (CAUTION)"""
   local( 'docker rmi $(docker images -q)' )
 
 @task
 def purge_containers():
-  """Remove ALL local docker containers (CAUTION)"""
+  """[local] Remove ALL local docker containers (CAUTION)"""
   local( 'docker rm $(docker ps -a -q)' )