Modularize puppet logic into classes

[jeff1evesque]

We need to modularize all of our puppet manifests, into classes, and possibly modules.

Note: this should be similar to several manifests found within drupal-demonstration.

[jeff1evesque]

The following https://bpaste.net/show/45016ab334a7 link may not exist, so we paste it below:

• init.pp
  • /medium
    • script1.pp
    • script2.pp
  • /high
    -script3.pp
    -script4.pp

## script1.pp
class sc1 {
  ...
}

IRC #puppet (02/24/16 ~ 11:15pm EST):

jeffreylevesque: can init.pp do include sc1 - https://bpaste.net/show/45016ab334a7?

ChrJantz: autoloader won't find this file jeffreylevesque

Volcane: jeffreylevesque: its easy class x::y goes in x/manifests/y.pp class x goes in x/manifests/init.pp. that is, then it just works
Volcane: jeffreylevesque: if you do something else, it wont work, stop, follow the rules and it just works again
Volcane: jeffreylevesque: there's no magic to it

ChrJantz: Volcane +1 well said

Volcane: in < 4 there are hacks you can make to convince it work in ways it was not designed
Volcane: these will not work in 4 anymore, so just use it as designed.

ChrJantz: just simple is great for getting used to the syntax
ChrJantz: once you know how to script manifests stop f****** around and follow the autoloading principles. as simple s that.
=-= Mode #puppet +v nigelk by ChanServ

jeffreylevesque: sorry, a little confused on class x::y versus class x
jeffreylevesque: class y inside class x

Volcane: class x { notify{"hello from X!": } that goes in module "x" in the file x/manifests/init.pp

ChrJantz: include x means include init.pp from module x

WhatsARanjit: jeffreylevesque: Think of it like website.com and website.com/hello.html

Volcane: class x::y { notify{"hello from X::Y!": } that goes in module "x" in the file x/manifests/y.pp
=-= Mode #puppet +v nigelk by ChanServ

ChrJantz: include x::hello means include hello.pp from module x

jeffreylevesque: thank you

Volcane: the same convention is true for defines

binford2k: https://docs.puppetlabs.com/puppet/latest/reference/modules_fundamentals.html#manifests
binford2k: https://docs.puppetlabs.com/puppet/latest/reference/lang_namespaces.html#autoloader-behavior

[jeff1evesque]

The following general solution will work:

• Vagrantfile
• puppet/
  • environment/
  • global/
    • manifests/
      • default.pp
    • medium/
      • manifests/
        • script1.pp
        • script2.pp
    • high/
      • manifests/
        • script3.pp
        • script4.pp

Note: typically, manifest(s), and module(s) should go in the environment/ directory.

---

## script1.pp
class medium::script1 {
  ...
}

---

## default.pp
include medium::script1

---

## Vagrantfile
...
  ## Custom Manifest:
  config.vm.provision 'puppet' do |puppet|
    puppet.environment_path = 'puppet'
    puppet.environment      = 'stig'
    puppet.manifests_path   = 'puppet/global/manifests'
    puppet.module_path      = ['puppet/environment/development/modules', 'puppet/global/modules']
    puppet.manifest_file    = 'default.pp'
  end
...

[jeff1evesque]

If we implement the following Vagrantfile:

...
  ## Custom Manifest:
  config.vm.provision 'puppet' do |puppet|
    puppet.environment_path = 'puppet'
    puppet.environment      = 'global'
    puppet.manifests_path   = 'puppet/global/manifests'
    puppet.module_path      = ['puppet/global/modules']
    puppet.manifest_file    = 'default.pp'
  end
...

Then, we get modules defined from our Puppetfile, downloaded into:

$ git status
On branch feature-xxxx
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        puppet/stig/modules/some_module/
        puppet/stig/modules/another_module/
        puppet/stig/modules/third_module/
        puppet/stig/modules/etc_module/

no changes added to commit (use "git add" and/or "git commit -a")

Along with the following vagrant up error traceback:

...
==> default: Running provisioner: puppet...
==> default: Running Puppet with environment global...
==> default: Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::medium::script1 for localhost.localdomain at /tmp/vagrant-puppet/manifests-a53f720de1108e13b0061de0d965bfa5/default.pp:xx:xx on node localhost.localdomain
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

[jeff1evesque]

Similarly, if we implement the following Vagrantfile:

...
  ## Custom Manifest:
  config.vm.provision 'puppet' do |puppet|
    puppet.environment_path = 'puppet'
    puppet.environment      = 'global'
    puppet.manifests_path   = 'puppet/global/manifests'
    puppet.module_path      = ['puppet/global/modules']
    puppet.manifest_file    = 'default.pp'
  end
...

Then, we get modules defined from our Puppetfile, replacing our current module manifests:

$ git status
On branch feature-xxx
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        deleted:    puppet/global/modules/high/manifests/script3.pp
        deleted:    puppet/global/modules/high/manifests/script4.pp
        deleted:    puppet/global/modules/medium/manifests/script1.pp
        deleted:    puppet/global/modules/medium/manifests/script2.pp

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        puppet/stig/modules/some_module/
        puppet/stig/modules/another_module/
        puppet/stig/modules/third_module/
        puppet/stig/modules/etc_module/

no changes added to commit (use "git add" and/or "git commit -a")

Along with the same vagrant up error traceback:

...
==> default: Running provisioner: puppet...
==> default: Running Puppet with environment global...
==> default: Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::medium::script1 for localhost.localdomain at /tmp/vagrant-puppet/manifests-a53f720de1108e13b0061de0d965bfa5/default.pp:xx:xx on node localhost.localdomain
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.
...

[jeff1evesque]

Finally, if we implement the following Vagrantfile:

...
  ## Custom Manifest:
  config.vm.provision 'puppet' do |puppet|
    puppet.environment_path = 'puppet'
    puppet.environment      = 'global'
    puppet.manifests_path   = 'puppet/global/manifests'
    puppet.module_path      = ['puppet/global/modules/contrib', 'puppet/environment/development/modules']
    puppet.manifest_file    = 'default.pp'
  end
...

Then, no error traceback exists following vagrant up, as well as no unnecessary modules being downloaded into incorrect directories. Instead, puppet modules are downloaded into:

$ cd puppet/global/modules
$ ls
contrib  high  medium

[jeff1evesque]

Later, if we add additional manifest references within the Vagrantfile, with the same directory structure proposed above:

...
  ## Custom Manifest: add sass, uglifyjs, imagemin compilers
  config.vm.provision 'puppet' do |puppet|
    puppet.environment_path = 'puppet/environment'
    puppet.environment      = 'development'
    puppet.manifests_path   = 'puppet/environment/development/manifests'
    puppet.module_path      = 'puppet/environment/development/modules'
    puppet.manifest_file    = 'configure_compilers.pp'
  end

  ## Custom Manifest:
  config.vm.provision 'puppet' do |puppet|
    puppet.environment_path = 'puppet'
    puppet.environment      = 'global'
    puppet.manifests_path   = 'puppet/global/manifests'
    puppet.module_path      = ['puppet/global/modules/contrib', 'puppet/environment/development/modules']
    puppet.manifest_file    = 'default.pp'
  end
...

We will get the following error:

...
==> default: Running provisioner: puppet...
==> default: Running Puppet with environment development...
==> default: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/environments.rb:38:in `get!': Could not find a directory environment named 'development' anywhere in the path: /tmp/vagrant-puppet/environments. Does the directory exist? (Puppet::Environments::EnvironmentNotFound)
==> default:    from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application_support.rb:29:in push_application_context'
==> default:    from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:337:in `run'
==> default:    from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:128:in `run'
==> default:    from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
==> default:    from /opt/puppetlabs/bin/puppet:5:in `<main>'
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

[jeff1evesque]

To eliminate the latest errors, we need to restructure our puppet/ directory as follows:

• Vagrantfile
• puppet/environment/development
  • manifests/
    • default.pp
    • configure_compilers.pp
  • modules/
    • medium/
      • manifests/
        • script1.pp
        • script2.pp
    • high/
      • manifests/
        • script3.pp
        • script4.pp
  • modules_contrib/
    • this is where puppet's r10k downloads contributed modules
  • scripts/
  • templates/
  • Puppetfile

---

Then, we can keep our puppet manifests the same:

## script1.pp
class medium::script1 {
  ...
}

## default.pp
include medium::script1

---

However, the above directory structure, and puppet manifests, would require the following Vagrantfile:

...
  ## ensure puppet/modules directory on the host before 'vagrant up'
  config.trigger.before :up do
    run 'mkdir -p puppet/environment/development/modules'
    run 'mkdir -p puppet/environment/development/modules_contrib'
  end
...
  ## Custom Manifest: add sass, uglifyjs, imagemin compilers
  config.vm.provision 'puppet' do |puppet|
    puppet.environment_path = 'puppet/environment'
    puppet.environment      = 'development'
    puppet.manifests_path   = 'puppet/environment/development/manifests'
    puppet.module_path      = 'puppet/environment/development/modules_contrib'
    puppet.manifest_file    = 'configure_compilers.pp'
  end

  ## Custom Manifest:
  config.vm.provision 'puppet' do |puppet|
    puppet.environment_path = 'puppet'
    puppet.environment      = 'global'
    puppet.manifests_path   = 'puppet/global/manifests'
    puppet.module_path      = ['puppet/environment/development/modules_contrib', 'puppet/environment/development/modules']
    puppet.manifest_file    = 'default.pp'
  end
...
  # clean up files on the host after the guest is destroyed
  config.trigger.after :destroy do
    run 'rm -Rf log'
    run 'rm -Rf puppet/environment/development/modules_contrib'
  end
...

[jeff1evesque]

Commenting out corresponding manifests from the Vagrantfile, associated with the database, redis, sklearn, then, adding the following within package/manifests/webcompilers.pp, will ensure the webcompilers are properly installed, and enabled:

...
class package::webcompilers {
    ## variables
    $webcompilers = [
        'uglify-js',
        'imagemin',
        'node-sass',
        'babel-core',
        'browserify',
        'babelify'
    ]

    exec { 'echo-versions':
        command => 'npm -v && node -v',
        path    => '/usr/bin',
        before  => Package[$webcompilers],
        logoutput => true,
    }

    ## packages: install general packages (npm)
    package { $webcompilers:
        ensure   => 'present',
        provider => 'npm',
    }
}

Otherwise, we get the following error traceback:

...
==> default: Running provisioner: puppet...
==> default: Running Puppet with environment development...
==> default: stdin: is not a tty
==> default: Notice: Compiled catalog for vagrant-ubuntu-trusty-64.home in envir
onment development in 1.84 seconds
==> default: Notice: /Stage[main]/Package::Webcompilers/Exec[echo-general-notice
]/returns: installing npm packages with:
==> default: Notice: /Stage[main]/Package::Webcompilers/Exec[echo-general-notice
]/returns: executed successfully
==> default: Notice: /Stage[main]/Package::Webcompilers/Exec[echo-versions]/retu
rns: 3.7.3
==> default: Notice: /Stage[main]/Package::Webcompilers/Exec[echo-versions]/retu
rns: v5.8.0
==> default: Notice: /Stage[main]/Package::Webcompilers/Exec[echo-versions]: Tri
ggered 'refresh' from 1 events
==> default: Notice: /Stage[main]/Package::Webcompilers/Package[uglify-js]/ensur
e: created
==> default: Error: Execution of '/usr/bin/npm install --global imagemin' return
ed 1: > optipng-bin@3.1.0 postinstall /usr/lib/node_modules/imagemin/node_module
s/optipng-bin
==> default: > node lib/install.js
==> default: Error: /Stage[main]/Package::Webcompilers/Package[imagemin]/ensure:
 change from absent to present failed: Execution of '/usr/bin/npm install --glob
al imagemin' returned 1: > optipng-bin@3.1.0 postinstall /usr/lib/node_modules/i
magemin/node_modules/optipng-bin
==> default: > node lib/install.js
==> default: Error: Execution of '/usr/bin/npm install --global node-sass' retur
ned 1: npm WARN deprecated npmconf@2.1.2: this package has been reintegrated int
o npm and is now out of date with respect to npm
==> default: npm WARN deprecated lodash@1.0.2: lodash@<3.0.0 is no longer mainta
ined. Upgrade to lodash@^4.0.0.
==> default: npm WARN deprecated graceful-fs@1.2.3: graceful-fs version 3 and be
fore will fail on newer node releases. Please update to graceful-fs@^4.0.0 as so
on as possible.
==> default: /usr/bin/node-sass -> /usr/lib/node_modules/node-sass/bin/node-sass

==> default:
==> default: > node-sass@3.4.2 install /usr/lib/node_modules/node-sass
==> default: > node scripts/install.js
==> default: Error: /Stage[main]/Package::Webcompilers/Package[node-sass]/ensure
: change from absent to present failed: Execution of '/usr/bin/npm install --glo
bal node-sass' returned 1: npm WARN deprecated npmconf@2.1.2: this package has b
een reintegrated into npm and is now out of date with respect to npm
==> default: npm WARN deprecated lodash@1.0.2: lodash@<3.0.0 is no longer mainta
ined. Upgrade to lodash@^4.0.0.
==> default: npm WARN deprecated graceful-fs@1.2.3: graceful-fs version 3 and be
fore will fail on newer node releases. Please update to graceful-fs@^4.0.0 as so
on as possible.
==> default: /usr/bin/node-sass -> /usr/lib/node_modules/node-sass/bin/node-sass
...

[jeff1evesque]

When commenting out the following portions of the Vagrantfile:

...
  ## Custom Manifest: build scikit-learn
  config.vm.provision "puppet" do |puppet|
    puppet.environment_path = 'puppet/environment'
    puppet.environment      = 'development'
    puppet.manifests_path   = 'puppet/environment/development/manifests'
    puppet.module_path      = ['puppet/environment/development/modules_contrib', 'puppet/environment/development/modules']
    puppet.manifest_file    = "install_sklearn.pp"
  end
...
  ## Custom Manifest: install, and configure SQL database
  config.vm.provision "puppet" do |puppet|
    puppet.environment_path = 'puppet/environment'
    puppet.environment      = 'development'
    puppet.manifests_path   = 'puppet/environment/development/manifests'
    puppet.module_path      = ['puppet/environment/development/modules_contrib', 'puppet/environment/development/modules']
    puppet.manifest_file  = "setup_database.pp"
  end
...

The associated webcompiler logic runs, without producing any error traceback:

...
  ## Custom Manifest: define webcompilers
  #
  #  Note: future parser allow heredoc sytnax (since puppet 3.5), and allows array
  #        iteration in the puppet manifest.
  config.vm.provision "puppet" do |puppet|
    puppet.environment_path = 'puppet/environment'
    puppet.environment      = 'development'
    puppet.manifests_path   = 'puppet/environment/development/manifests'
    puppet.module_path      = ['puppet/environment/development/modules_contrib', 'puppet/environment/development/modules']
    puppet.manifest_file  = "compile_asset.pp"
  end
...

Therefore, we need to determine if we can narrow the error to a single manifest:

• install_sklearn.pp
• setup_database.pp

Then, we can attempt a build, by placing the latter two manifest (or the single manifest culprit), after all other manifests declarations, within the Vagrantfile.

[jeff1evesque]

We ran a vagrant up build, with just the database manifest commented out from the Vagrantfile, and no errors were present within the build trace:

...
  ## Custom Manifest: install, and configure SQL database
  config.vm.provision "puppet" do |puppet|
    puppet.environment_path = 'puppet/environment'
    puppet.environment      = 'development'
    puppet.manifests_path   = 'puppet/environment/development/manifests'
    puppet.module_path      = ['puppet/environment/development/modules_contrib', 'puppet/environment/development/modules']
    puppet.manifest_file  = "setup_database.pp"
  end
...

So, we will place the database manifest, after all other manifest references, within the Vagrantfile.

Note: we should create a separate issue to further debug the above condition.

[jeff1evesque]

We've placed the setup_database.pp manifest, after all other manifests references within our Vagrantfile. Then, a corresponding vagrant up, results in none of the above traceback errors. Therefore, the above suggestion, to incorporate a logoutput for the node, and npm version, within package/manifest/webcompilers.pp, is no longer required, and will be removed:

...
    exec { 'echo-versions':
        command => 'npm -v && node -v',
        path    => '/usr/bin',
        before  => Package[$webcompilers],
        logoutput => true,
    }
...