Puppet Labs Spec Helper
The Short Version
This repository is meant to provide a single source of truth for how to initialize different Puppet versions for spec testing.
The common use case is a module such as stdlib that works with many versions of Puppet. The stdlib module should require the spec helper in this repository, which will in turn automatically figure out the version of Puppet being tested against and perform version specific initialization.
Other "customers" that should use this module are:
When developing or testing modules, simply clone this repository and install the gem it contains. The recommended way to do this is using bundler.
Add this to your project's spec_helper.rb:
Add this to your project's Rakefile:
And run the spec tests:
$ cd $modulename
$ rake spec
Parallel Fixture Downloads
Fixture downloads will now execute in parallel to speed up the testing process. Which can represent >= 600% speed increase (depending on number of threads). You can control the amount of threads by setting the
MAX_FIXTURE_THREAD_COUNT environment variable
to a positive integer, the default is currently 10. We don't suggest going higher than 25 as the gains are marginal due to some repos taking a long time to download. Please be aware that your internal VCS system may not be able to handle a high load in which case the server would fail to clone the repository. Because of this issue, this setting is tunable via
Additionally, you can also speed up cloning when using the ssh protocol by multiplexing ssh sessions. Add something similar to your ssh config. Note: you may need to change the host if you're using an internal git server.
Note: parallel downloads are available for repositories and forge modules.
It is also possible to use the
parallel_tests Gem via the
:parallel_spec Rake task to run rspec commands in parallel on groups of spec files.
Use of parallelization at this level can result in large performance benefits when the Rspec examples tend to cause a number of large, CPU-intensive catalog compilations to occur. An example of where this might be the case is in a complex module with a lot of tests or a control repo with many hosts.
Be aware however that in other circumstances this parallelization can result in the tests actually taking longer to run. The best thing to do is time
rspec spec and
rspec parallel_spec and use the parallelization only when there is a clear benefit.
To enable this feature, add the
parallel_tests Gem to your project's Gemfile:
And then to run spec tests in parallel:
$ rake parallel_spec
Please file issues against this project at the Puppet Labs Issue Tracker
The Long Version
Purpose of this Project
This project is intended to serve two purposes:
- To serve as a bridge between external projects and multiple versions of puppet; in other words, if your project has a dependency on puppet, you shouldn't need to need to worry about the details of how to initialize puppet's state for testing, no matter what version of puppet you are testing against.
- To provide some convenience classes / methods for doing things like creating tempfiles, common rspec matchers, etc. These classes are in the puppetlabs_spec directory.
- To provide a common set of Rake tasks so that the procedure for testing modules is unified.
To Use this Project
The most common usage scenario is that you will check out the 'main' branch of this project from github, and install it as a rubygem. There should be few or no cases where you would want to have any other branch of this project besides main/HEAD.
Running on non-current ruby versions
Since gem and bundler, ruby's package management tools, do not take the target ruby version into account when downloading packages, the puppetlabs_spec_helper gem can only depend on gems that are available for all supported ruby versions. If you can/want to use features from other packages, install those additional packages manually, or have a look at the Gemfile, which provides code to specify those dependencies in a more "friendly" way. This currently affects the following gems:
Initializing Puppet for Testing
In most cases, your project should be able to define a spec_helper.rb that includes just this one simple line:
Then, as long as the gem is installed, you should be all set.
If you are using rspec-puppet for module testing, you will want to include a different library:
NOTE that this is specifically for initializing Puppet's core. If your project does not have any dependencies on puppet and you just want to use the utility classes, see the next section.
A number of the Puppet parser features, controlled via configuration during a normal puppet run, can be controlled by exporting specific environment variables for the spec run. These are:
STRICT_VARIABLES- Set to "no" to disable strict variable checking. See strict_variables in puppet.conf for details.
ORDERING- set to the desired ordering method ("title-hash", "manifest", or "random") to set the order of unrelated resources when applying a catalog. Leave unset for the default behavior, currently "random". This is equivalent to setting ordering in puppet.conf.
As an example, to run spec tests with the future parser, strict variable checking, and manifest ordering, you would:
ORDERING=manifest rake spec
Using Utility Classes
If you'd like to use the Utility classes (PuppetlabsSpec::Files, PuppetlabsSpec::Fixtures), you just need to add this to your project's spec_helper.rb:
NOTE that the above line happens automatically if you've required 'puppetlabs_spec_helper/puppet_spec_helper', so you don't need to do both.
In either case, you'll have all of the functionality of Puppetlabs::Files, Puppetlabs::Fixtures, etc., mixed-in to your rspec context.
puppetlabs_spec_helper has the ability to populate the
spec/fixtures/modules directory with dependent modules when
rake spec or
rake spec_prep is run. To do so, all required modules should be listed in a
.fixtures.yml in the root of the project. You can specify a alternate location for that file in the
FIXTURES_YML environment variable.
You can use the
MODULE_WORKING_DIR environment variable to specify a diffent location when installing module fixtures via the forge. By default the
working directory is
When specifying the repo source of the fixture you have a few options as to which revision of the codebase you wish to use, and optionally, the puppet versions where the fixture is needed.
repo- the url to the repo
scm- options include git or hg. This is an optional step as the helper code will figure out which scm is used.
scm: git scm: hg
target- the directory name to clone the repo into ie.
target: mymoduledefaults to the repo name (Optional)
subdir- directory to be removed from the cloned repo. Its contents will be moved to the root directory (Optional)
ref- used to specify the tag name (like version) or commit hash to be checked out (Optional). Branch names should use the
ref: 1.0.0 ref: 880fca52c
branch- used to specify the branch name you want to use ie.
flags- additional flags passed to the module installer (both puppet and scm)
puppet_version- versions of puppet for which the fixture should be installed. Ruby version constraints are supported. Only works when the
semantic_puppetgem is available (shipped with puppet 4.0 and up, by default).
puppet_version: '>= 6.0.0'
branchcan be used together to get a specific revision on a specific branch
- Top level
defaultsoption could be used to set global options
Basic fixtures that will symlink
spec/fixtures/modules/my_modules to the
This is the same as specifying no symlinks fixtures at all.
stdlib as required module fixtures:
Put a supplementary repository at a different location
Specify that the git tag
stdlib should be checked out:
Only install the
yumrepo_core module when testing against Puppet 7 or greater:
puppet_version: ">= 6.0.0"
Move manifests and siblings to root directory when they are inside a
Install modules from Puppet Forge:
Pass additional flags to module installation:
flags: "--module_repository https://my_repo.com"
defaults to define global parameters:
flags: "--module_repository https://my_repo.com"
Any module that has a
spec/lib directory will be available on the ruby
LOAD_PATH for tests to consume. This allows modules to provide additional helper code to be supplied. The augeasprovider_core module has some examples.
Testing Parser Functions
This whole section is superseded by improved support of accessing the scope in rspec-puppet.
Modify rspec behavior
Running Rspec with additional settings
You can add command line options to rspec using the
CI_SPEC_OPTIONS environment variable. Any text in the
CI_SPEC_OPTIONS environment variable is added as an rspec option. For example:
If you wanted to output JUnit test reports for a Jenkins CI server you could use;
export CI_SPEC_OPTIONS = "-r yarjuf -f JUnit -o result.xml"
$ENV:CI_SPEC_OPTIONS = '-r yarjuf -f JUnit -o result.xml'
And then run
bundle exec rake spec
This would cause rspec to load the
yarjuf gem and output the results in JUnit format to the file
Running specs in parallel
When executing tests in a matrix CI environment, tests can be split up to run
a share of specs per CI node in parallel. Set the
variable to the total number of nodes, and the
CI_NODE_INDEX to a number
between 1 and the
If using Travis CI, add new lines to the "env" section of .travis.yml per node, remembering to duplicate any existing environment variables:
- FUTURE_PARSER=yes CI_NODE_TOTAL=2 CI_NODE_INDEX=1
- FUTURE_PARSER=yes CI_NODE_TOTAL=2 CI_NODE_INDEX=2
Running tests tagged with test tiers
To run tests tagged with risk levels set the
TEST_TIERS environment variable to a comma-separated list of the appropriate tiers.
For example: to run tests marked
tier_high => true and
tier_medium => true in the same test run set the
TEST_TIERS only accepts low, medium and high as valid tiers. If you would like to use your own keywords to set the environment variable
For example: to use the keywords dev, rnd, staging and production you can set
TEST_TIERS_ALLOWED=dev,rnd,staging,production. Then you would be able to run tests marked
tier_dev => true,
tier_production => true with
Note, if the
TEST_TIERS environment variable is set to empty string or nil, all tiers will be executed.
Generating code coverage reports
This section describes how to add code coverage reports for Ruby files (types, providers, ...). See the documentation of RSpec-Puppet for Puppet manifest coverage reports.
Starting with Ruby 1.9, the de facto standard for Ruby code coverage is SimpleCov. It is implemented as a Rake task in this gem.
To run code coverage:
$ rake spec:simplecov
Reports are written to
/coverage/, which you should add to
The reports can be generated every time you invoke RSpec, e.g. via
You can enable it, set the following environment variable:s
Remember to add the simplecov-console gem to your
Gemfile. If you run
spec:simplecov on Travis-CI or any of the other supported CI services, reports get generated which can then be uploaded to codecov.io with the recommended codecov uploader.
Some Notes for Windows Users
A windows users may need to do one of two things to execute
Although things may appear to work, the init.pp may not transfer to the fixtures folder as needed or may transfer as an empty file.
This is related to a registry security setting requiring elevated privileges to create symbolic links.
Currently, there are two known approaches to get around this problem.
- run your windows shell (cmd) as an Administrator or
- modify the registry entry settings to allow symbolic links to be created.
The following links may give you some insight into why...
There are two major mocking frameworks in modules test suites today: mocha and rspec-mocks. We recommend that you choose rspec-mocks explicitly by specifying
mock_with, either in your
spec_helper.rb like so:
RSpec.configure do |c|
or by using Puppet Development Kit's
mock_with option in
You can also continue to use mocha by explicitly specifying
:mocha, following the mocha documentation.
To migrate from mocha to rspec-mocks, in many simple cases the following two kinds of changes are all you need:
Translate all stubs:
Translate all expectations:
- As a part of the RSpec project, rspec-mocks integration is better.
- mocha is extending base ruby objects with its
expectsmethods, polluting the global namespace, with a potential for subtle and gnarly errors.
- Currently both rspec-mocks and mocha get loaded, leading to test suites that use both.
- puppetlabs_spec_helper loads and configures mocha unconditionally, causing friction when a different mocking framework is wanted.
- mocha is an additional dependency to carry.
This codebase is licensed under Apache 2.0. However, the open source dependencies included in this codebase might be subject to other software licenses such as AGPL, GPL2.0, and MIT.