Restructured Integration Testing Suite using InSpec Profile #78

americanhanko · 2018-03-24T21:41:45Z

This PR

Converts the chef generate cookbook smoke test layout into an InSpec profile layout, created by inspec init profile.
Breaks up each of the InSpec tests into control groups that make a little more sense (e.g. admin-user vs. standard-user controls)

Why

I feel this is a better design for a testing suite that allows the easy re-use of InSpec controls.
With an InSpec profile, there is potential for reusability for other macOS admins/ops personnel.

Takeaways

The controls are named and organized according to System Preferences.app. Although this structure feels purposeful and familiar, it may need to be revisited if it is too consumer forward rather than administrator forward.
Significantly more test coverage is required for things like preferences and each of their domain scopes. To combat this, I hope to add some high-level functionality via the use of attributes or recipes that will make the resources of the cookbook more useful for less technical users

- A few minor changes to test recipes - Addition of full integration test suite using InSpec profile structure

jazaval · 2018-03-24T22:26:31Z

.kitchen.yml

@@ -4,79 +4,72 @@ driver:
  provider: parallels

 provisioner:
-  product_name: chef
-  product_version: 13.8
+  name: chef_zero


I think we should still be forcing the latest chef-client here, instead of what might be in a testing box.

provisioner: product_name: chef product_version: latest install_strategy: always

jazaval · 2018-03-24T22:27:43Z

.kitchen.yml

+  - recipe[macos_test::spotlight]
+  verifier:
+    controls:
+    - indexing/searching


These slashes make me thing there's some sort of namespacing happening at first glance - may just indexing_and_seaching?

jazaval · 2018-03-24T22:27:58Z

.kitchen.yml

+  - recipe[macos_test::xcode]
+  verifier:
+    controls:
+    - xcode/simulators


These slashes make me thing there's some sort of namespacing happening at first glance - may just xcode_and_simulators?

jazaval · 2018-03-24T22:28:52Z

test/integration/default/README.md

+# macOS Cookbook InSpec Profile
+
+The macOS Cookbook InSpec profile tests the custom resources and the recipes
+that utilize them to assertain their full functionality and dependability. We


jazaval · 2018-03-24T22:30:02Z

test/integration/default/controls/desktop_and_screensaver.rb

@@ -0,0 +1,35 @@
+title 'desktop & screen saver'


I would prefer and over &.

Why I wrote it with the ampersand:

jazaval · 2018-03-24T22:34:01Z

test/integration/default/controls/energy_saver.rb

+
+  describe command('systemsetup -getremoteappleevents') do
+    apple_events_pattern = Regexp.union('Remote Apple Events: On')
+    its('stdout') { should match apple_events_pattern }


You can use the include matcher to write this more simply.

its('stdout') { should include 'Remote Apple Events: On' }

jazaval · 2018-03-24T22:36:22Z

test/integration/default/controls/xcode.rb

+    end
+
+    describe command('/opt/chef/embedded/bin/xcversion simulators') do
+      sim_pattern = Regexp.union 'iOS 10.3.1 Simulator (installed)'


Consider using the include matcher here as well.

jazaval · 2018-03-24T22:36:33Z

test/integration/default/controls/xcode.rb

+    end
+
+    describe command('/opt/chef/embedded/bin/xcversion simulators') do
+      sim_pattern = Regexp.union 'iOS 9.3 Simulator (installed)'


Eric Hanko added 6 commits March 22, 2018 13:45

Adds cherry-picked changes from a local commit

595a892

- A few minor changes to test recipes - Addition of full integration test suite using InSpec profile structure

general refinement and polishing; delete old smokes

05b93ed

Fix up some integration test bugs with hostname

d26d841

add insurance test for software updates

384243d

fix integration test

2e26751

Merge branch 'release/1.10' into feature/inspec-profile

e83ef62

jazaval approved these changes Mar 24, 2018

View reviewed changes

Eric Hanko added 3 commits March 25, 2018 18:35

address comments in PR

d20e374

suite names get converted to hyphens anyway

7070a33

revert incorrect simulator tests

a6c93e9

americanhanko merged commit 984719a into release/1.10 Mar 26, 2018

americanhanko deleted the feature/inspec-profile branch March 26, 2018 16:36

americanhanko mentioned this pull request Mar 26, 2018

Release/1.10 #82

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restructured Integration Testing Suite using InSpec Profile #78

Restructured Integration Testing Suite using InSpec Profile #78

americanhanko commented Mar 24, 2018

jazaval Mar 24, 2018

jazaval Mar 25, 2018

jazaval Mar 24, 2018

jazaval Mar 24, 2018

jazaval Mar 24, 2018

jazaval Mar 24, 2018

americanhanko Mar 24, 2018

jazaval Mar 24, 2018

jazaval Mar 24, 2018

jazaval Mar 24, 2018

Restructured Integration Testing Suite using InSpec Profile #78

Restructured Integration Testing Suite using InSpec Profile #78

Conversation

americanhanko commented Mar 24, 2018

This PR

Why

Takeaways

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment