PDK 1.2.1 test unit fails for unsupported OSes

[bittner]

I'm testing roles in our control repository, and I've notices that while pdk unit test passed successfully last week (with PDK 1.2.0) it fails since the release of PDK 1.2.1, e.g.

$ pdk test unit 2> /dev/null
failed: rspec: ./spec/classes/role_server_customer_standalone_spec.rb:10: error during compilation: Evaluation Error: Unknown variable: '::lsbdistcodenameid'. at /builds/comsolit/puppet-control/spec/fixtures/modules/php/manifests/params.pp:38:27 on node runner-4c61c68a-project-186-concurrent-0.comsolit.local
  role::server::customer_standalone on debian-9-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

failed: rspec: ./spec/classes/role_workstation_backoffice_spec.rb:14: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 9.1. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-4c61c68a-project-186-concurrent-0.comsolit.local
  role::workstation::backoffice on debian-9-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

failed: rspec: ./spec/classes/role_workstation_developer_spec.rb:14: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 9.1. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-4c61c68a-project-186-concurrent-0.comsolit.local
  role::workstation::developer on debian-9-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

ERROR: Job failed: exit code 1

Our metadata.json specifies Debian 9 as our target OS version, that is why compiling mayflower-php (or puppet-php) seems to fail:

  "operatingsystem_support": [
    {
      "operatingsystem": "Debian",
      "operatingsystemrelease": [
        "9"
      ]
    },

It is probably intended to more strongly enforce the manifest now, correct?

[DavidS]

The facts for Debian9 in facterdb had an issue where some of them were from a sid system. This was fixed in voxpupuli/facterdb@5d61581 , which was shipped with 1.2.1.

The error message itself looks like its coming from your module balking on operatingsystemrelease being 9.1, which needs to be fixed in your module.

[bittner]

I'm getting the same error for Debian 8.0, so is this really an issue of "my module"? (Note: I'm testing roles of a control repository.)

$ pdk test unit 2> /dev/null
failed: rspec: ./spec/classes/role_server_customer_standalone_spec.rb:10: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 8.0. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-fa1040f1-project-186-concurrent-0.comsolit.local
  role::server::customer_standalone on debian-8-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

failed: rspec: ./spec/classes/role_server_customer_standalone_spec.rb:10: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 9.1. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-fa1040f1-project-186-concurrent-0.comsolit.local
  role::server::customer_standalone on debian-9-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

failed: rspec: ./spec/classes/role_workstation_backoffice_spec.rb:14: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 8.0. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-fa1040f1-project-186-concurrent-0.comsolit.local
  role::workstation::backoffice on debian-8-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

failed: rspec: ./spec/classes/role_workstation_backoffice_spec.rb:14: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 9.1. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-fa1040f1-project-186-concurrent-0.comsolit.local
  role::workstation::backoffice on debian-9-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

failed: rspec: ./spec/classes/role_workstation_developer_spec.rb:14: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 8.0. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-fa1040f1-project-186-concurrent-0.comsolit.local
  role::workstation::developer on debian-8-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

failed: rspec: ./spec/classes/role_workstation_developer_spec.rb:14: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 9.1. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-fa1040f1-project-186-concurrent-0.comsolit.local
  role::workstation::developer on debian-9-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

This build result is from an adapted metadata.json with both Debian 8 and 9:

  "operatingsystem_support": [
    {
      "operatingsystem": "Debian",
      "operatingsystemrelease": ["8", "9"]
    },

[bittner]

It looks like PDK only reports one dependency at a time. So, having around 40 modules of dependencies in our control repository, it will take me potentially 40 test runs and 40 Git repository forks and adaptions of metadata.json to get the build passing.

Also note that each test run over our control repo takes around 3 minutes, because PDK installs all forge_modules afresh from the .fixtures.yml, and deletes the downloaded modules right after the test run.

Not very encouraging prospects for us getting the build pass again.

[DavidS]

david@davids:~/git/puppet-php$ sudo dpkg -i /home/david/Downloads/pdk_1.2.1.0-1xenial_amd64.deb 
[sudo] password for david: 
(Reading database ... 263333 files and directories currently installed.)
Preparing to unpack .../pdk_1.2.1.0-1xenial_amd64.deb ...
Unpacking pdk (1.2.1.0-1xenial) over (1.2.0.0-1xenial) ...
Setting up pdk (1.2.1.0-1xenial) ...
Processing triggers for libc-bin (2.24-14) ...
david@davids:~/git/puppet-php$ pdk --version
1.2.1
david@davids:~/git/puppet-php$ rm -rf .bundle Gemfile.lock 
david@davids:~/git/puppet-php$ pdk test unit
[✔] Installing missing Gemfile dependencies.
[✔] Preparing to run the unit tests.
[✔] Running unit tests.
  Evaluated 325 tests in 29.423709008 seconds: 0 failures, 0 pending.
[✔] Cleaning up after running unit tests.
david@davids:~/git/puppet-php$

Looking at your errors,

 failed: rspec: ./spec/classes/role_server_customer_standalone_spec.rb:10: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 8.0. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-fa1040f1-project-186-concurrent-0.comsolit.local

it fails in spec/fixtures/modules/software/manifests/params.pp line 192 every time, except for the very first instance which fails on Unknown variable: '::lsbdistcodenameid'., which might not be in the default set.

Some more general notes: The PDK in its current form is targeted at single module development. Many modules have not yet switched to using the pdk (look at the metadata.json to see if the module is created using the PDK). Running the PDK over "random" modules is fine, if you want to help improve them, but likely will not provide added value for running an infrastructure, as the cost of forking the module locally far outweighs the advantages of cleaning up the code.

[bittner]

Some more general notes: The PDK in its current form is targeted at single module development. Many modules have not yet switched to using the pdk

I know, we're all pioneers, and only a vanishing portion of the modules out there use the thing that was just born a month ago. We all know that, and it's normal.

And, maybe we at @comsolit are the only ones that are using PDK for testing control repos, despite the fact that it's not designed to do that. But: it worked, or seemed to work, just a few days ago. And now we need to get it working again. We need to get this fixed, not explained why "it's good as it is". It's broken now. And must be fixed.

So, I'll go through all the modules that PDK complains about (php and software are the ones we've seen in this thread) and try to fix things via PRs. There's probably no other choice.

Running the PDK over "random" modules is fine, if you want to help improve them

There's no "running the PDK" over anything, if I understand the inner workings right. It's rake and something, with a few configuration files, things that existed before the name "PDK" was ever coined. Hence, there should be no reason why "PDK" shouldn't work with appropriate configuration files for rake (.fixtures.yaml and the spec_helper.rb, mainly), because there is no such thing as "PDK". PDK is a new name for something that existed before the name was invented.

[bittner]

Bottom line: It should probably be possible to turn the Error into a Warning to make the test run pass for arbitrary dependencies.

[bmjen]

Hey @bittner , this is definitely an interesting scenario that we'd like to see through for you. Given what @DavidS has found in regards to the recent changes to Debian facts in facterdb, can you try pinning the facterdb gem in your Gemfiles to 0.4.0 (the previous release) and see if PDK 1.2.1 starts successfully running on your modules? I think this would help us isolate the problem and address it.