allow open source puppet to use os_patching

[nanowinner]

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 6.27.0
• Ruby: none
• Distribution: Ubuntu 20.04.4 LTS
• Module version: 0.17.0

How to reproduce (e.g Puppet code you use)

Run puppet agent -vt on a node that is running puppetlabs-patching_as_code v1.1.2 and albatrossflavour-os_patching v0.17.0 configured for biweekly updates via Hiera

What are you seeing

root@ip:/var/cache/os_patching# puppet agent -tv
Info: Using environment 'dev'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for i-0c17fbc6b2bfc0b02
Info: Applying configuration version '1657539015'
Notice: /Stage[main]/Os_patching/File[/var/cache/os_patching/patch_window]/content:
--- /var/cache/os_patching/patch_window 2022-07-11 11:09:20.676119445 +0000
+++ /tmp/puppet-file20220711-257440-192pvd4     2022-07-11 11:30:16.180638784 +0000
@@ -1 +1 @@
-biweekly
\ No newline at end of file
+late
\ No newline at end of file

Info: Computing checksum on file /var/cache/os_patching/patch_window
Info: /Stage[main]/Os_patching/File[/var/cache/os_patching/patch_window]: Filebucketed /var/cache/os_patching/patch_window to puppet with sum ea3762809bfb9473e886441a2a609747
Notice: /Stage[main]/Os_patching/File[/var/cache/os_patching/patch_window]/content: content changed '{md5}ea3762809bfb9473e886441a2a609747' to '{md5}f2c67381db28fa11c59fe7a6df0f2587'
Info: /Stage[main]/Os_patching/File[/var/cache/os_patching/patch_window]: Scheduling refresh of Exec[os_patching::exec::fact_upload]
Error: /Stage[main]/Os_patching/Exec[os_patching::exec::fact_upload]: Failed to call refresh: Could not find command '/usr/local/bin/puppet'
Error: /Stage[main]/Os_patching/Exec[os_patching::exec::fact_upload]: Could not find command '/usr/local/bin/puppet'
Info: Class[Os_patching]: Unscheduling all events on Class[Os_patching]
Info: Computing checksum on file /opt/puppetlabs/facter/facts.d/patching_configuration.json
Info: /Stage[main]/Patching_as_code/File[patching_configuration.json]: Filebucketed /opt/puppetlabs/facter/facts.d/patching_configuration.json to puppet with sum 85f6987da87079326a4b768682a6adad
Notice: /Stage[main]/Patching_as_code/File[patching_configuration.json]/content: content changed '{md5}85f6987da87079326a4b768682a6adad' to '{md5}1c3407b840fdba682247b4fa8e8fcbbd'
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 0.15 seconds

What behaviour did you expect instead

root@ip:/var/cache/os_patching# puppet agent -tv
Info: Using environment 'dev'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for i-0c17fbc6b2bfc0b02
Info: Applying configuration version '1657539097'

Output Log

root@ip:~# whereis puppet
puppet: /opt/puppetlabs/bin/puppet

Any additional information you'd like to impart

Upon running puppet agent -tv a second time, the expected behaviour occurs. But if there is a change to the code, the first time puppet agent runs, it throws the error as described above.

This error is present on a Windows Server 2019, as well, with the exception that the output reads:

Error: /Stage[main]/Os_patching/Exec[os_patching::exec::fact_upload]: Failed to call refresh: No such file or directory - CreateProcess
Error: /Stage[main]/Os_patching/Exec[os_patching::exec::fact_upload]: No such file or directory - CreateProcess

Notice the lack of a path here, unlike an explicit path /usr/local/bin/puppet in the Linux example.

[albatrossflavour]

Sounds like the puppet agent isn't in the path for the service account yet. Is it just on the first run? We could fully qualify the path if that makes a difference.

[albatrossflavour]

I've had a closer look at this. If this really is the very first puppet run, it could be that the standard symlinks aren't yet in place, though pretty sure the installer does that before the first proper run.

My guess is that it isn't actually working on the second run, just that you've bypassed the problem. The exec will only be called when one of the files that notify it are changed (the patch window in this case). Does /usr/local/bin/puppet exist on these nodes?

If it's causing you a significant issue and you know that the link is created later in that first run, you could put a require dependency in place when you call the os_patching class ( require => File['/usr/local/bin/puppet'],), but obviously you'd need to cater for any non *nix nodes that you might be running.

Another option is to use hiera to override os_patching::puppet_binary which is set to /usr/local/bin/puppet by default on Linux.

[gmenuel]

In puppet open source standard installation (using the packages from puppetlabs) the puppet binary is in /opt/puppetlabs/bin , not in /usr/local/bin/puppet. Since the exec already have /opt/puppetlabs/bin and /usr/local/bin in the path, the cleanest way might be to juste launch "puppet" and not the absolute path to the puppet binary. This is currently impossible to set because the puppet_binary requires an absolute path.

[nanowinner]

Hello,

Thank you for your replies! Here's the further info I can provide.

To answer your questions:

Does /usr/local/bin/puppet exist on these nodes?

No, as @gmenuel said, Puppet open source installs in opt/puppetlabs/bin, not in /usr/local/bin/ as I have also mentioned in the original post. The contents of /usr/local/bin/ on my nodes:

ubuntu@server:/usr/local/bin$ ll
total 28
drwxr-xr-x  2 root root 4096 Jul 11 12:02 ./
drwxr-xr-x 10 root root 4096 Apr 19 19:32 ../
-rwx------  1 root root 5605 Jun 13 13:02 os_patching_fact_generation.sh*

Regarding this:

The exec will only be called when one of the files that notify it are changed (the patch window in this case)

I changed the patch window by altering the hours in which it should execute (not sure if that suffices for the purpose of this test). Then I ran the puppet agent on the node, I received the following output.

root@server:~# puppet agent -vt
Info: Using environment 'dev'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for instanceID
Info: Applying configuration version '1659956250'
Info: Computing checksum on file /opt/puppetlabs/facter/facts.d/patching_configuration.json
Info: /Stage[main]/Patching_as_code/File[patching_configuration.json]: Filebucketed /opt/puppetlabs/facter/facts.d/patching_configuration.json to puppet with sum 3c3229d5cb452283cbf79dee829960b3
Notice: /Stage[main]/Patching_as_code/File[patching_configuration.json]/content: content changed '{md5}3c3229d5cb452283cbf79dee829960b3' to '{md5}7f271cfb70ed763264be5f989bf9a10d'
Notice: Applied catalog in 0.11 seconds

Regarding the symlink creation you mention here:

If this really is the very first puppet run, it could be that the standard symlinks aren't yet in place, though pretty sure the installer does that before the first proper run. My guess is that it isn't actually working on the second run, just that you've bypassed the problem.

Can you assist in checking whether or not the symlinks are there? I am not sure where or what to look for.

[gmenuel]

I changed the patch window by altering the hours in which it should execute (not sure if that suffices for the purpose of this test).

I don't think it will suffice, you need to for example change the name of the patch window. Another easier option is just to delete /var/cache/os_patching/patch_window , it will be recreated on the next run and the trigger for the exec will be run.

[nanowinner]

I don't think it will suffice, you need to for example change the name of the patch window. Another easier option is just to delete /var/cache/os_patching/patch_window , it will be recreated on the next run and the trigger for the exec will be run.

You are correct! It was not enough. Thank you for the guidance! Here is the output after I removed the patch_window file and reran puppet:

root@server:/var/cache/os_patching# rm patch_window
root@server:/var/cache/os_patching# ll
-rw-r--r--  1 root root    0 Aug  8 11:58 apps_to_restart
-rw-r--r--  1 root root    0 Aug  8 11:58 catalog_version_locked_packages
-rw-r--r--  1 root root    0 Aug  8 11:58 mismatched_version_locked_packages
-rw-r--r--  1 root root    0 Aug  8 11:58 os_version_locked_packages
-rw-r--r--  1 root root   54 Aug  8 11:58 package_updates
-rw-r--r--  1 root root    7 Jun 13 13:02 reboot_override
-rw-r--r--  1 root root    5 Aug  8 11:58 reboot_required
-rw-r--r--  1 root root    0 Aug  8 11:58 security_package_updates
root@server:/var/cache/os_patching# puppet agent -tv
Info: Using environment 'dev'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for instanceID
Info: Applying configuration version '1659961300'
Notice: /Stage[main]/Os_patching/File[/var/cache/os_patching/patch_window]/ensure: defined content as '{md5}ea3762809bfb9473e886441a2a609747' (corrective)
Info: /Stage[main]/Os_patching/File[/var/cache/os_patching/patch_window]: Scheduling refresh of Exec[os_patching::exec::fact_upload]
Error: /Stage[main]/Os_patching/Exec[os_patching::exec::fact_upload]: Failed to call refresh: Could not find command '/usr/local/bin/puppet'
Error: /Stage[main]/Os_patching/Exec[os_patching::exec::fact_upload]: Could not find command '/usr/local/bin/puppet'
Info: Class[Os_patching]: Unscheduling all events on Class[Os_patching]
Notice: Applied catalog in 0.08 seconds

Thus, albatrossflavour was correct:

My guess is that it isn't actually working on the second run, just that you've bypassed the problem.

Let me know if I can provide further info!

[albatrossflavour]

I did some testing using the Ubuntu packaged puppet agent, which installs in /usr/bin. I get the same errors about not being able to find the agent.

I changed the puppet_binary parameter and things worked much better:

class base {
  class { 'os_patching':
    puppet_binary => '/usr/bin/puppet',
  }
}

Could you try amending that and including it in your setup?

[nanowinner]

I followed the installation steps as defined in the official Puppet Open Source documentation:
Configure Puppet 6 repository and Install Puppet Agents

1. Download the latest version of Puppet on a client node:
   wget https://apt.puppet.com/puppet6-release-focal.deb

2. Install the package by using:
   sudo dpkg -i puppet6-release-focal.deb

3. Update the package repository one more time:
   sudo apt-get update -y

4. Install the Puppet agent by using:
   sudo apt-get install puppet-agent -y

5. Locate puppet

ubuntu@brand-new-ubuntu-20-04:~$ whereis puppet
puppet: /opt/puppetlabs/bin/puppet

I could definitely try including the path /opt/puppetlabs/bin/puppet as a parameter under the os_patching class, but should it not be handled as a genuine installation location option, given that is what the docs guide people to?

[Perkka2]

The same issue and install location is valid for Debian 10 and Debian 11 for Puppet 7

[nanowinner]

Do we have any info on the progress of this issue?

[nanowinner]

Has there been any investigation ran into this?

[albatrossflavour]

OK. I've cleared the decks on the other PRs and issues. I'm going to take a look at this tomorrow.

[albatrossflavour]

Since this only (?) affects open-source installs, it might be best to look at having a separate hiera entry to supply the puppet_binary value depending on if you're using PE or POS.

I'll do some more testing in the morning,

[albatrossflavour]

I've pushed a new branch up (#215), which has a possible workaround. I'm not happy with it but happy to discuss alternatives and for you lot to kick the tyres.

Let me know what you think @nanowinner

[jcpunk]

How is the testing going?

[albatrossflavour]

Haven't heard from anyone yet @jcpunk. It's passed my testing, but I'm not 100% happy with the method. Would really like to get other people to weigh in before we release it. I think I'm going to cut a new release for the forge today which will have the fixes already committed.

[albatrossflavour]

@nanowinner @gmenuel @jcpunk @Perkka2 Has anyone been able to take a look at this branch and do some testing on it?

[jcpunk]

I'm not showing any problems in my testing.