(PDB-4945) Default to json fact cache

[joshcooper]

Puppetserver 7 now defaults the fact cache to json[1]. When running in
agent-server mode, configure routes.yaml the same way. There is no change in
behavior when running with puppetserver < 7 or puppet apply.

As a result of this change, when the puppetdb module is used to install the
puppetdb terminus and routes.yaml, facts will now be cached as non-pretty json
in:

 /opt/puppetlabs/server/data/puppetserver/server_data/facts/<node>.json

Instead of pretty yaml in:

 /opt/puppetlabs/server/data/puppetserver/yaml/<node>.yaml

The difference in pathing is due to the abstract yaml and json termini using
different base directories in which to store data: Puppet[:yamldir] vs
Puppet[:server_data].

[1] https://github.com/puppetlabs/puppetserver/blob/794e725628545aa075e3f8b5fa91f21628373594/src/ruby/puppetserver-lib/puppet/server/puppet_config.rb#L56

[puppet-community-rangefinder]

puppetdb::master::routes is a class

that may have no external impact to Forge modules.

This module is declared in 33 of 575 indexed public Puppetfiles.

---

These results were generated with Rangefinder, a tool that helps predict the downstream impact of breaking changes to elements used in Puppet modules. You can run this on the command line to get a full report.

Exact matches are those that we can positively identify via namespace and the declaring modules' metadata. Non-namespaced items, such as Puppet 3.x functions, will always be reported as near matches only.

[joshcooper]

I think we still want -1 here so we use yaml on < 7.0:

$ bx puppet apply -e "notice(versioncmp('6.19', '7.0'))"
Notice: Scope(Class[main]): -1

[mihaibuzgau]

Updated.
Now if serverversion is defined and if the version is greater than 7.0, we're going to prefer json. If not we're going to prefer yaml

[joshcooper]

With environment variables:

BEAKER_HELPER=acceptance/lib/helper.rb
BEAKER_LOADPATH=ruby/puppet/acceptance/lib
BEAKER_OPTIONSFILE=acceptance/config/beaker/options.rb
BEAKER_POSTSUITE=acceptance/suites/post_suite
BEAKER_PRESUITE=acceptance/suites/pre_suite/foss
BEAKER_TESTSUITE=ruby/puppet/acceptance/tests/puppet_apply_basics.rb
BEAKER_TYPE=aio
GEM_SOURCE=https://artifactory.delivery.puppetlabs.net/artifactory/api/gems/rubygems/
PACKAGE_BUILD_VERSION=7.0.0.SNAPSHOT.2020.11.02T1917
PLATFORM_VERSION=puppet7-nightly

I installed puppetdb using puppetserver CI:

$ bundle exec beaker-hostgenerator redhat7-64m-64a > redhat7-64m-64.cfg
$ ./acceptance/scripts/generic/testrun.sh -p

At this point there are json facts cached in the new place (due to agents running prior to the old puppetdb being installed):

# ls -la /opt/puppetlabs/server/data/puppetserver/server_data/facts/
total 44
drwxr-x--- 2 puppet puppet  4096 Nov  4 23:29 .
drwxr-x--- 3 puppet puppet  4096 Nov  4 23:28 ..
-rw-rw---- 1 puppet puppet 13866 Nov  4 23:28 nucleic-germ.*.json
-rw-rw---- 1 puppet puppet 10879 Nov  4 23:29 real-default.*.json
-rw-rw---- 1 puppet puppet  4840 Nov  4 23:28 sublime-lawyer.*.json

Then I copied a build of this module to the server:

# cd /etc/puppetlabs/code/environments/production/modules
# mv puppetdb puppetdb-old
# tar -zxvf ~/puppetlabs-puppetdb-7.6.0.tar.gz
# mv puppetlabs-puppetdb-7.6.0 puppetdb
# cat /etc/puppetlabs/code/environments/production/manifests/site.pp
  class { 'puppetdb':
    manage_firewall => false,
  }

  class { 'puppetdb::master::config':
    manage_report_processor => true,
    enable_reports          => true,
  }
# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
/opt/puppetlabs/puppet/vendor_modules/cron_core/lib/puppet/type/cron.rb:160: warning: deprecated Object#=~ is called on Integer; it always returns nil
/opt/puppetlabs/puppet/vendor_modules/cron_core/lib/puppet/type/cron.rb:165: warning: deprecated Object#=~ is called on Integer; it always returns nil
/opt/puppetlabs/puppet/vendor_modules/cron_core/lib/puppet/type/cron.rb:170: warning: deprecated Object#=~ is called on Integer; it always returns nil
/opt/puppetlabs/puppet/vendor_modules/cron_core/lib/puppet/type/cron.rb:76: warning: deprecated Object#=~ is called on Integer; it always returns nil
/opt/puppetlabs/puppet/vendor_modules/cron_core/lib/puppet/type/cron.rb:160: warning: deprecated Object#=~ is called on Integer; it always returns nil
/opt/puppetlabs/puppet/vendor_modules/cron_core/lib/puppet/type/cron.rb:165: warning: deprecated Object#=~ is called on Integer; it always returns nil
/opt/puppetlabs/puppet/vendor_modules/cron_core/lib/puppet/type/cron.rb:170: warning: deprecated Object#=~ is called on Integer; it always returns nil
/opt/puppetlabs/puppet/vendor_modules/cron_core/lib/puppet/type/cron.rb:76: warning: deprecated Object#=~ is called on Integer; it always returns nil
Info: Caching catalog for real-default.*.net
Info: Applying configuration version '1604533695'
Warning: The method 'Puppet::Network::HttpPool.http_instance' is deprecated. Use Puppet.runtime[:http] instead
   (file & line not available)
...
Notice: /Stage[main]/Puppetdb::Master::Routes/File[/etc/puppetlabs/puppet/routes.yaml]/content:
--- /etc/puppetlabs/puppet/routes.yaml	2020-11-04 23:29:57.824250144 +0000
+++ /tmp/puppet-file20201104-23160-11hxo4i	2020-11-04 23:48:20.558316985 +0000
@@ -4,5 +4,5 @@
 master:
   facts:
     terminus: puppetdb
-    cache: yaml
+    cache: json

Notice: /Stage[main]/Puppetdb::Master::Routes/File[/etc/puppetlabs/puppet/routes.yaml]/content: content changed '{sha256}3dd38adda5913e3dfb0b36e98af75c9192d66b90380ddffe943b5376922ec627' to '{sha256}aa643bfc4074016c18033c466e513d70ccdc44a5b245fa625013003ccc3ebd3e'
Info: Class[Puppetdb::Master::Routes]: Scheduling refresh of Service[puppetserver]
Notice: /Stage[main]/Puppetdb::Master::Config/Service[puppetserver]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 18.78 seconds

The cron_core issue is fixed in puppetlabs/puppetlabs-cron_core@ae6c349

Since the old terminus is in effect when the agent gets its catalog, the server writes facts as yaml to the old place:

# ls -la /opt/puppetlabs/server/data/puppetserver/yaml/facts/
total 20
drwxr-x--- 2 puppet puppet  4096 Nov  4 23:48 .
drwxr-x--- 3 puppet puppet  4096 Nov  4 23:44 ..
-rw-rw---- 1 puppet puppet 11667 Nov  4 23:48 real-default.*.yaml

And from that point forward, the json facts are updated with each run:

# ls -la /opt/puppetlabs/server/data/puppetserver/server_data/facts/real-default.*.json
-rw-rw---- 1 puppet puppet 10897 Nov  5 00:00 /opt/puppetlabs/server/data/puppetserver/server_data/facts/real-default.*.json