Upgrade from Debian 7 Wheezy (Puppet 3) to Debian 11 Bullseye (Puppet 7)

[Krinkle]

List of hosts

https://github.com/jquery/infrastructure/blob/puppet-stage/manifests/site.pp

• puppet.ops.jquery.net
• wp-01, jquery.com
  • jquery.com
  • api.jquery.com
  • learn.jquery.com
  • plugins.jquery.com
• wp-02, most other sites (incl *.jquery.org, jqueryui.com, etc)
• wp-03, codeorigin.jquery.com, releases.jquery.com, and recipient of Git assets
• wp-01.stage, WordPress doc sites, staging, all domains (stage.api.jquery.com, etc)
• builder-01
• builder-03.stage
• jq03.stage.jquery.com (stage.demos.jquerymobile.com, stage.themeroller.jquerymobile.com)
• jenkins-01 decommissioned
• cla-01.ops.jquery.net
• cla-01.stage.jquery.net
• gruntjs.ops.jquery.net
• gruntjs.stage.jquery.net
• origin-01.ops.jquery.net, contentorigin (content.jquery.com, static.jquery.com)
• swarm-01.ops.jquery.net, TestSwarm
• view-01.ops.jquery.net, View, git assets
• trac.ops.jquery.net, Trac, (bugs.jquery.com, bugs.jquerui.com)

Dedicated tickets:

• Reduce jQuery CDN risk, separate and upgrade codeorigin site. https://github.com/jquery/infrastructure/issues/554
• Upgrade TestSwarm to PHP 7. https://github.com/jquery/infrastructure/issues/444
• Upgrade WordPress hosts to PHP 7. Upgrade WordPress hosts to PHP 8 #6
• Trac: Replace with static export. Make Trac read-only, possibly replace by a static dump #10
• CLA export and decom. Decom CLA droplets #12
• Plugins: Replace with static export Archive plugins.jquery.com, replace with static site #29

Overview

In order to get away from the very outdated Debian versions and such, we need to also get to a newer Puppet version.

We are currently using numerous Puppet 2 features that were deprecated in Puppet 3 and removed in Puppet 4. The main change that I think affects us is the change from "environment configs" to "environment directories".

Some relevant links:

• https://elatov.github.io/2016/07/upgrade-puppet-3x-to-4x/
• https://puppet.com/docs/puppet/4.10/upgrade_major_server.html
• https://puppet.com/docs/puppet/3.8/environments.html
• https://puppet.com/docs/puppet/3.8/dirs_manifest.html

Status quo: Puppet 3

The puppet server runs at puppet.ops.jquery.net (in legacy docs: puppet-master). The config for the server is at /etc/puppet/puppet.conf. There are two Git clones that we care about on this server:

• /etc/puppet - This is a clone of jquery/infrastructure.git at branch puppet-master. This currently replaces the entire /etc/puppet directory.
• /etc/puppet-stage – This is a directory we made up, containing another clone of jquery/infrastructure.git at branch puppet-stage.

In /etc/puppet/puppet.conf (the only place the Puppet server actually looks at) we have the following stuff:

[main]
# …
templatedir=$confdir/templates
manifest=/etc/puppet/manifests/site.pp

[stage]
manifest=/etc/puppet-stage/manifests/site.pp
modulepath=/etc/puppet-stage/modules
# …

[master]
# …

By default, with one of our droplets that runs a puppet agent asks for provisioning, it gets provisoned by the main config which points simply at the subdirectories within /etc/puppet. On staging hosts, we have another /etc/puppet/puppet.conf file that may contain environment = stage, which the agent passes on to the Puppet server, and so the Puppet server will consider that manifest and modulepath directory instead (in addition to compling it with $::environment = "stage").

Beyond this, the only other thing worth knowing is that we use jquery::postreceive instances (similar to for the content sites) to automatically update these git checkouts after commits to them. The actual applying of changes however is passive, based on puppet agents checking in with the server every 30 minutes (default Puppet agent behaviour).

Puppet 4

Under Puppet 4, things are a little bit different. There is no longer support for the templatedir, manifest, and modulepath parameters, and there is no longer support for per-environment configuration section overrides.

Instead, modules are read from a directory like /etc/puppet/code/environments/:environment/modules and manifests are read from a directory like /etc/puppet/code/environments/:environment/manifests. For example: /etc/puppet/code/environments/production/modules.

I think global templates are no longer supported, or at least not varying by environment. But that's okay, we only have one file in /templates and that'll either just not support staging or maybe we can even get rid of it (do we still use Zabbix?).

The new directory layout seems feasible, we just create two more clones and keep both for a little while.

Transition

I noticed just now that, apart from a few minor tweaks being needed for deprecated features, more generally it is not supported to connect Puppet 4 clients to a Puppet 3 server. However, the other way around is supported. So, the puppet master will have to go first, and that means a master switch, and setting up a new one of those first as well.

The good news is, a Puppet server is relatively easy to configure and gradually switch to...

[Krinkle]

Things I think we are not using, and I will omit initially in the Puppet 4 branch:

• Zabbix
• New Relic
• Postfix
• etckeeper

I'll mention this during this Friday's infra meeting (tomorrow) in case any we know of any of these definitely still being used.

/cc @mgol @brianwarner

[mgol]

I don't know much about our usage of any of these services with the exception of the fact that running:

sudo puppet agent --test

on jenkins-01 now results in the following output:

Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for jenkins-01.ops.jquery.net
Info: Applying configuration version '1616105347'
Notice: /Stage[main]/Main/Node[default]/Apt::Source[dotdeb]/Apt::Key[Add key: 3D624A3B from Apt::Source dotdeb]/Apt_key[Add key: 3D624A3B from Apt::Source dotdeb]/ensure: created
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install libasound2 ' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libasound2
Error: /Stage[main]/Main/Node[jenkins]/Package[libasound2 ]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install libasound2 ' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libasound2
Notice: /Stage[main]/Jquery::Newrelic/Exec[install newrelic license]/returns: executed successfully
Error: Could not start Service[newrelic-sysmond]: Execution of '/usr/sbin/service newrelic-sysmond start' returned 1: Job for newrelic-sysmond.service failed. See 'systemctl status newrelic-sysmond.service' and 'journalctl -xn' for details.
Wrapped exception:
Execution of '/usr/sbin/service newrelic-sysmond start' returned 1: Job for newrelic-sysmond.service failed. See 'systemctl status newrelic-sysmond.service' and 'journalctl -xn' for details.
Error: /Stage[main]/Jquery::Newrelic/Service[newrelic-sysmond]/ensure: change from stopped to running failed: Could not start Service[newrelic-sysmond]: Execution of '/usr/sbin/service newrelic-sysmond start' returned 1: Job for newrelic-sysmond.service failed. See 'systemctl status newrelic-sysmond.service' and 'journalctl -xn' for details.
Notice: /Stage[main]/Main/Node[jenkins-01.ops.jquery.net]/Jquery::Ssh::Host[jenkins]/Exec[chmod 0600 /etc/ssh/ssh_host*]/returns: executed successfully
Notice: /Stage[main]/Main/Node[jenkins-01.ops.jquery.net]/Jquery::Ssh::Host[jenkins]/Exec[chmod 0644 /etc/ssh/*.pub]/returns: executed successfully
Notice: Finished catalog run in 8.30 seconds

so it looks like New Relic is somehow interfering with Puppet runs. I'm not sure if those errors are blocking anything.

[Krinkle]

Aye, that looks familiar. I'll continue the jenkins-01 issue at https://github.com/jquery/infrastructure/issues/433.

[Krinkle]

This cleanup was useful and benefitted our current droplets as well.

But, the issue as originally written I'm closing for now per https://github.com/jquery/infrastructure/issues/482#issuecomment-907890935. Might pick it up again depending on whether if/when we're going to have droplets running newer Debian versions.

[Krinkle]

@atdt I've created the puppet-02.ops.jquery.net instance (IP: 104.131.63.112, DNS not yet assigned), with a Debian 11 image, a small 2-CPU / 4GB RAM plan, and both of our SSH keys attached for initial bootstrapping.

Empty repo for Puppet manifests: https://github.com/jquery/infrastructure-puppet.
This is a new repo rather than a branch, so that we can manage most server configuration in public going forward. I suppose we can keep issue tracking and wiki pages in this repo for now. To be discussed at the infra meeting.

[brianwarner]

This is in place as well, also with proxying enabled until you tell me not to.

[Krinkle]

@brianwarner Aye, yeah, this one should be without proxying as it's for internal use such as shell access and receiving webhooks.

[atdt]

What version of Puppet should we target?

Puppet <= 5 has already reached EOL, and Puppet 6 is projected to reach EOL in less than a year. (See: Puppet platform lifecycle.

OTOH, Puppet 7 has not yet been packaged for Debian 11 Bullseye. Puppet Labs estimates packages for Debian 11 will be available within the next month.

[Krinkle]

@atdt I see. I suppose we could wait another month.

Alternatively, we could go with Puppet 7 now if we use Debian 10 Buster, I think?
https://puppet.com/docs/puppet/7/server/install_from_packages.html

I don't see Debian 6 for Debian 11 Bullseye, but I'm not sure if I'm looking in the right place, is there one? I can't tell from the raw index at https://apt.puppet.com/. Either way, I imagine from one major Debian or Puppet to the next should be relatively simple with a few inline conditionals perhaps.