[BUG] Latest update causing minions to throw Unable to sign_in to master: Invalid master key despite master key being correct. Deleting minion_master.pub recreates it identically.

[ipaqmaster]

Description

Doing the rounds of updates for CentOS 7, Rocky 9.2 and two Archlinux salt-masters has resulted in the first two distros to throw the familiar The master key has changed paragraph saying it can be found at /etc/salt/pki/minion/minion_master.pub. The pubkey in that file is accurate and deleting it before doing another test.ping on any of the minions simply causes it to get recreated.

My initial thoughts leave me wondering if there's some crypto library difference or modern policy causing this.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

The environment is a mix of newer (replacement) and older (to be replaced) servers. We're moving from CentOS 7 (And older) to Rocky 9.2, though even the Rocky hosts are experiencing problems speaking to their master server.

There are two salt masters. One for test/dev and another for prod which reads the prod branch consisting of approved commits of the test/dev salt server's salt root.

The two salt masters run Archlinux and this has been flawless for their 8 or so months of operation.

The current salt-master version is: 3006.5 (Sulfur)

The current salt-minion version for Rocky is: 3007.0 (Chlorine) (3006.7 also did not work resulting in this upgrade)

The current salt-minion version for CentOS is: 3006.7 (Sulfur)

Prior to this months run of updates all was well but now these RedHat-based distros cannot speak to the salt-master despite having and re-generating the same minion_master.pub file.

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

1. salt-call test.ping on a minion already accepted on the master and known-working after the most recent updates
2. See The master key has changed spiel.

Expected behavior
A clear and concise description of what you expected to happen.

The module to return True.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

PASTE HERE

Additional context
Add any other context about the problem here.

[ipaqmaster]

It seems CentOS and Rocky now have 3007.0 in their repos as checked this morning but Archlinux does not.

On the Archlinux package page for Salt, it is already flagged as "Out of date" so once the newer version is packaged and available I will upgrade the two master's to that and see if the issue resolves.

[ipaqmaster]

Its difficult. The rpm repos and salt bootstrap script won't go lower than 3006.7. I'm trying to build 3006.7 myself for Arch to see if that gets past this.

Starting the master with debug logging on the command line shows it clearly signing and responding to minions. There is evidently some communication different or a library problem.

[ipaqmaster]

That did the trick. I'll let the Arch maintainers know about this in hopes they can build 3006.7 officially.

• Here is that issue: https://gitlab.archlinux.org/archlinux/packaging/packages/salt/-/issues/1

• It appears a similar experience with 3006.5 and >=3006.7 is also discussed here: https://www.mail-archive.com/ports@openbsd.org/msg124133.html