GitLab 18 → 19 upgrade blocked by removed Mattermost (omnibus / Linux package)

Upgrading a self-hosted GitLab CE/EE omnibus (Linux package) install from 18.x to 19.x aborts during the package pre-install step with:

* mattermost has been deprecated since 19.0 and was removed in 19.0. Bundled Mattermost
  has been removed from the Linux package in 19.0; `mattermost[...]` keys are no longer
  supported. Deploy Mattermost separately and point GitLab at it with
  `gitlab_rails['mattermost_host']`. ...
Deprecations found. Please correct them and try again.
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_19.x.x-ce.0_amd64.deb (--unpack):
 new gitlab-ce package pre-installation script subprocess returned error exit status 1

The confusing part: you get this even if you never used Mattermost and it isn't running, and editing gitlab.rb doesn't fix it. This repo explains why and gives the fix.

⚠️ Read this first. This is for the common case where bundled Mattermost is NOT actually in use — the block is leftover secrets/attributes from an old default. If you do run bundled Mattermost and have data in it, do not just delete it: migrate Mattermost to a standalone deployment first (see GitLab's docs), then apply this. The same procedure applies to the spamcheck service, which 19.0 also removed.

---

TL;DR

The 19.0 pre-install deprecation gate reads the compiled node attributes, not gitlab.rb. Clear the dead mattermost/spamcheck blocks from the secrets file and the node attributes, confirm the check passes, then upgrade one required stop at a time.

# 0. Back up first (secrets + full app backup)
sudo gitlab-ctl backup-etc
sudo gitlab-backup create

# 1. Remove the dead top-level "mattermost" key from the secrets file
sudo cp -a /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.bak
sudo /opt/gitlab/embedded/bin/ruby scripts/remove-secrets-key.rb mattermost

# 2. Remove "mattermost" and "spamcheck" from the compiled node attributes
#    (this is the file check-config actually reads)
sudo /opt/gitlab/embedded/bin/ruby scripts/clear-node-attrs.rb mattermost spamcheck

# 3. Confirm the exact check the installer runs now passes (exit 0, no output)
sudo /opt/gitlab/bin/gitlab-ctl check-config --version=19.0 ; echo "exit=$?"

# 4. Upgrade through the required stop (latest 19.0 first, then your target 19.x)
sudo apt-get update
sudo apt-get install -y gitlab-ce=19.0.4-ce.0     # use the latest 19.0.x available
#   ...wait for background migrations to finish (see below), then:
sudo apt-get install -y gitlab-ce=19.1.2-ce.0     # your target 19.x

---

Why it happens

GitLab 19.0 removed bundled Mattermost and Spamcheck from the Linux package. The .deb's pre-install script runs a deprecation gate:

gitlab-ctl check-config --version=<incoming_minor>

The non-obvious detail is what that check reads. It does not parse gitlab.rb or gitlab-secrets.json directly. From check_config.rb:

node_json_file = Dir.glob("#{base_path}/embedded/nodes/*.json")[0]
node_json      = JSON.load_file(node_json_file)
existing_config = node_json['normal']
messages = Gitlab::Deprecations.check_config(opts[:version], existing_config, :removal)

It reads the compiled node attributes written by the last gitlab-ctl reconfigure — /opt/gitlab/embedded/nodes/<fqdn>.json, under the normal key — and flags any top-level mattermost (or spamcheck) block, per deprecations.rb:

{ config_keys: %w(mattermost), deprecation: '19.0', removal: '19.0', note: "..." },
{ config_keys: %w(spamcheck),  deprecation: '19.0', removal: '19.0', note: "..." },

Where does the mattermost block come from if it isn't in your gitlab.rb? gitlab-secrets.json. Older installs carry a mattermost secrets section (OAuth-app id/secret, salts) generated when Mattermost was part of the default package. At reconfigure, those secrets are merged into the node's normal attributes — so the gate sees normal.mattermost and fails, even though nothing Mattermost is enabled or running.

That's why editing gitlab.rb does nothing — the offending data lives in the secrets file and the compiled node attributes.

---

Diagnose

# Current version / install method
sudo gitlab-rake gitlab:env:info | grep -i version
dpkg -l | grep -iE 'gitlab-(ce|ee)'

# Is bundled Mattermost actually running? (usually: no)
sudo gitlab-ctl status mattermost || echo "not present"

# Where the leftover mattermost data lives
sudo grep -n '"mattermost"' /etc/gitlab/gitlab-secrets.json
sudo grep -rn '"mattermost"' /opt/gitlab/embedded/nodes/*.json

# Reproduce the exact gate the installer runs
sudo /opt/gitlab/bin/gitlab-ctl check-config --version=19.0 ; echo "exit=$?"

If check-config exits non-zero and prints the mattermost message, you have this problem.

---

Fix (details)

1. Back up. A major upgrade runs schema migrations; have a restore path.

sudo gitlab-ctl backup-etc        # /etc/gitlab (gitlab.rb + gitlab-secrets.json) -> config_backup/
sudo gitlab-backup create         # application data -> /var/opt/gitlab/backups/

2. Remove the dead mattermost key from the secrets file. Use the included script (it rewrites valid JSON and re-parses to validate):

sudo cp -a /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.bak
sudo /opt/gitlab/embedded/bin/ruby scripts/remove-secrets-key.rb mattermost

3. Clear mattermost/spamcheck from the compiled node attributes — the file the check actually reads:

sudo /opt/gitlab/embedded/bin/ruby scripts/clear-node-attrs.rb mattermost spamcheck

Don't try to fix this by running gitlab-ctl reconfigure on 18.x: the 18.x package still ships the Mattermost cookbook, so a reconfigure will just regenerate the secrets and re-add the block. Edit the files directly; the 19.x reconfigure (which has no Mattermost cookbook) regenerates everything cleanly.

4. Verify the gate passes:

sudo /opt/gitlab/bin/gitlab-ctl check-config --version=19.0 ; echo "exit=$?"   # want exit=0, no output

5. Upgrade through the required stop. GitLab enforces an upgrade path — you must land on the first minor of the new major (19.0) from the last minor of the old major (18.11) before going to a later 19.x. A plain apt upgrade tries to jump straight to the newest 19.x and is rejected. Install explicit versions:

sudo apt-get update
apt-cache madison gitlab-ce | grep 19.0        # find the latest 19.0.x
sudo apt-get install -y gitlab-ce=19.0.4-ce.0  # hop 1: the required 19.0 stop (reconfigures automatically)

Wait for that version's batched background migrations to finish before the next hop:

# done = status 3 (finished) or 6 (finalized); wait until nothing is 1/active, 0/paused, 4/failed
sudo gitlab-psql -tAc "select status, count(*) from batched_background_migrations group by status order by status"
sudo gitlab-rake db:migrate:status | grep -c '^ *down'   # want 0

Then hop to your target 19.x:

sudo apt-get install -y gitlab-ce=19.1.2-ce.0  # hop 2 (or your chosen 19.x)

Confirm health (allow a few minutes for Puma to boot — see gotchas):

sudo gitlab-ctl status
curl -ksS -o /dev/null -w "%{http_code}\n" https://127.0.0.1/-/readiness   # want 200

---

Gotchas worth knowing

• The empty placeholder is fine. After the 19.x reconfigure you may still see normal.mattermost = {} (an empty hash) in the node attributes. That's harmless — the deprecation check only flags non-empty blocks, so an empty hash won't trip 19.x or 20.0.
• The "skip the check" file was renamed. The pre-install honors /etc/gitlab/skip-fail-config-check to downgrade a fatal deprecation to a warning. Older docs/installs reference skip-failed-config-check (with an extra "ed"), which the current preinst ignores. Fixing the config properly (above) is better than skipping the check anyway.
• finalize job-arguments quoting. If a migration finishes all its jobs but stays active, finalize it with:

  sudo gitlab-rake "gitlab:background_migrations:finalize[JobClassName,table,column,[]]"

  The empty job-args must be a bare [] — the quoted '[]' form throws JSON::ParserError.
• 502 right after a reconfigure is normal. Each hop restarts Puma, which preloads the Rails app for a few minutes (longer on low-core VMs) before it serves. nginx returns 502 until then. Watch /var/log/gitlab/puma/current for Use Ctrl-C to stop, which means it's up.

---

Scripts

• scripts/remove-secrets-key.rb — removes a top-level key from /etc/gitlab/gitlab-secrets.json, keeping valid JSON.
• scripts/clear-node-attrs.rb — removes one or more keys from node['normal'] in /opt/gitlab/embedded/nodes/*.json (backs up each file first).

Both are read-mostly, make backups, and re-parse their output to validate. Run them with GitLab's embedded Ruby (/opt/gitlab/embedded/bin/ruby) so no extra gems are needed.

Disclaimer

Provided as-is, no warranty. Test on a snapshot/backup first. Paths and behavior are for the omnibus / Linux package GitLab install and were verified on a 18.11.7 → 19.0.4 → 19.1.2 CE upgrade; Docker and source installs differ.

License

MIT