general: update base image to Debian 12 ("Bookworm") #3403

georglauterbach · 2023-06-20T20:16:23Z

Description

Update to Debian 12 ("Bookworm") as the base image and fix/adjust all parts affected by this change.

Superseeds #3402

Type of change

Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation (README.md or the documentation under docs/)
If necessary I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

docs/content/config/security/rspamd.md

polarathene

Bulk of this review is just notes for my benefit, and optional / future changes.

I don't mind applying these myself, but I'll let the feedback available for discussion for a day or so prior. I have some other tasks to pursue and get PRs opened for, this review was a bit of a time sink detour due to SA 😅

Dockerfile

target/postfix/main.cf

target/scripts/build/packages.sh

test/tests/parallel/set1/spam_virus/amavis.bats

test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats

target/scripts/build/packages.sh

polarathene · 2023-08-27T00:41:01Z

target/scripts/build/packages.sh

-  _log 'debug' 'Installing Fail2ban'
-  apt-get "${QUIET}" --no-install-recommends install python3-pyinotify python3-dnspython


I almost forgot about this 😅

You've skipped these two packages, but they're added for a reason to support Fail2Ban better, they're still relevant, check git blame (commit message below, see relevant section of PR #3032 for more details):

fix: Install optional python packages for fail2ban

These have barely any overhead in layer weight. The DNS package may provide some QoL improvements, while the pyinotify is a better alternative than polling logs to check for updates.

_We have gamin package installed but fail2ban would complain in the log that it was not able to initialize the module for it. There only appears to be a python-gamin dependent on EOL python 2, no longer available from Debian Bullseye.

python3-pyinotify is also listed as a recommended package in Debian fail2ban, as is nftables, whois, bsd-mailx (elsewhere in packages.sh but specifically for fail2ban support).

I will add this back.

polarathene · 2023-08-27T04:03:13Z

Review context regarding version changes in Debian 12 packages, plus some related history/notes for my benefit 😅

Major version updates should be included in changelog.

fetchmail no notable change, 6.4 on both Debian 11 and 12.
getmail6 (6.18 => 6.18)
- Debian 11 provided 6.14 in Bullseye repo and 6.18 in Bullseye-backports repo, but we were installing getmail6 via the python package manager pip. Switching to Debian package with Debian 12.
openssl (1.1.1n => 3.0.11)
- Tests caught a slight difference in some output. I haven't looked over the changelog for if this major release impacts anything else.
amavisd-new (2.11 => 2.13)
- Adds dep of systemd | systemd-standalone-sysusers | systemd-sysusers. Installing systemd-standalone-sysusers is an appropriate non-default choice for our non-systemd requirement, we should add that dependency explicitly.
- NOTE: Instead of check_policy_service integration to Postfix, amavisd-milter package could be used instead.
- Suggested deps (which DMS explicitly installs):
  - clamav (0.103.8 => 1.0.3)
  - spamassassin (3.4.6 => 4.0.0)
postfix (3.5.23 => 3.7.9)
redis-server (6.0 => 7.0)
rsyslog (8.2102 => 8.2302)
supervisor (4.2.2 => 4.2.5)
dovecot-core (2.3.13 => 2.3.19)
- The current default in DMS v13.3 is a community edition install (from Dovecot repos instead of Debian) that is presently at 2.3.21 (Sep 2023 release).
- Debian 12 (Bookworm) is expected to get 2.3.21 or newer (Dovecot devs plan to release 2.4.0 in 2023H2) via the bookworm backports repo later this year. UPDATE: As of Jan 2024, this is still not the case.
- Downgrading to Dovecot 2.3.19 notably affects support for OAuth2 (with JWT or using client_id + client_secret with POST requests) and Lua HTTP client (DNS related).
dovecot-fts-xapian (1.4.9 => 1.5.5)
- libxapian30 dep >= 1.4.19 + dovecot-abi-2.3.abiv19 virtual dep (provided via dovecot-core)
- Package previously broke compatibility with Debian 11 Bullseye packages when mixed with third-party Dovecot repo for a newer dovecot-core which could no longer satisfy the ABI virtual dependency.
  - Solution was to build package dovecot-fts-xapian 1.5.5 from source via introducing build/compile.sh.
  - dovecot-fts-xapian was originally integrated into DMS in July 2021 via single package and a documented config guide.
- When dovecot-core 2.4.0 is available, migrating to the flatcurve plugin becomes a viable option and may instead be a better replacement for this feature.
- Additionally, the suggestion to migrate to the flatcurve plugin was proposed by the contributor that added support for the xapian plugin in DMS (which was due to the existing solr plugin having notable drawbacks).
postsrsd (1.10 - no notable change from Debian 11)
- There is a bug within common container environments that have containerd or similar service on the host configured with LimitNOFILE=infinity where fs.nr_open is a value of 2^30 instead of 2^20 (Debian).
  - That bug was fixed in Aug 2022 but only available with PostSRSd v1.12 / v2.
  - Thus users of this feature in DMS may need to explicitly configure the --ulimit to 1024:524288 (systemd implicit default since v240).
  - The fix also requires at least glibc of 2.35, which Debian 12 provides with glibc 2.36. Docker host kernel is required to be >= 5.9.
postfix-policyd-spf-python (2.9.2 => 3.04)
- 3.x dropped the python3-spf dep, which supposedly is not needed anymore to provide Postfix with an SPF check_policy_service.
- NOTE: There also exists an alternative milter-based package pyspf-milter which appears to be part of python3-spf-engine too (available in spf-engine since 2.9.0). See the spf-engine README for advice on configuring Postfix with milter package.
- References:
  - spf-engine upstream source and Debian source.
  - Some resources online provide nicer docs to browse.
    - policyd-spf Postfix integration advice (states version 2.0, but this seems to match the python-policy-spf 2.0.0 mention in README source (upstream)) notes that when using the check_policy_service it must come after reject_unauth_destination to avoid creating an open relay (DMS handles this correctly at present, but that expectation is not clearly documented inline for maintainers).
    - Similar pyspf-milter docs.

Regarding base image change consideration for Alpine vs Fedora:

getmail6 - Fedora and Alpine both lack getmail6 and would need to install via pip. At least for Alpine 3.18 there is no getmail6 package, Alpine edge has a package which may make it available in a future release (Alpine 3.19?).
rspamd - Fedora does not have any official package for this. There is a relatively maintained third-party COPR repo (github repo), and I had some success IIRC building for Fedora via Dockerfile myself in the past. Would be better if upstream rspamd had more official support, but unclear when that'd happen and that's one of the bigger drawbacks for choosing Fedora.
postfix-policyd-spf-python - Alpine doesn't have this, closest might be a Perl version, Fedora has it as pypolicyd-spf. Probably acceptable if rspamd becomes the only option in future release of DMS.
spamassassin - Alpine is still on 3.4, even on edge channel (with a July 2023 build).
postgrey - Alpine doesn't seem to have this package available (nor on edge).

I still have reasons for cautioning against Alpine as the base image for DMS, but lacking time to do a write-up going into detail 😅

EDIT: Presently there is a bug with QEMU that's being triggered by ARM builds that affects images using Alpine musl, stalling due to timeout. Thought I'd mention that as it's affecting quite a few CI users building docker images that use Alpine. As mentioned before, there is a variety of problems you can experience adopting Alpine and I'd strongly discourage that for DMS.

Also as an additional reference, there was a recent discussion about base image change and migrating from bash to a language like rust. While neither is likely to happen (as per the discussion), it does highlight various problems known with Debian + bash.

georglauterbach · 2023-08-28T15:07:04Z

I will come back to this PR. @polarathene thank you for your elaborate review; I will make sure to go all of it as soon as I have time again. Probably October.

casperklein · 2024-01-21T22:39:15Z

I have finished the review. Looks overall good to me, just a few questions/comments.

target/scripts/build/packages.sh

@polarathene

- removed `source /etc/os-release` and used `VERSION_CODENAME` manually - adding PPAs is now done in a separate function - one invocation of `curl` was streamlined - manually applied suggestions from @polarathene (I could not find them on GitHub, I don't know why...) - brought back removal of `/etc/postsrsd.secret` Co-authored-by: Casper <casperklein@users.noreply.github.com> Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>

CHANGELOG.md

polarathene · 2024-01-23T23:45:07Z

I've resolved both of these for you directly.

Renamed the amavis process to test against
Disabled XOAUTH2 test-case (could bring back netcat, but OAUTHBEARER should be equivalent coverage on the functionality we're interested in)

@polarathene can you please have a look at these test failures
_check_if_process_is_running amavisd-new

Should be a fairly obvious one to fix considering the related changes in this PR. Amavis process name changed, it needs to be adjusted in the test accordingly. The test only fails now from the revision because the coverage / specificity improved to catch it 😅

Should just be changing the check from amavisd-new to amavisd.

'__should_login_successfully_with 'XOAUTH2'' failed

dovecot: imap-login: Login: user=<user1@localhost.localdomain>, method=OAUTHBEARER

This one is odd.

The logs found in the output to match have OAUTHBEARER instead of XOAUTH2. However these two methods are split into separate test cases and XOAUTH2 comes first.

Perhaps a bug with the updated curl? (EDIT: Confirmed, curl 7.74.0 => 7.88.1, bug was introduced in curl 7.80.0 release from Nov 2021, still present in the latest Dec 2023 8.5.0 release)

Should be ok for the most part, `OAUTHBEARER` is roughly equivalent in coverage.

polarathene

Looks like all feedback / concerns have now been addressed 👍

Let's finally merge this! 💪 🥳 (EDIT: After releasing v13.3.1, which I thought had already been done 😬 )

polarathene

Optional - Neither need to be addressed by this PR though.

polarathene · 2024-01-24T01:53:08Z

target/scripts/build/packages.sh

@@ -1,10 +1,13 @@
 #!/bin/bash

-# -eE :: exit on error (do this in functions as well)
-# -u  :: show (and exit) when using unset variables
+# -eE         :: exit on error (do this in functions as well)


Not sure if this was what @casperklein had in mind 🤷‍♂️

Suggested change

# -eE :: exit on error (do this in functions as well)

# -e :: exit on error (do this in functions as well)

# -E :: inherit the ERR trap to functions, command substitutions and sub-shells

target/scripts/build/packages.sh

georglauterbach · 2024-01-24T15:44:21Z

I have applied your latest review comments about the Rspamd installation and will merge this PR now when tests are passing :)

georglauterbach · 2024-01-24T16:06:24Z

Finally 🚀🚀🚀 Thank you guys for the nice feedback and review!

casperklein · 2024-01-29T12:15:24Z

I noticed, that the timestamp format in mail.log has changed.

Old:

Oct  1 03:10:21 mail dovecot: pop3-login: Login: user=<mail@domain.de>, method=PLAIN, rip=46.xx.xx.xx, lip=172.xx.xx.xx, mpid=6198, TLS, session=<yWpVUp0GWpMu78L+>                                                             │l
Oct  1 03:10:21 mail dovecot: pop3(mail@domain.de)<6198><yWpVUp0GWpMu78L+>: Disconnected: Logged out top=0/0, retr=0/0, del=0/944, size=105872624

New:

2024-01-29T12:40:52.411491+01:00 mail dovecot: pop3-login: Login: user=<mail@domain.de>, method=PLAIN, rip=37.xx.xx.xx, lip=172.xx.xx.xx, mpid=23610, TLS, session=<XGE0HhQQVRwlyPVQ>                                                     │i
2024-01-29T12:40:52.471401+01:00 mail dovecot: pop3(mail@domain.de)<23610><XGE0HhQQVRwlyPVQ>: Disconnected: Logged out top=0/0, retr=0/0, del=0/0, size=0

It's not only dovecot, postfix etc. uses the same new format. Any idea how to revert this back to the old format?

PS: This is a breaking change for someone parsing mail.log entrys.

casperklein · 2024-01-29T12:29:53Z

The change of the time format ist mentioned here. There are other worth mentioning changes, e.g. the files /var/log/mail.{info,warn,err} are no longer created.

Any idea how to revert this back to the old format?

Adding $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat at the top of /etc/rsyslog.conf brings back the old style.

georglauterbach · 2024-01-29T13:12:02Z

It seems like rsyslog finally adopted ISO8601 as a time stamp format. I actually really appreciate that (I always thought the rsyslog format was weird and most-notable non-standard)!

We should provide an entry in the changelog though, indeed.

polarathene · 2024-01-29T13:22:35Z

It seems like rsyslog finally adopted ISO8601 as a time stamp format.

That's probably the newer syslog RFC format not just the timestamp change, whereas before it was BSD syslog which was less formalized in spec.

casperklein · 2024-01-29T13:34:51Z

I always thought the rsyslog format was weird and most-notable non-standard

I really like the "weird" format, because it's very easily human readable, compared to the high precision format now used. Having lots of servers and a central logging, the new format seems to be the better choice. But for a single server I prefer it simple and easy accessible.
The fix is now in my user-patches.sh.

georglauterbach · 2024-01-29T13:54:21Z

I understand that :) Can you additionally provide a PR that updates the CHANGELOG @casperklein?

georglauterbach added priority/medium kind/update Update an existing feature, configuration file or the documentation labels Jun 20, 2023

georglauterbach added this to the v13.0.0 milestone Jun 20, 2023

georglauterbach requested review from casperklein and polarathene June 20, 2023 20:16

georglauterbach self-assigned this Jun 20, 2023

georglauterbach changed the title ~~Update base image~~ general: update base image to Debian 12 ("Bookworm") Jun 20, 2023

georglauterbach force-pushed the update-base-image branch from f597f21 to 69d3402 Compare June 20, 2023 20:25

polarathene reviewed Jun 20, 2023

View reviewed changes

docs/content/config/security/rspamd.md Outdated Show resolved Hide resolved

This comment was marked as outdated.

Sign in to view

georglauterbach added the stale-bot/ignore Indicates that this issue / PR shall not be closed by our stale-checking CI label Jun 21, 2023

georglauterbach marked this pull request as draft June 22, 2023 07:18

This comment was marked as outdated.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as outdated.

Sign in to view

georglauterbach force-pushed the update-base-image branch 3 times, most recently from 6d2cc16 to 8812a9e Compare August 25, 2023 10:22

This comment was marked as outdated.

Sign in to view

georglauterbach force-pushed the update-base-image branch from 8812a9e to f276853 Compare August 25, 2023 11:48

This comment was marked as outdated.

Sign in to view

polarathene requested changes Aug 26, 2023

View reviewed changes

casperklein reviewed Aug 26, 2023

View reviewed changes

target/scripts/build/packages.sh Show resolved Hide resolved

polarathene reviewed Aug 27, 2023

View reviewed changes

polarathene requested changes Jan 21, 2024

View reviewed changes

target/scripts/build/packages.sh Outdated Show resolved Hide resolved

target/scripts/build/packages.sh Outdated Show resolved Hide resolved

target/scripts/build/packages.sh Outdated Show resolved Hide resolved

target/scripts/build/packages.sh Outdated Show resolved Hide resolved

georglauterbach and others added 2 commits January 23, 2024 20:21

Merge branch 'master' into update-base-image

60dd425

georglauterbach requested review from polarathene and casperklein January 23, 2024 19:32

This comment was marked as resolved.

Sign in to view

casperklein reviewed Jan 23, 2024

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

polarathene added 3 commits January 24, 2024 14:38

tests: process_check_restart.bats - amavisd-new => amavisd

bd50b24

tests: mail_with_oauth2.bats - Disable XOAUTH2 test-case

032f030

Should be ok for the most part, `OAUTHBEARER` is roughly equivalent in coverage.

Merge branch 'master' into update-base-image

fe69852

polarathene previously approved these changes Jan 24, 2024

View reviewed changes

polarathene reviewed Jan 24, 2024

View reviewed changes

This comment was marked as outdated.

Sign in to view

georglauterbach added 2 commits January 24, 2024 16:40

Merge branch 'master' into update-base-image

c6f89a3

apply final feedback

4b8f36b

georglauterbach dismissed polarathene’s stale review via 4b8f36b January 24, 2024 15:40

georglauterbach merged commit 00018e7 into master Jan 24, 2024
7 checks passed

georglauterbach deleted the update-base-image branch January 24, 2024 16:05

casperklein mentioned this pull request Jan 29, 2024

chore: CHANGELOG.md - Add rsyslog breaking changes for v14 #3854

Merged

12 tasks

polarathene mentioned this pull request Aug 3, 2024

docs: OAuth2 - Add an example for /introspect endpoints with basic auth credentials #4081

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

general: update base image to Debian 12 ("Bookworm") #3403

general: update base image to Debian 12 ("Bookworm") #3403

georglauterbach commented Jun 20, 2023 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as off-topic.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

polarathene left a comment •

edited by georglauterbach

Loading

polarathene Aug 27, 2023 •

edited

Loading

georglauterbach Dec 3, 2023

polarathene commented Aug 27, 2023 •

edited

Loading

georglauterbach commented Aug 28, 2023

casperklein commented Jan 21, 2024

This comment was marked as resolved.

polarathene commented Jan 23, 2024 •

edited

Loading

polarathene left a comment •

edited

Loading

polarathene left a comment

polarathene Jan 24, 2024

This comment was marked as outdated.

georglauterbach commented Jan 24, 2024

georglauterbach commented Jan 24, 2024

casperklein commented Jan 29, 2024 •

edited

Loading

casperklein commented Jan 29, 2024 •

edited

Loading

georglauterbach commented Jan 29, 2024

polarathene commented Jan 29, 2024

casperklein commented Jan 29, 2024

georglauterbach commented Jan 29, 2024

		_log 'debug' 'Installing Fail2ban'
		apt-get "${QUIET}" --no-install-recommends install python3-pyinotify python3-dnspython

	# -eE :: exit on error (do this in functions as well)
	# -e :: exit on error (do this in functions as well)
	# -E :: inherit the ERR trap to functions, command substitutions and sub-shells

general: update base image to Debian 12 ("Bookworm") #3403

general: update base image to Debian 12 ("Bookworm") #3403

Conversation

georglauterbach commented Jun 20, 2023 • edited Loading

Description

Type of change

Checklist:

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as off-topic.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

polarathene left a comment • edited by georglauterbach Loading

Choose a reason for hiding this comment

polarathene Aug 27, 2023 • edited Loading

Choose a reason for hiding this comment

georglauterbach Dec 3, 2023

Choose a reason for hiding this comment

polarathene commented Aug 27, 2023 • edited Loading

georglauterbach commented Aug 28, 2023

casperklein commented Jan 21, 2024

This comment was marked as resolved.

polarathene commented Jan 23, 2024 • edited Loading

polarathene left a comment • edited Loading

Choose a reason for hiding this comment

polarathene left a comment

Choose a reason for hiding this comment

polarathene Jan 24, 2024

Choose a reason for hiding this comment

This comment was marked as outdated.

georglauterbach commented Jan 24, 2024

georglauterbach commented Jan 24, 2024

casperklein commented Jan 29, 2024 • edited Loading

casperklein commented Jan 29, 2024 • edited Loading

georglauterbach commented Jan 29, 2024

polarathene commented Jan 29, 2024

casperklein commented Jan 29, 2024

georglauterbach commented Jan 29, 2024

georglauterbach commented Jun 20, 2023 •

edited

Loading

polarathene left a comment •

edited by georglauterbach

Loading

polarathene Aug 27, 2023 •

edited

Loading

polarathene commented Aug 27, 2023 •

edited

Loading

polarathene commented Jan 23, 2024 •

edited

Loading

polarathene left a comment •

edited

Loading

casperklein commented Jan 29, 2024 •

edited

Loading

casperklein commented Jan 29, 2024 •

edited

Loading