fix: Ensure state persisted to /var/mail-state retains correct group

[polarathene]

Description

This PR covers changes mostly related to misc-stack.sh:

• Some minor refactoring of the conditional handling in the symlink loop (see commit for details).
• Ensures chown -R is more explicit about not only setting the user, but also the correct group. Postfix also needed a little adjustment there.
• Moves the related functionality for Amavis to be handled with the rest instead of a separate function call.
• Fixes misc-stack.sh rspamd ENV condition to add to FILES array when enabled, not disabled. This typo was missed during initial review.

Likewise we've had a similar issue with our Dockerfile copying over ClamAV database files from their official image. This prompted reviewing misc-stack.sh which should fix the issue for the user.

Fixes #2942

References

• I previously wrote a similar PR adjusting misc-stack.sh, particularly for Postfix. It may be useful reference for any future maintainer stumbling on these kind of changes.
• Contribution of the original Amavis fix was in Feb 2017. Another contributor describes the bug in the associated issue, which is the same bug we've recently had reported with ClamAV I think. Hence being more thorough with the chown -R usage in misc-stack.sh (and making it more clear about the purpose via comment).
• COPY --link was introduced into the Dockerfile in the v11.2 release in this Sep 2022 PR.
• The chown -R fixes for other users/groups misc-stack.sh appear to have originated in this May 2017 PR (a fix for this March 2017 issue), presumably based off the earlier Feb 2017 PR for Amavis.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Improvement (non-breaking change that does improve existing functionality)

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• New and existing unit tests pass locally with my changes

[polarathene]

Update: This turns out to be a bug / incompatibility with buildx when using the docker-container driver to build an image. (EDIT: Not a bug, without buildx, current Docker bundled BuildKit is too old and just silently ignores --link)

We can perform a chown -R afterwards. That doubles the image size though I think, but as it's the base stage, it shouldn't matter as it won't contribute towards the final stage that we export/publish. (EDIT: This was not required for the workaround, but also entirely avoidable, see new comment)

---

CI failure:

Error: buildx failed with: ERROR: failed to solve: invalid user index: -1

Full build log

/usr/bin/docker buildx build --cache-from type=local,src=/tmp/.buildx-cache --cache-to type=local,dest=/tmp/.buildx-cache-new,mode=max --iidfile /tmp/docker-build-push-r1dOWM/iidfile --output type=cacheonly --platform linux/amd64 --metadata-file /tmp/docker-build-push-r1dOWM/metadata-file .
#1 [internal] load .dockerignore
#1 transferring context: 59B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 12.09kB done
#2 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:1
#3 ...

#4 [auth] docker/dockerfile:pull token for registry-1.docker.io
#4 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:1
#3 DONE 0.5s

#5 docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc
#5 resolve docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s done
#5 sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1[124](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:128)d72dadce04db8badbe7b08546 9.94MB / 9.94MB 0.1s done
#5 extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546
#5 extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 0.2s done
#5 DONE 0.3s

#6 [auth] clamav/clamav:pull token for registry-1.docker.io
#6 DONE 0.0s

#7 [auth] library/debian:pull token for registry-1.docker.io
#7 DONE 0.0s

#8 [internal] load metadata for docker.io/library/debian:11-slim
#8 DONE 0.2s

#9 [internal] load metadata for docker.io/clamav/clamav:latest
#9 DONE 0.2s

#10 [internal] load build context
#10 DONE 0.0s

#11 [stage-base  1/49] FROM docker.io/library/debian:11-slim@sha256:98d3b4b0cee264301eb1354e0b549323af2d0633e1c43375d0b25c01826b6790
#11 resolve docker.io/library/debian:11-slim@sha256:98d3b4b0cee264301eb1354e0b549323af2d0633e1c43375d0b25c01826b6790 0.0s done
#11 DONE 0.0s

#12 importing cache manifest from local:9656101879626942798
#12 DONE 0.0s

#13 FROM docker.io/clamav/clamav:latest@sha256:314c46478306f1bbf3216e2a8ca4b3cb87ba5dd1e14fe4d43f0e3d13712a4af1
#13 resolve docker.io/clamav/clamav:latest@sha256:314c46478306f1bbf3216e2a8ca4b3cb87ba5dd1e14fe4d43f0e3d13712a4af1
#13 ...

#10 [internal] load build context
#10 transferring context: 295.46kB 0.0s done
#10 DONE 0.1s

#13 FROM docker.io/clamav/clamav:latest@sha256:314c46478306f1bbf3216e2a8ca4b3cb87ba5dd1e14fe4d43f0e3d13712a4af1
#13 resolve docker.io/clamav/clamav:latest@sha256:314c46478306f1bbf3216e2a8ca4b3cb87ba5dd1e14fe4d43f0e3d13712a4af1 0.0s done
#13 sha256:24cb207b181c6103680fa838d6c19b2baa1869aee6afcd891ce46b7b174f7b26 1.26kB / 1.26kB 0.0s done
#13 sha256:2d8c7c5ce39eaa53a3997135d3c16598f489f99655024514de17424d064b03e4 348B / 348B 0.0s done
#13 sha256:68331520d622a7054a5e17fa38f2f5e25bf0680d93adfee677445f671ec4e3ba 1.66MB / 1.66MB 0.1s done
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 7.34MB / 231.14MB 0.2s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 19.92MB / 231.14MB 0.3s
#13 sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 3.37MB / 3.37MB 0.1s done
#13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 11.53MB / 78.00MB 0.2s
#13 extracting sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715
#13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 30.41MB / 78.00MB 0.3s
#13 extracting sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 0.2s done
#13 extracting sha256:68331520d622a7054a5e17fa38f2f5e25bf0680d93adfee677445f671ec4e3ba
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 34.83MB / 231.14MB 0.5s
#13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 47.63MB / 78.00MB 0.5s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 57.67MB / 231.14MB 0.6s
#13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 59.77MB / 78.00MB 0.6s
#13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 77.59MB / 78.00MB 0.8s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 83.96MB / 231.14MB 0.9s
#13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 78.00MB / 78.00MB 0.9s done
#13 extracting sha256:68331520d622a7054a5e17fa38f2f5e25bf0680d93adfee677445f671ec4e3ba 0.5s done
#13 extracting sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 101.71MB / 231.14MB 1.1s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 114.29MB / 231.14MB 1.2s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 [131](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:135).76MB / 231.14MB 1.4s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 152.70MB / 231.14MB 1.5s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 170.92MB / 231.14MB 1.7s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 188.74MB / 231.14MB 1.8s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 214.96MB / 231.14MB 2.0s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 229.64MB / 231.14MB 2.1s
#13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 231.14MB / 231.14MB 3.0s done
#13 extracting sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 2.3s done
#13 extracting sha256:2d8c7c5ce39eaa53a3997[135](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:139)d3c16598f489f99655024514de17424d064b03e4 done
#13 extracting sha256:24cb207b181c6103680fa838d6c19b2baa1869aee6afcd891ce46b7b174f7b26 done
#13 DONE 3.3s

#13 FROM docker.io/clamav/clamav:latest@sha256:314c46478306f1bbf3216e2a8ca4b3cb87ba5dd1e14fe4d43f0e3d[137](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:141)12a4af1
#13 extracting sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d[183](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:187)2dd2e0f2f8cf9e33
#13 extracting sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 1.5s done
#13 DONE 4.8s

#14 [stage-base  3/49] RUN chmod +x /usr/local/bin/sedfile
#14 CACHED

#15 [stage-base  4/49] COPY target/scripts/build/* /build/
#15 CACHED

#16 [stage-base  5/49] COPY target/scripts/helpers/log.sh /usr/local/bin/helpers/log.sh
#16 CACHED

#17 [stage-base  6/49] RUN /bin/bash /build/packages.sh
#17 CACHED

#18 [stage-base  2/49] COPY target/bin/sedfile /usr/local/bin/sedfile
#18 CACHED

#19 [stage-base  7/49] RUN <<EOF (echo '0 */6 * * * clamav /usr/bin/freshclam --quiet' >/etc/cron.d/clamav-freshclam...)
#19 CACHED
ERROR: failed to solve: invalid user index: -1

---

Related workflow logs for bug reporting:

Run docker/build-push-action@v3.3.0

  with:
    context: .
    platforms: linux/amd6[4](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:4)
    cache-from: type=local,src=/tmp/.buildx-cache
    cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
    outputs: type=cacheonly
    load: false
    no-cache: false
    pull: false
    push: false
    github-token: ***

Docker info

/usr/bin/docker version
  Client:
   Version:           20.10.22+azure-1
   API version:       1.41
   Go version:        go1.18.9
   Git commit:        3a2c30b63ab20acfcc3f3550ea756a0561655a77
   Built:             Thu Dec 15 15:37:38 UTC 2022
   OS/Arch:           linux/amd64
   Context:           default
   Experimental:      true
  
  Server:
   Engine:
    Version:          20.10.22+azure-1
    API version:      1.41 (minimum version 1.12)
    Go version:       go1.18.9
    Git commit:       42c8b314993e5eb3cc2776da0bbe41d5eb4b707b
    Built:            Thu Dec 15 22:17:04 2022
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          1.5.16+azure-1
    GitCommit:        2e3140a0e09d[28](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:30)8a9086474752b4478aa0964e7c
   runc:
    Version:          1.1.4
    GitCommit:        5fd4c4d144137e991c4acebb2146ab1483a97925
   docker-init:
    Version:          0.19.0
    GitCommit:        
  /usr/bin/docker info
  Client:
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc., 0.9.1+azure-2)
    compose: Docker Compose (Docker Inc., 2.15.1+azure-1)
  
  Server:
   Containers: 1
    Running: 1
    Paused: 0
    Stopped: 0
   Images: 20
   Server Version: 20.10.22+azure-1
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 1
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: 2e3140a0e09d288a9086474752b4478aa0964e7c
   runc version: 5fd4c4d144137e991c4acebb2146ab1483a97925
   init version: 
   Security Options:
    apparmor
    seccomp
     Profile: default
   Kernel Version: 5.15.0-10[30](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:32)-azure
   Operating System: Ubuntu 20.04.5 LTS
   OSType: linux
   Architecture: x86_64
   CPUs: 2
   Total Memory: 6.781GiB
   Name: fv-az628-879
   ID: J5HJ:OPXN:RYBU:[55](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:57)NP:DUWN:INRV:KGP3:GG3Y:CDQE:4K4D:LC24:LEZB
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Registry: https://index.docker.io/v1/
   Labels:
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false

Buildx version

  /usr/bin/docker buildx version
  github.com/docker/buildx 0.9.1+azure-2 ed00243a0ce2a0aee75311b06e32d33b44729689
/usr/bin/docker buildx build --cache-from type=local,src=/tmp/.buildx-cache --cache-to type=local,dest=/tmp/.buildx-cache-new,mode=max --iidfile /tmp/docker-build-push-r1dOWM/iidfile --output type=cacheonly --platform linux/amd64 --metadata-file /tmp/docker-build-push-r1dOWM/metadata-file .

---

This seems to be a bug with the CI caching as I can run make build locally fine and it correctly applies the --chown change. (EDIT: This was misleading due to Dec 2022 Docker Engine using outdated BuildKit 0.8.4 release from almost 2 years ago)

[georglauterbach]

While it's never nice to use a workaround, good job at implementing it! LGTM 👍🏼

[polarathene]

Worked around the issue by treating it a separate stage to avoid the double file size chown -R would cause.

That might affect the image cache still however. I have noticed in the CI build workflow that we've gone from 345MB cache to 786MB, but I'm not entirely sure if that's a new min size 😅

The image build itself seems to remain at the ~600MB size at least (300MB is from the package install layer, 230MB is clamav and about 80MB from the debian base image).

---

The review feedback to align paths in misc-stack.sh for chown -R did trigger a new cache key. The previous build in this PR was a cache hit, then at the end the new derived cache key was uploaded. Both 786MB.. so this results in 443MB increase in image build cache weight.

Probably because the separate ClamAV stage is like 460MB, and then the usual 230MB layer in the later stage it's COPY --link into. Which is probably right if that extra 230MB (x2) is compressed a little to 443MB. I kind of expected the --link to reduce that weight, but I guess it's only effective in an images layers being pulled, not in the build cache stages 🤔

[polarathene]

Below sections is just reference / info, not too important for review :)

I looked into this a bit more and found we can avoid the extra 230-460MB increase to build cache by eagerly adding the clamav user before we install the clamav package.

I've matched how the package creates the user (source linked below), and with an explicit UID + GID, we can now use COPY --link with --chown (which can be used with --link if we only specify numeric ID values).

Now build cache remains at 345MB, keeping rate of eviction from CI cache storage low 👍

---

Users + Groups we add

While working on this PR, I looked into the users we explicitly create or are added from packages installed.

For the majority I've looked up the source for how they were created and documented that all below. Might be useful reference for maintenance someday? 😅

We could technically create those users upfront ourselves.

There isn't much benefit doing that except for when I experimented with splitting different package installs into separate stages and using a naive COPY --from=some-stage / / (as that would overwrite the common /etc/passwd, /etc/group, /etc/shadow files affected).

That used a fair amount of additional space as COPY layers (even with --link if I recall correctly), but minimized with another separate stage to bundle copies into, and a single COPY from that into the output image layers was the same with --link (but larger build cache size) and a few MB less without --link (still used extra build cache for the separate stages, which can build concurrently). Alternatively if each stage depended on the previous instead, it seems to not bulk up the build cache.

One benefit is a central place of where each user is setup, although there's a small chance of maintenance drift from upstream at risk. And should we change these due to add/removing packages, where the UID/GID values shift anyway, then it's not preventing the main problem 😅 (hence these chown -R startup scripts to patch up such changes)

Setup users script (with package source references):

#!/bin/bash

# -eE :: exit on error (do this in functions as well)
# -u  :: show (and exit) when using unset variables
# -o pipefail :: exit on error in pipes
set -eE -u -o pipefail

function _setup_users() {
  addgroup --quiet --system --force-badname ssl-cert
  addgroup --quiet --system crontab
  addgroup --quiet --system postdrop

  # `--disabled-password` and `--disabled-login` aren't necessary, results in roughly the same /etc/shadow output
  # `--disabled-password` allows for login that is not password-based. It appears to be default for `--system`, retained for clarity.
  local COMMON=(
    --quiet
    --system
    --disabled-password
  )

  # clamav + fetchmail: Should both prefer `nologin` to `/bin/false`
  # for their shell as it is effectively equivalent.
  # fetchmail installs with a home group of nobody, so we add that user without `--group`
  adduser "${COMMON[@]}" --home '/var/lib/fetchmail'    fetchmail

  COMMON+=('--group')
  adduser "${COMMON[@]}" --home '/run/opendkim'         opendkim
  adduser "${COMMON[@]}" --home '/run/opendmarc'        opendmarc
  adduser "${COMMON[@]}" --home '/var/lib/redis'        redis
  # Amavis needs a shell for Dockerfile `razor-admin` lines:
  adduser "${COMMON[@]}" \
    --home '/var/lib/amavis' \
    --gecos 'AMaViS system user' \
    --shell /bin/sh \
    amavis

  COMMON+=('--no-create-home')
  adduser "${COMMON[@]}" --home '/nonexistent'          policyd-spf
  adduser "${COMMON[@]}" --home '/var/spool/postfix'    postfix
  adduser "${COMMON[@]}" --home '/var/lib/postgrey'     postgrey
  adduser "${COMMON[@]}" --home '/var/lib/postsrsd'     postsrsd
  adduser "${COMMON[@]}" --home '/var/lib/spamassassin' debian-spamd

  adduser "${COMMON[@]}" \
    --home '/usr/lib/dovecot' \
    --gecos 'Dovecot mail server' \
    dovecot

  adduser "${COMMON[@]}" \
    --home '/nonexistent' \
    --gecos 'Dovecot login user' \
    dovenull

  adduser "${COMMON[@]}" \
    --home '/var/lib/rspamd' \
    --gecos 'rspamd spam filtering system' \
    --force-badname \
    _rspamd

  # Explicit UID set (--group will use it also for GID), for use with `COPY --chown`
  adduser "${COMMON[@]}" --home '/var/lib/clamav' --uid 200 clamav

  # Add clamav and amavis users to each others group
  adduser clamav amavis
  adduser amavis clamav

  # rsyslog package in Debian (unlike in Ubuntu) does not add the syslog user
  adduser "${COMMON[@]}" --home '/nonexistent' syslog

  # DMS user (used by Dovecot?):
  # NOTE: `adduser` isn't suitable for non-interactive creation of users with passwords.
  # NOTE: This requires openssl to already be installed. A group is implicitly created matching user name and uid values.
  useradd --uid 5000 --home-dir '/home/docker' --shell '/bin/bash' --password "$(echo docker | openssl passwd -1 -stdin)" docker
}

_setup_users

### Additional References ###

# Docs for adduser + useradd:
# https://manpages.debian.org/bullseye/adduser/adduser.8.en.html
# https://manpages.debian.org/bullseye/passwd/useradd.8.en.html

# Default additions to /etc/passwd without this script:
#
# postfix:x:101:102::/var/spool/postfix:/usr/sbin/nologin
# fetchmail:x:102:65534::/var/lib/fetchmail:/bin/false
# postsrsd:x:103:104::/var/lib/postsrsd:/usr/sbin/nologin
# opendkim:x:104:106::/run/opendkim:/usr/sbin/nologin
# opendmarc:x:105:107::/run/opendmarc:/usr/sbin/nologin
# clamav:x:106:108::/var/lib/clamav:/bin/false
# postgrey:x:107:109::/var/lib/postgrey:/usr/sbin/nologin
# policyd-spf:x:108:110::/nonexistent:/usr/sbin/nologin
# debian-spamd:x:109:111::/var/lib/spamassassin:/usr/sbin/nologin
# amavis:x:110:112:AMaViS system user,,,:/var/lib/amavis:/bin/sh
# dovecot:x:111:113:Dovecot mail server,,,:/usr/lib/dovecot:/usr/sbin/nologin
# dovenull:x:112:114:Dovecot login user,,,:/nonexistent:/usr/sbin/nologin
# _rspamd:x:113:115:rspamd spam filtering system,,,:/var/lib/rspamd:/usr/sbin/nologin
# redis:x:114:116::/var/lib/redis:/usr/sbin/nologin
# syslog:x:115:65534::/home/syslog:/usr/sbin/nologin
# docker:x:5000:5000::/home/docker:/bin/bash

# Default additions to /etc/group without this script:
#
# ssl-cert:x:101:
# postfix:x:102:
# postdrop:x:103:
# postsrsd:x:104:
# crontab:x:105:
# opendkim:x:106:
# opendmarc:x:107:
# clamav:x:108:amavis
# postgrey:x:109:
# policyd-spf:x:110:
# debian-spamd:x:111:
# amavis:x:112:clamav
# dovecot:x:113:
# dovenull:x:114:
# _rspamd:x:115:
# redis:x:116:
# docker:x:5000:

# Default additions to /etc/shadow without this script:
# '*' is no password, cannot login as these system users.
# --disabled-login causes the locked '!' value for clamav,spamassassin,rspamd.
#
# postfix:*:19377:0:99999:7:::
# fetchmail:*:19377:0:99999:7:::
# postsrsd:*:19377:0:99999:7:::
# opendkim:*:19377:0:99999:7:::
# opendmarc:*:19377:0:99999:7:::
# clamav:!:19377:0:99999:7:::
# postgrey:*:19377:0:99999:7:::
# policyd-spf:*:19377:0:99999:7:::
# debian-spamd:!:19377:0:99999:7:::
# amavis:*:19377:0:99999:7:::
# dovecot:*:19377:0:99999:7:::
# dovenull:*:19377:0:99999:7:::
# _rspamd:!:19377:0:99999:7:::
# redis:*:19377:0:99999:7:::
# syslog:*:19377:0:99999:7:::
# docker:$1$/ZRgRKw5$kXOPJlnHGZLVzNV8Tfcav/:19377:0:99999:7:::


### Package sources for users and groups created above ###

# amavis (adduser --system --ingroup amavis --home /var/lib/amavis --gecos "AMaViS system user" --shell /bin/sh --quiet --disabled-password amavis)
# https://packages.debian.org/source/bullseye/amavisd-new
# https://salsa.debian.org/debian/amavisd-new/-/blob/dda3c9ef0ec85ac8bf4a3f26caa82e96fbb6d381/debian/amavisd-new.postinst#L53-55

# clamav (adduser --system --no-create-home --quiet --disabled-password --disabled-login --shell /bin/false --group --home /var/lib/clamav clamav)
# https://packages.debian.org/source/bullseye/clamav
# https://salsa.debian.org/clamav-team/clamav/-/blob/4a25d28c0379c4b034d60c686f63f3d518d85286/debian/clamav-base.postinst.in#L35-38

# clamav installs cron package, adds crontab group
# addgroup --system crontab
# https://packages.debian.org/source/bullseye/cron
# https://salsa.debian.org/debian/cron/-/blob/bf78c756347c4533b1eba1ee24aadf6952b03531/debian/cron-daemon-common.postinst#L13

# user: postfix, groups: postfix postdrop
# addgroup --system postdrop
# addgroup --system postfix
# adduser --system --home /var/spool/postfix --no-create-home --disabled-password --ingroup postfix postfix
# https://packages.debian.org/source/bullseye/postfix
# https://salsa.debian.org/postfix-team/postfix-dev/-/blob/ddcd3d9659338f44de47c2e8ece866ea0ade85c5/debian/postfix.postinst#L210-213
# https://salsa.debian.org/postfix-team/postfix-dev/-/blob/ddcd3d9659338f44de47c2e8ece866ea0ade85c5/debian/postfix.postinst#L21-22

# ssl-cert package installed as part of postfix package
# addgroup --quiet --system --force-badname ssl-cert
# https://packages.debian.org/source/bullseye/ssl-cert
# https://salsa.debian.org/apache-team/ssl-cert/-/blob/6d7133f393d044297418a34ca0738276005e55de/debian/postinst#L9

# opendkim (adduser --quiet --system --group --home /run/opendkim opendkim)
# https://packages.debian.org/source/bullseye/opendkim
# https://salsa.debian.org/debian/opendkim/-/blob/3a900f7373dce378c425ca832dea7bfa3e230d3d/debian/opendkim.postinst#L24

# opendmarc (adduser --quiet --system --group --home /run/opendmarc opendmarc)
# https://packages.debian.org/source/bullseye/opendmarc
# https://salsa.debian.org/kitterman/opendmarc/-/blob/b2ba99c5918d0b50048239090f61d985fe970e30/debian/opendmarc.postinst#L28

# postsrsd (adduser --quiet --system --group --no-create-home --home /var/lib/postsrsd postsrsd)
# https://packages.debian.org/source/bullseye/postsrsd
# https://salsa.debian.org/debian/postsrsd/-/blob/9d0c2fbc1b5df858d0fd27763eb4cc2f6f0d9352/debian/postsrsd.postinst#L34

# postgrey
# Git source presently in transition to salsa.debian.org, lacking debian post-install script for postgrey user?
# Installation output does indicate `--no-create-home` used, but prior `/var/lib/postgrey` is created or `chown postgrey:postgrey`.
# https://packages.debian.org/source/bullseye/postgrey
# https://salsa.debian.org/debian/postgrey

# fetchmail (Missing Debian package source)
# https://packages.debian.org/source/bullseye/fetchmail

# policyd-spf (adduser --quiet --system --group --no-create-home --home /nonexistent policyd-spf)
# https://packages.debian.org/source/bullseye/postfix-policyd-spf-perl
# https://salsa.debian.org/kitterman/postfix-policyd-spf-perl/-/blob/38b5b4d4b83f564651261f830e895ee2769ce62e/debian/postfix-policyd-spf-perl.postinst#L7-10

# debian-spamd (adduser --system --group --disabled-login --disabled-password --home /var/lib/spamassassin --no-create-home debian-spamd)
# https://packages.debian.org/source/bullseye/spamassassin
# https://salsa.debian.org/debian/spamassassin/-/blob/ead27ee27f508f0b91f408e688b6446b15cd3018/debian/spamassassin.postinst#L8-10

# dovecot (adduser --system --group --home /usr/lib/dovecot --gecos "Dovecot mail server" --no-create-home --disabled-password --quiet dovecot || true)
# dovenull (adduser --system --group --home /nonexistent --no-create-home --gecos "Dovecot login user" --disabled-password --quiet dovenull || true)
# https://packages.debian.org/source/bullseye/dovecot
# https://salsa.debian.org/debian/dovecot/-/blob/5092256610ddd679e366402abbe7854058f8a78b/debian/dovecot-core.postinst#L35-39

# _rspamd (adduser --quiet --system --group --home /var/lib/rspamd --no-create-home --disabled-login --gecos "rspamd spam filtering system" --force-badname _rspamd)
# https://packages.debian.org/source/bullseye/rspamd
# https://salsa.debian.org/debian/rspamd/-/blob/895f9465cc3559f6f50ba7093d61b57173192f46/debian/postinst#L37-45

# redis (adduser --system --home /var/lib/redis --quiet --group redis || true)
# https://packages.debian.org/source/bullseye/redis
# https://salsa.debian.org/lamby/pkg-redis/-/blob/9f2df070e3a23e5eae432f8e44d3925f1d291830/debian/redis-tools.postinst#L23-28

---

Package install layer weight

I was curious and observed the following breakdown (docker image history layer sizes):

Image layer weight breakdown:

build/packages.sh layer is 300 MB (up to 326 MB when split into separate stages) and approx 95 MB compressed (base image is 80 MB, approx 30MB compressed).

• _pre_installation_steps
• _install_postfix (42 MB)
• _install_packages (197 MB, ~210 MB if each sub-group is it's own stage)
  • ANTI_VIRUS_SPAM_PACKAGES (110 MB)
  • CODECS_PACKAGES (12 MB)
  • MISCELLANEOUS_PACKAGES (73 MB)
  • POSTFIX_PACKAGES (6 MB)
  • MAIL_PROGRAMS_PACKAGES (10 MB)
• _install_dovecot (32 MB)
• _install_rspamd (40 MB)
• _install_fail2ban (2 MB)
• _post_installation_steps

And then we add the 230MB ClamAV files to /var/lib/clamav.

---

COPY --link --chown=clamav

Improved workaround by using numeric user ID

I realized that the COPY --link in the separate stage doesn't seem relevant and could instead be COPY --chown=clamav if it can source the clamav user installed. Then the RUN chown -R isn't required, saving 230MB from build cache size.

While COPY --link is not compatible with --chown=clamav together, we can provide numeric IDs which is compatible. However we'd need to know those values in advance and they should not be prone to change as packages are added / removed in future.

If we create the clamav user in advance prior to installing the package, this is no longer a concern and the package install will use that already setup user. Thus we create that user with an explicit UID + GID and we can drop the separate workaround stage, removing another 230MB from build cache.

This keeps the CI build cache that we store back to the same ~345MB size 👍

[polarathene]

Err ok that's a first for me. I thought a rebase on master button kept the approval like it does when you update with a merge commit from master 🤔

Won't be doing that again then, sorry about that 😅

[polarathene]

@georglauterbach are you alright with this PR being merged? Or do you want to wait until your rspamd PR is merged first to avoid dealing with the likely merge conflict on misc-stack.sh?

[georglauterbach]

Go ahead 🚀