check-for-changes.sh uses a lot of resources

[ontheair81]

Subject

One day after upgrading to 10.4.0 I realized the process check-for-changes.sh consumes a lot of memory and cpu resources.
I did not see this behaviour with 10.3.0.

Description

The next day after the upgrade the container used 2.5G of memory (this usually was less than 800M) and the four cpu cores have been very busy. After restarting docker-mailserver everything was as usual again.

Some hours later I realized that memory and cpu usage was growing again. It seems to be caused by the process bin/bash /usr/local/bin/check-for-changes.sh.
After starting docker-mailserver, the memory consumption of this process is around 6M, but growing 800k per minute (ca. 50M per hour). The higher the memory usage goes, the higher the cpu load.

This makes it necessary for me to restart docker-mailserver periodically (let's say: every day) as I have 4G of RAM available.

Question

It would be nice to hear if there are other people experiencing the same issue, or if it is just me ;-) Help is very appreciated!
I am not an expert in docker (yet), but I am very willing to learn as much as possible and to provide more information as needed and requested.

Thank you very much in advance!
(I did not label this issue as "bug" as I am not sure if this is a general issue or just happening in my configuration.)

Environment

Debian Bullseye, latest patches
Let'sEncrypt-certificates are generated from nginx-reverse-proxy with acme-companion.

version: "3.7"

services:

    mailserver:
        image: "docker.io/mailserver/docker-mailserver:10.4.0"
        container_name: "mailserver"
        hostname: "mail"
        domainname: "[removed].de"
        env_file: "/home/user/docker_compose/mailserver/mailserver.env"
        ports:
           - "25:25"     # SMTP  (explicit TLS => STARTTLS)
           - "143:143"   # IMAP4 (explicit TLS => STARTTLS)
           - "465:465"   # ESMTP (implicit TLS)
           - "587:587"   # ESMTP (explicit TLS => STARTTLS)
           - "993:993"   # IMAP4 (implicit TLS)
           - "4190:4190" # ManageSieve
        volumes:
           - "data:/var/mail/"
           - "state:/var/mail-state/"
           - "logs:/var/log/mail/"
           - "config:/tmp/docker-mailserver/"
           - "/etc/localtime:/etc/localtime:ro"
           - "/home/user/docker_compose/reverse-proxy/certs:/etc/letsencrypt/live/"
        restart: "always"
        cap_add:
          - "NET_ADMIN"
          - "SYS_PTRACE"
        dns:
          - "9.9.9.9"
          - "8.8.8.8"
          - "1.1.1.1"

volumes:
        data:
        state:
        logs:
        config:

And some (maybe useful and related) informations from .env. Please ask if there is more information needed!

ENABLE_CLAMAV=0
ENABLE_AMAVIS=1
SSL_TYPE=letsencrypt
ENABLE_QUOTAS=1

[georglauterbach]

I've flagged this as bug, because if the script indeed does not behave well, it is a bug :)

ping @docker-mailserver/maintainers

I will have a look at this tomorrow too.

[eleith]

mine is not growing as quickly as OPs, but it is certainly growing.

[polarathene]

The only diff for changes on that file between the two releases was related to fixing an issue with SSL_TYPE=letsencrypt for non-Traefik users, so that cert renewal would actually be reacted to (previously the certs were not being checked).

I guess the looping logic is causing a memory leak since it never ends and bash doesn't know any better?

If you're ok with check-for-changes.sh being slower to respond, increasing the sleep time between loop iterations would likely slow the memory leak issue by a similar factor.

---

I would like to use file watchers to trigger scripts instead of a polling loop that we have currently, but some users apparently require polling. I have plans to support both, time permitting that should hopefully happen in January.

For now 10.4.0 doesn't offer much differences that should matter, other than ensuring your cert renewal is detected to restart Postfix and Dovecot.

If you're not using Traefik for provisioning certs (I know ontheair81 isn't), and your certificate doesn't need renewal within the next 30 days, use 10.3.0 for now, or restart your docker-mailserver container after cert renewal occurs.

[polarathene]

With nothing changing to run the other logic, this is the repeated loop iteration over and over:

docker-mailserver/target/scripts/check-for-changes.sh

Line 18 in 014ddda

cd /tmp/docker-mailserver || exit 1

docker-mailserver/target/scripts/check-for-changes.sh

Lines 48 to 58 in 014ddda

	while true
	do
	# get chksum and check it, no need to lock config yet
	_monitored_files_checksums >"${CHKSUM_FILE}.new"
	cmp --silent -- "${CHKSUM_FILE}" "${CHKSUM_FILE}.new"
	# cmp return codes
	# 0 – files are identical
	# 1 – files differ
	# 2 – inaccessible or missing argument
	if [[ ${?} -eq 1 ]]
	then

docker-mailserver/target/scripts/check-for-changes.sh

Lines 135 to 141 in 014ddda

	fi
	# mark changes as applied
	mv "${CHKSUM_FILE}.new" "${CHKSUM_FILE}"
	sleep 1
	done

Could you try with a local bind mount to a host directory instead of a Docker data volume? Curious if the constant writing of a new file that we rename with mv is causing the issue.

After starting docker-mailserver, the memory consumption of this process is around 6M, but growing 800k per minute (ca. 50M per hour). The higher the memory usage goes, the higher the cpu load.

If this isn't filesystem cache memory (which is common to use available memory, and all these written/read files could eat into that but the memory itself should be available to any process that actually needs it), and not some CoW like issue with the file management/writes, then maybe the file itself is growing excessively large? It shouldn't AFAIK, hard to say.

[eleith]

after a restart, i can confirm it's not growing anymore. so i can no longer reproduce. fyi.

[polarathene]

Then it might have been something related to the file locking logic, that's also going to be addressed and should improve things.

[ontheair81]

Thank you very much for all these replies!

The only diff for changes on that file between the two releases was related to fixing an issue with SSL_TYPE=letsencrypt for non-Traefik users [...] I guess the looping logic is causing a memory leak since it never ends and bash doesn't know any better?

And the other possible change I could think about, could be a new version of bash, as the base of this image was updated to Debian 11?

If this isn't filesystem cache memory (which is common to use available memory, and all these written/read files could eat into that but the memory itself should be available to any process that actually needs it), [...]

Sure. This process is really using memory, it is not the filesystem cache memory.

[...] then maybe the file itself is growing excessively large?

If I am correct, this means the file /tmp/docker-mailserver-config-chksum. After a restart it has the size of 9253B (the cert-path contains some more certificates). The size of this file did not change/grow in the last 30 minutes.

Could you try with a local bind mount to a host directory instead of a Docker data volume? Curious if the constant writing of a new file that we rename with mv is causing the issue.

If I am not wrong, the file /tmp/docker-mailserver-config-chksum is not in a Docker data volume, and not on a host directory, but in the container itself:
/tmp/docker-mailserver/ is a Docker data volume, but docker-mailserver-config-chksum resides in the parent /tmp folder, not in /tmp/docker-mailserver/.

after a restart, i can confirm it's not growing anymore. so i can no longer reproduce. fyi.

That's strange. In my case, after a restart it is growing again ;-)

Again, thank you very much for all these informations and thoughts!

[polarathene]

If I am not wrong, the file /tmp/docker-mailserver-config-chksum is not in a Docker data volume, and not on a host directory, but in the container itself:
/tmp/docker-mailserver/ is a Docker data volume, but docker-mailserver-config-chksum resides in the parent /tmp folder, not in /tmp/docker-mailserver/.

Hmmm... perhaps temporarily try bind mounting a local volume for the entire /tmp directory and see what happens? I don't think you can reliably bind mount the file since it is being renamed and a new file created every second, and file mounts are tied to the filesystem inode AFAIK, thus it would not help, but a directory mount should work.

• What OS are you using?
• Docker and docker-compose versions?
• Do you have many certificates provisioned within that certs directory?
• Does it make any difference for you if you remove domainname and use only hostname like so: hostname: "mail.[removed].de"?

After a restart it has the size of 9253B (the cert-path contains some more certificates).

Are you saying it has always contained more certificates, or there is more after restarting?

And the other possible change I could think about, could be a new version of bash, as the base of this image was updated to Debian 11?

It could be due to the image upgrade.. if you're comfortable building a local image to verify, you can do so this way:

git clone https://github.com/docker-mailserver/docker-mailserver.git --recurse-submodules

cd docker-mailserver

# Change the Docker image back to Buster (Debian 10)..
# 1st line of `Dockerfile`:
# `FROM docker.io/debian:11-slim` to `FROM docker.io/debian:10-slim` should work.

# Build the image
make build

# You now have a new local docker-mailserver image called `mailserver-testing:ci`, as v10.4.0 but with Debian 10 base image instead

---

A related change between 10.3.0 and 10.4.0 was also this function (diff shows it is only for the letsencrypt paths):

docker-mailserver/target/scripts/helper-functions.sh

Lines 221 to 255 in 014ddda

	# ? --------------------------------------------- File Checksums
	# file storing the checksums of the monitored files.
	# shellcheck disable=SC2034
	CHKSUM_FILE=/tmp/docker-mailserver-config-chksum
	# Compute checksums of monitored files,
	# returned output is lines of hashed content + filepath pairs.
	function _monitored_files_checksums
	{
	# If a wildcard path pattern (or an empty ENV) would yield an invalid path
	# or no results, `shopt -s nullglob` prevents it from being added.
	shopt -s nullglob
	# React to any cert changes within the following letsencrypt locations:
	local CERT_FILES=(
	/etc/letsencrypt/live/"${SSL_DOMAIN}"/*.pem
	/etc/letsencrypt/live/"${HOSTNAME}"/*.pem
	/etc/letsencrypt/live/"${DOMAINNAME}"/*.pem
	)
	# CERT_FILES should expand to separate paths, not a single string;
	# otherwise fails to generate checksums for these file paths.
	#shellcheck disable=SC2068
	(
	cd /tmp/docker-mailserver || exit 1
	exec sha512sum 2>/dev/null -- \
	postfix-accounts.cf \
	postfix-virtual.cf \
	postfix-aliases.cf \
	dovecot-quotas.cf \
	/etc/letsencrypt/acme.json \
	${CERT_FILES[@]}
	)
	}

As the checksum file is not changing in size, it seems unlikely that this would introduce a memory leak, the logic should all be the same apart from now checking the letsencrypt folder properly for cert files to create checksums of each .pem file (which the previous release was failing to do).

[ontheair81]

Hmmm... perhaps temporarily try bind mounting a local volume for the entire /tmp directory and see what happens?

Ok, I copied the whole /tmp directory to a local volume on the host and startet up with using this local directory.

#          - "config:/tmp/docker-mailserver/"
           - "/home/user/TEST/tmp/:/tmp/" 

docker-mailserver starts up fine and is working as before. The new local volume is now used (double-checked by the timestamp of docker-mailserver-config-chksum, which is updated every minute or so).
Sadly, there is no change; the process is growing big again.

Does it make any difference for you if you remove domainname and use only hostname like so: hostname: "mail.[removed].de"?

There is no difference (means everything works like before and the memory consumption is constantly growing).

What OS are you using?

Debian 11.2 (amd64)

Docker and docker-compose versions?

Just to clarify: I am using docker compose instead of docker-compose.

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:48 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:56 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker Compose version v2.1.1

Do you have many certificates provisioned within that certs directory?

There are 33 certificates in this directory, as this machine additionally provides some simple static websites on a bunch of (sub)domains.

Are you saying it has always contained more certificates, or there is more after restarting?

No, the (re)start of docker-mailserver does not change anything in the certs-path. I just wanted to mention, that there are more certificates in this directory, used for other things and domains too, not just the certificate for docker-mailserver.
So, nothing weird about that, just imprecise wording by me ;-)

if you're comfortable building a local image to verify, you can do so this way

As I am still a bit new to Docker and never built a local image myself, I am - honestly speaking - not so comfortable with doing it the first time - but I tried ;-)

make build just started fine, but after a couple of seconds failed with this message:

Processing triggers for libc-bin (2.28-10) ...
E: Unable to locate package dovecot-fts-xapian
The command '/bin/bash -o pipefail -c apt-get -qq update &&   apt-get -qq install apt-utils 2>/dev/null &&   apt-get -qq dist-upgrade &&   apt-get -qq install postfix &&   apt-get -qq --no-install-recommends install   altermime amavisd-new apt-transport-https arj binutils bzip2 bsd-mailx   ca-certificates cabextract clamav clamav-daemon cpio curl   dbconfig-no-thanks dovecot-core dovecot-fts-xapian dovecot-imapd   dovecot-ldap dovecot-lmtpd dovecot-managesieved dovecot-pop3d   dovecot-sieve dovecot-solr dumb-init   ed fetchmail file gamin gnupg gzip iproute2 iptables   locales logwatch lhasa libdate-manip-perl libldap-common liblz4-tool   libmail-spf-perl libnet-dns-perl libsasl2-modules lrzip lzop   netcat-openbsd nomarch opendkim opendkim-tools opendmarc   pax pflogsumm postgrey p7zip-full postfix-ldap postfix-pcre   postfix-policyd-spf-python postsrsd pyzor   razor rpm2cpio rsyslog sasl2-bin spamassassin supervisor   unrar-free unzip uuid whois xz-utils &&   gpg --keyserver ${FAIL2BAN_GPG_PUBLIC_KEY_SERVER}     --recv-keys ${FAIL2BAN_GPG_PUBLIC_KEY_ID} 2>&1 &&   curl -Lkso fail2ban.deb ${FAIL2BAN_DEB_URL} &&   curl -Lkso fail2ban.deb.asc ${FAIL2BAN_DEB_ASC_URL} &&   FINGERPRINT="$(LANG=C gpg --verify   fail2ban.deb.asc fail2ban.deb 2>&1     | sed -n 's#Primary key fingerprint: \(.*\)#\1#p')" &&   if [[ -z ${FINGERPRINT} ]]; then     echo "ERROR: Invalid GPG signature!" >&2; exit 1; fi &&   if [[ ${FINGERPRINT} != "${FAIL2BAN_GPG_FINGERPRINT}" ]]; then     echo "ERROR: Wrong GPG fingerprint!" >&2; exit 1; fi &&   dpkg -i fail2ban.deb 2>&1 &&   rm fail2ban.deb fail2ban.deb.asc &&   apt-get -qq autoremove &&   apt-get -qq autoclean &&   apt-get -qq clean &&   rm -rf /var/lib/apt/lists/* &&   c_rehash 2>&1' returned a non-zero code: 100
make: *** [Makefile:14: build] Error 100

For sure I missed something as I never built an image before.

[ontheair81]

I tried to dig a bit deeper and maybe found some related thing. When DMS is starting up, the output looks like this:

[ TASKLOG ]  Welcome to docker-mailserver 10.4.0
[ TASKLOG ]  Initializing setup
[ TASKLOG ]  Checking configuration
[ TASKLOG ]  Configuring mail server
[ WARNING ]  Clamav is disabled. You can enable it with 'ENABLE_CLAMAV=1'
[ TASKLOG ]  Applying user patches
[ TASKLOG ]  Post-configuration checks
[ TASKLOG ]  Starting daemons & mail server
cron: started
rsyslog: started
dovecot: started
update-check: started
opendkim: started
opendmarc: started
postfix: started
fail2ban: started
changedetector: started
amavis: started
[ TASKLOG ]  mail.[removed].de is up and running

Some lines of this output are obviously generated by start-mailserver.sh with the help of the function _notify from helper-functions.sh.

It looks like check-for-changes.sh should create output too, with the same _notify function, for example:

docker-mailserver/target/scripts/check-for-changes.sh

Line 14 in 014ddda

_notify 'task' "${LOG_DATE} Start check-for-changes script."

Or:

docker-mailserver/target/scripts/check-for-changes.sh

Line 46 in 014ddda

_notify 'inf' "$(_log_date) check-for-changes is ready"

But none of these outputs are visible during startup of DMS (see above). Maybe this shows that check-for-changes.sh is not starting up well at all?

I don't know if this could be an interesting find, or if it is completely irrelevant and output from _notify is working in a different way than I understand it.

[casperklein]

To see inf/task notifications, DMS_DEBUG must be set to 1.

[georglauterbach]

For sure I missed something as I never built an image before.

Not necessarily, as Docker builds on different OS can be compromised by different problems (and the build for this image has some quirks / difficulties - so no worries). The error you are experiencing however, does not make a lot of sense to me. The package dovecot-fts-xapian should have been found. On Ubuntu 21.10, the build works just fine. Can you try running make all?

As mentioned, to see a more detailed log output, set the environment variable DMS_DEBUG to 1.

I'm not sure whether docker-compose vs docker compose makes a difference here as both are "wrappers" around the Docker engine that is still running the container.

---

I had a look at my setup but could not make out such a problem (running on 1.4GiB in main memory and low idle CPU usage as reported by the Kubernetes pod management). I'm using

ENABLE_CLAMAV=1
ENABLE_AMAVIS=1
SSL_TYPE=manual
ENABLE_QUOTAS=0

[ontheair81]

I tried building it again, now with make all instead of make build:

With Debian 11 the build process looks fine, reaching the test process (which I cancelled).
With Debian 10 the process quits after a short while with E: Unable to locate package dovecot-fts-xapian

Wondering if this problem just occurs with SSL type Let'sEncrypt. I will try to use SSL_TYPE=manual instead, and see if the RAM consumption is different. Will report back.

[casperklein]

You can manually remove dovecot-fts-xapian from Dockerfile when building on debian 10. That package was introduced in debian 11 and is not available in earlier versions.

[ontheair81]

Thank you! The built of DMS 10.4.0 with Debian 10 was successful and is running right now.

The excessive growth of the process check-for-changes.sh is gone; the process occupies 4120B and is not growing a single byte anymore (exactly the same size and expected behaviour like in 10.3.0).

So it looks like this bug was introduced with the switch to Debian 11. I don't have any idea about the reason, sadly.

For now I will keep running this local built image, if there is no reason not to do so (if there is, please tell me).
If I can provide more informations to help finding the problem, just ask. Thank you all for your help!

[polarathene]

Thanks for the detailed responses and sharing your findings, much appreciated! ❤️

---

Wondering if this problem just occurs with SSL type Let'sEncrypt. I will try to use SSL_TYPE=manual instead, and see if the RAM consumption is different. Will report back.

Did you give that a try?

So it looks like this bug was introduced with the switch to Debian 11.

Probably difficult to attempt debugging that, but it's good to know it's related to the base image change. There was another recent issue where the problem was due to running the host OS Debian 11, but no issue on Ubuntu; It may be related here too:

Seems that out of the box, a Debian 11 x64 system running Docker 20.10.10 is not compatible.

[georglauterbach]

I just tested the script with Valgrind, and there are no leaks on the heap. So this does not seem to be an issue with Bash or the script...

Maybe this is indeed a problem with the specific version of Docker. As @polarathene mentioned, did you give SSL_TYPE=manual a try and see whether this makes a difference (I doubt it), and could you upgrade to the latest version of Docker (20.10.12 for me) with the latest Compose plugin please and see whether this changes things?

[ontheair81]

Maybe this is indeed a problem with the specific version of Docker. As @polarathene mentioned, did you give SSL_TYPE=manual a try and see whether this makes a difference (I doubt it) [...]

I tried that yesterday quickly, but ran into errors. For sure I just made a mistake when specifying the path to the certificates, as they have not been found at startup of DMS. Unfortunately I had no time to figure this out and correct it, as I had to bring up the mailserver again to reduce downtime. Sorry that I can not provide information about this topic now.

could you upgrade to the latest version of Docker (20.10.12 for me) [...]

Same for me, and I am running this version already.

[...] with the latest Compose plugin

I updated compose from 2.1.1 to the most recent 2.2.2 now. Tested it again with the original DMS 10.4.0 and had the same problems with check-for-changes.sh.

As this problem is hard to figure out, maybe it would be an idea to temporarily provide two different flavours of DMS for the next releases, one based on Debian 11, the other one based on Debian 10?

[georglauterbach]

We're not yet 100% sure whether it is actually Debian 11 or a package with Debian 11 that causes this, or something else entirely. The fact that all works well again with Debian 10 seems to be "obvious" as v10.4 is mainly about Debian 11, albeit not entirely. We will need to investigate further. I have made sure I'm on :edge and I will monitor resource usage over the next weeks too.

[wernerfred]

@docker-mailserver/maintainers I would like to build the image for DMS v10.4.0 again by re-triggering the pipeline. It may be the case, some mysterious issues with the base image at the time of building DMS v10.4.0 caused this issues. What's your take?

Did someone achive this local first with success? Would avoid it if it's effect is unclear

[georglauterbach]

@ontheair81 have you tried whether this issue persists with the current :edge build? This is basically (not really though) the same as building v10.4.0 from source. If :edge has the same problem, we could be more certain whether the particular v10.4.0 image has a problem in this regard. Could everyone try :edge to tell us whether this issue persists please?

[ontheair81]

Just tested edge for ten minutes right now. Sadly have to say that the process is still growing.

[georglauterbach]

Alright. Then rebuilding is not going to change anything. One more thing we can rule out. For the future, we may want to try out some fixes. We will let you know when we need some assistance. Everyone keep us posted out their findings.

[oblitum]

The higher the memory usage goes, the higher the cpu load.

Can confirm identical behavior. This morning RAM was at 5%, CPU usage at 1%~2%, now late at night, RAM is at 10%, CPU usage at 2%~5%.

[casperklein]

We switched from Debian 10 to 11 with the 10.4.0 release.

To rule out a regression from that, did someone already try to build the 10.4.0 release for Debian 10? I think undoing this commit should be sufficient: 8a47e7d#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557

[ontheair81]

Yes, I am running a local built version of 10.4.0 based on Debian 10 since December 29. It is working without any problems.

[georglauterbach]

#2401 was merged which refactored a part of the changedetector functionality that could behave weird. Can someone please try :edge to see whether something changed?

[ontheair81]

I guess it is time to open a bottle of champagne! Trying :edge now and ressources of this process are not growing anymore.

Maybe other people affected by this issue can give it a try and report back, too?

[NorseGaud]

If this ends up being the solution, this was a huge blocker for #2157 and may allow me to easily finalize it :)

[georglauterbach]

Very nice! It would be good to hear from all the others whether their issue is resolved as well (trying :edge)!

[oblitum]

I've been on :edge for 9 hours with (from what I can tell until now) zero leaks.

[zaphoodb]

Running more than 6hrs on edge. So far looks good! No increase in memory consumption.
(441 is the process if of check-for-changes.sh)

root@mail:/# pmap 441 | grep total
 total             4576K

In addition, the script used 13 seconds of CPU time during this time.

[georglauterbach]

Looks very promising. For those who want to reduce the CPU load further (like me), you can use

sed -i -E 's|  sleep.*|  sleep X|' /usr/local/bin/check-for-changes.sh

to adjust the sleep time to X seconds.

[polarathene]

Awesome that it appears to be fixed from the responses 🎉

Bit surprised it was an issue/bug from using a sub-shell though? 🤯

If this ends up being the solution, this was a huge blocker for #2157 and may allow me to easily finalize it :)

I have been lacking time, but I really think we should approach that as my comment on the PR describes. Hopefully I can find time to contribute to the project again at some point 🤞

I'm pretty sure that the old file locking approach without sleep polling can work with NFS, or we can fallback to polling for that specifically while the majority of users can use more efficient change detection. watchexec would be good for the change detection and flock for file locking IIRC.

[tyedco]

I noticed this issue as well. Fixed it by setting sleep from 1 second to 10 hours like @georglauterbach

.

[georglauterbach]

The proper fix should be included in the upcoming release (v10.5.0) 🚀

If the issue comes up again, please let us know :D

[github-actions]

This issue has become stale because it has been open for 20 days without activity.
This issue will be closed in 10 days automatically unless:

• a maintainer removes the meta/stale label or adds the stale-bot/ignore label
• new activity occurs, such as a new comment

[MichaelSp]

Looks like this is fixed now.

political statement @tyedco? 🇺🇦 🤣 nice!

[oblitum]

@MichaelSp no one was thinking of that at that time.

[casperklein]

The fix is included in the current 10.5.0 release.