UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0'

[mammique]

Hello, I've been trying to run the mailserver, but after many retries I keep ending up having the following error :

mailserver      | 2017-05-30T15:39:19.929715+00:00 mail dovecot: auth: Fatal: sql: driver not set in configuration file /etc/dovecot/dovecot-sql.conf.ext
mailserver      | 2017-05-30T15:39:19.931327+00:00 mail dovecot: master: Error: service(auth): command startup failed, throttling for 60 secs

I repeats itself over and over every 30~60 seconds, PostfixAdmin and RainLoop interfaces are showing in the browser, but impossible to connect to the service (via Thunderbird or RainLoop, howerver I'm getting a prompt response via telnet on ports 993 and 587). I followed every step of the README, except for the optional NSD section that I've commented in docker-compose.yml and DKIM entries that I omitted from the DNS configuration (I'm not familiar with this extra security stuff, I'd like to enable it later when the basic services work).

Am I getting something or is there a bug somewhere?

Thank you very much,

Camille.

[hardware]

Try this command : docker exec -ti cat /etc/dovecot/dovecot-sql.conf.ext and post the result here please.

[mammique]

Hi thank you for your reply. Here is the result of the command, but I guess that some container name is missing in the args or something like that:

# docker exec -ti cat /etc/dovecot/dovecot-sql.conf.ext
Error response from daemon: No such container: cat

Thanks,

Camille.

[mammique]

I think this was the intended command, it seems that the file is empty:

# docker exec -ti mailserver cat /etc/dovecot/dovecot-sql.conf.ext
#

If I try with another file path it says No such file or directory so I guess the file is indeed here, but empty…

[mammique]

I have tried to install mailserver several times from scratch, using the following commands to purge the previous installations:

docker rm -f $(docker ps -a -q)
docker rmi -f $(docker images -q)
docker rm -v -f $(docker ps -a -q)
docker volume rm $(docker volume ls -q)
rm -rf /mnt/docker/

I always get the same dovecot-sql.conf.ext error. But early install problems was memory limitation, so I increased the RAM wich solved problem, but maybe some corrupted files of first intallation remain somewhere and are disrupting the new installation process…?

[hardware]

Try to build yourself :

docker rmi hardware/mailserver
docker build --no-cache -t hardware/mailserver https://github.com/hardware/mailserver.git#master
docker-compose up -d

[mammique]

Same error with the self build :-/ dovecot-sql.conf.ext still empty… What can I do ? Thanks!

[hardware]

docker info

[mammique]

Here it is:

Containers: 5
 Running: 5
 Paused: 0
 Stopped: 0
Images: 15
Server Version: 17.03.1-ce
Storage Driver: devicemapper
 Pool Name: docker-202:1-160624-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.087 GB
 Data Space Total: 107.4 GB
 Data Space Available: 2.272 GB
 Metadata Space Used: 2.834 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.145 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.90 (2014-09-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Kernel Version: 3.10.105-xenU-34-082da8f-x86_64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.954 GiB
Name: yooook
ID: YLBO:D4KA:IVIR:NUKB:PIYQ:HGZ4:SZH2:QUBZ:VW5E:ISOI:3WXS:HPQR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Thank you,

Camille.

[hardware]

I have not tested hardware/mailserver with the devicemapper storage driver.

Officially, this docker image has been tested with AUFS and OverlayFS storage drivers.

https://docs.docker.com/engine/userguide/storagedriver/selectadriver/
https://docs.docker.com/engine/userguide/storagedriver/aufs-driver/
https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/

[hardware]

I just tried devicemapper driver on ArchLinux with Docker 17.05.0-ce and 4.11.2 kernel, it works well. Can you describe your host environment, your server setup ? You said you increased memory so I guess you're in a virtualized environment.

[mammique]

Thanks for your reply. Yes my host is a VM at https://www.gandi.net/hosting/iaas it's an old installation, about 4 years, but I've upgraded it regularly, it works well. Now Debian GNU/Linux 8.8 (jessie) with a 3.10.105-xenU-34-x86_64 Linux Kernel. There are very few basic LAMP services (I already have an Apache so I reconfigured mailserver's NGnix to ports 8001 and 4431, directly accessible as well as via Apache reverse proxy on ports 80 and 443). There is also a very old debian that I need to keep running for some services in a chroot. I installed docker from apt repositories, the kernel is compiled by the PaaS, known to run docker properly: https://wiki.gandi.net/en/iaas/references/server/docker That's all I think about as useful information. What else could I tell? Does it raise some flag for you?

Thank you,

Camille.

PS: PaaS talks about possible inode problems, but it seems to be plenty available still:

# df -ih
Sys. de fichiers Inœuds IUtil. ILibre IUti% Monté sur
/dev/root          640K    97K   544K   16% /
devtmpfs           250K    379   250K    1% /dev
tmpfs              251K      1   251K    1% /dev/shm
tmpfs              251K    585   250K    1% /run
tmpfs              251K      8   251K    1% /run/lock
tmpfs              251K     13   251K    1% /sys/fs/cgroup
tmpfs              251K      5   251K    1% /var/gandi
/dev/xvdc          1,4M    93K   1,3M    7% /yooook-chroot
tmpfs              251K      4   251K    1% /run/user/0
tmpfs              251K      4   251K    1% /run/user/1000
/dev/dm-1          640K    13K   628K    2% /var/lib/docker/devicemapper/mnt/ba9aa8afc42c1f29a9727917193192c3eadd6dabb45005e960384558bec56507
shm                251K      1   251K    1% /var/lib/docker/containers/7e959aec7a21400914805d421b55874d9876b1e4c4175d6bb93fd74b16886d08/shm
/dev/dm-2          640K    13K   628K    3% /var/lib/docker/devicemapper/mnt/6b3ebc4905e7be183397dfa11a643f27b6e1df0ead3c253dad9af66417fc7e01
shm                251K      2   251K    1% /var/lib/docker/containers/a2fc819193e0ba35afa1e0f94c34121bbd39b90c2464cc4a8055db7f04485d45/shm
/dev/dm-4          640K   3,3K   637K    1% /var/lib/docker/devicemapper/mnt/6f45ff37663e4587c3d6ba6de1a8dbff1798cd3c016ffd1cbf5dc77e22a801bd
shm                251K      1   251K    1% /var/lib/docker/containers/a5d3a5930d327959bb0af79ef409872378588dcff8860d66a00ecfdfce0d8d2c/shm
/dev/dm-3          640K   4,8K   636K    1% /var/lib/docker/devicemapper/mnt/1c80653a868cf75b8c76c7ab5926a7cfebf1f9f87ca469207e99c389ae3f6471
shm                251K      1   251K    1% /var/lib/docker/containers/31d17a2ba2fbe765dd90c4c926087f085370cc195ffc5724b89af4ec34fa7029/shm
/dev/dm-5          640K    599   640K    1% /var/lib/docker/devicemapper/mnt/af0c499a8ee82e0dbfe5b6c8f42f7c5ae4a5efa091906c17032c9b624f1e9a68
shm                251K      1   251K    1% /var/lib/docker/containers/f35b9d6dec7a9422fb85f8a8e9f8e711fc97e2148aab434597f99f8c6acf0329/shm

[hardware]

If you can, try with OverlayFS as described here : https://wiki.gandi.net/en/iaas/references/server/docker
Just in case.

[mammique]

Hi, sorry for the delay, OverlayFS wasn't available on my kernel, I had to migrate the whole VM to HVM to get a newer kernel, new configuration broke the network so it took time to get it back up. I now have OverlayFS:

# docker info | grep Storage
Storage Driver: overlay

But still the same SQL problem… :-/

What can I do?

Thank you,

Camille.

[michael-k]

Does the container have any mounts that shadow the original file?

docker inspect -f "{{json .Mounts}}" mailserver

Which image is the container based on?

docker ps | grep "CONTAINER\|mailserver"

Which id does the image have?

docker images | grep -e "REPOSITORY\|mailserver"

[mammique]

Hi, thanks for your reply, here are the outputs:

# docker inspect -f "{{json .Mounts}}" mailserver
[{"Type":"bind","Source":"/mnt/docker/mail/opendkim","Destination":"/etc/opendkim/keys","Mode":"rw","RW":true,"Propagation":""},{"Type":"bind","Source":"/mnt/docker/mail","Destination":"/var/mail","Mode":"rw","RW":true,"Propagation":""},{"Type":"volume","Name":"9c7ac7dbda3a8c1f9a7b3fda4f86330e02201ffb247aab2e2aed159610fe422f","Source":"/var/lib/docker/volumes/9c7ac7dbda3a8c1f9a7b3fda4f86330e02201ffb247aab2e2aed159610fe422f/_data","Destination":"/etc/letsencrypt","Driver":"local","Mode":"","RW":true,"Propagation":""}]

# docker ps | grep "CONTAINER\|mailserver"
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                                                                                                   NAMES
8d3304e66572        hardware/mailserver       "tini -- startup"        43 seconds ago      Up 42 seconds       0.0.0.0:25->25/tcp, 0.0.0.0:143->143/tcp, 0.0.0.0:587->587/tcp, 0.0.0.0:993->993/tcp, 0.0.0.0:4190->4190/tcp, 465/tcp   mailserver

# docker images | grep -e "REPOSITORY\|mailserver"
REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
hardware/mailserver       latest              09e33a1281f6        2 hours ago         236 MB

Thanks,

Camille.

[hardware]

Check the file attributes and Dovecot configuration :

docker exec -ti mailserver ls -l /etc/dovecot/
docker exec -ti mailserver doveconf -N

Verify the content of several other files, it may not be the only one in this case.

[mammique]

Indeed two files are empty, dovecot-dict-sql.conf.ext and dovecot-sql.conf.ext:

# docker exec -ti mailserver ls -l /etc/dovecot/
total 28
-rw-r----- 1 vmail dovecot  121 Apr 11 07:54 README
drwxr-x--- 1 vmail dovecot 4096 Jun  3 20:17 conf.d
-rw-r----- 1 vmail dovecot 1507 Apr 11 07:54 dovecot-dict-auth.conf.ext
-rw-r----- 1 vmail dovecot    0 Jun  3 20:17 dovecot-dict-sql.conf.ext
-rw-r----- 1 vmail dovecot  324 Jun  3 18:11 dovecot-dict-sql.conf.ext.tpl
-rw-r----- 1 vmail dovecot    0 Jun  3 20:17 dovecot-sql.conf.ext
-rw-r----- 1 vmail dovecot  613 Jun  3 18:11 dovecot-sql.conf.ext.tpl
-rw-r----- 1 vmail dovecot  117 Jun  3 20:17 dovecot.conf
drwx------ 1 vmail dovecot 4096 Apr 11 07:53 private

Wich other files should I check?

Thanks,

Camille.

[hardware]

-rw-r----- 1 vmail dovecot    0 Jun  3 20:17 dovecot-dict-sql.conf.ext
-rw-r----- 1 vmail dovecot  324 Jun  3 18:11 dovecot-dict-sql.conf.ext.tpl
-rw-r----- 1 vmail dovecot    0 Jun  3 20:17 dovecot-sql.conf.ext
-rw-r----- 1 vmail dovecot  613 Jun  3 18:11 dovecot-sql.conf.ext.tpl

Interesting, envtpl rendering does not seem to work in your case, strange.

Wich other files should I check?

Some of them : https://github.com/hardware/mailserver/blob/master/rootfs/usr/local/bin/startup#L182-L207

I need mailserver logs too : docker logs -f mailserver

[mammique]

Here it is, I've also added the build logs.

docker build --no-cache -t hardware/mailserver https://github.com/hardware/mailserver.git#master → https://pastebin.com/DUadSrYE

ls: cannot access /etc/cron.d/fetchmail: No such file or directory
-rw-r--r-- 1 root      root       426 Jun  4 14:05 /etc/amavis/conf.d/05-node_id
-rw-r--r-- 1 root      root      9848 Jun  4 14:05 /etc/amavis/conf.d/20-debian_defaults
-rw-r--r-- 1 root      root       803 Jun  4 14:05 /etc/amavis/conf.d/50-user
-rw-r--r-- 1 root      root       204 Jun  4 14:05 /etc/cron.d/spamassassin
-rw-r----- 1 vmail     dovecot    253 Jun  4 14:05 /etc/dovecot/conf.d/10-mail.conf
-rw-r----- 1 vmail     dovecot    255 Jun  4 14:05 /etc/dovecot/conf.d/10-ssl.conf
-rw-r----- 1 vmail     dovecot    120 Jun  4 14:05 /etc/dovecot/conf.d/15-lda.conf
-rw-r----- 1 vmail     dovecot    101 Jun  4 14:05 /etc/dovecot/conf.d/20-lmtp.conf
-rw-r----- 1 vmail     dovecot      0 Jun  4 14:05 /etc/dovecot/dovecot-dict-sql.conf.ext
-rw-r----- 1 vmail     dovecot      0 Jun  4 14:05 /etc/dovecot/dovecot-sql.conf.ext
-rw-r--r-- 1 root      root        15 Jun  4 14:05 /etc/mailname
-rw-r--r-- 1 opendkim  opendkim   769 Jun  4 14:05 /etc/opendkim/opendkim.conf
-rw-r--r-- 1 opendmarc opendmarc  478 Jun  4 14:05 /etc/opendmarc/opendmarc.conf
-rw-r--r-- 1 root      root       863 Jun  4 14:05 /etc/postfix/header_checks
-rw-r--r-- 1 root      root      7947 Jun  4 14:05 /etc/postfix/main.cf
-rw-r--r-- 1 root      root         0 Jun  4 14:05 /etc/postfix/sql/sender-login-maps.cf
-rw-r--r-- 1 root      root         0 Jun  4 14:05 /etc/postfix/sql/virtual-alias-domain-catchall-maps.cf
-rw-r--r-- 1 root      root         0 Jun  4 14:05 /etc/postfix/sql/virtual-alias-domain-mailbox-maps.cf
-rw-r--r-- 1 root      root         0 Jun  4 14:05 /etc/postfix/sql/virtual-alias-domain-maps.cf
-rw-r--r-- 1 root      root         0 Jun  4 14:05 /etc/postfix/sql/virtual-alias-maps.cf
-rw-r--r-- 1 root      root         0 Jun  4 14:05 /etc/postfix/sql/virtual-mailbox-domains.cf
-rw-r--r-- 1 root      root         0 Jun  4 14:05 /etc/postfix/sql/virtual-mailbox-maps.cf
-rw-r--r-- 1 root      root       140 Jun  4 14:05 /etc/postfix/virtual
-rw-r--r-- 1 root      root         0 Jun  4 14:05 /etc/postfixadmin/fetchmail.conf
-rwxr-xr-x 1 root      root       247 Jun  4 14:05 /usr/local/bin/quota-warning

docker logs -f mailserver → https://pastebin.com/QcdetTYG

Thanks,

Camille.

[mammique]

Oh, I see envtpl is crashing because of Python UTF-8/unicode problem… I saw it first time, but I thought it was just a warning, but in fact it crashes…

[hardware]

Ok, I see the problem. Try to use only ASCII printable characters in environment variables (ASCII table).

To solve your problem, checks the value of these variables and remove any non-ASCII characters :

DBHOST, DBPORT, DBNAME, DBUSER, DBPASS

\xb0 = °
http://www.fileformat.info/info/unicode/char/b0/index.htm

envtpl/envtpl#20 (comment)

Thanks for reporting this.

[mammique]

Thanks! It worked! :-) Bad characters were in DBPASS, BTW it seems to dislike # as well. Also I had a bug building the git version of hardware/mailserver:

Installing collected packages: envtpl, Jinja2, MarkupSafe
  Running setup.py install for envtpl
    
    Installing envtpl script to /usr/local/bin
Compiling /tmp/pip-build-bkv1Bj/Jinja2/jinja2/asyncfilters.py ...
  File "/tmp/pip-build-bkv1Bj/Jinja2/jinja2/asyncfilters.py", line 7
    async def auto_to_seq(value):
            ^
SyntaxError: invalid syntax

Compiling /tmp/pip-build-bkv1Bj/Jinja2/jinja2/asyncsupport.py ...
  File "/tmp/pip-build-bkv1Bj/Jinja2/jinja2/asyncsupport.py", line 22
    async def concat_async(async_gen):
            ^
SyntaxError: invalid syntax

I had to pull the ready made image. It worked.

So I'll finally give a try to NSD, but can I use the same key for multiple domains? I see /mnt/docker/mail/opendkim/ lists only the initial domain despite the fact that I add new ones on postfixadmin and rainloop?

Thanks!

Camille.

[hardware]

So I'll finally give a try to NSD, but can I use the same key for multiple domains? I see /mnt/docker/mail/opendkim/ lists only the initial domain despite the fact that I add new ones on postfixadmin and rainloop?

Use ADD_DOMAINS environment variable.

[mammique]

My bad, RTFM. Thanks a lot!

Camille.