Dovecot Xapian plugin broken? (undefined symbol: fts_user_get_language_list)

[bocan]

Is it just me or has the xapian indexing broken with the latest edge build? I try to manually index my mail and the command just instantly returns. The logs say:

Debug: Skipping module doveadm_fts_plugin, because dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so:
undefined symbol: fts_user_get_language_list

Thought it might be a variety of PRs at fault. Tried reverting them and rebuilding it a few times and it's bloody 3:25am. Nothing fixed it - so maybe it's just me.

Is this not the command that should do it?
docker exec mailserver doveadm index -A -q \*

[polarathene]

I've opened an associated bug report issue: #4719

---

There was a recent PR which would affect the Xapian feature with :edge builds. It was due to Github having taken down the upstream source, but that has since been restored.

If the commit prior doesn't encounter the same error, we could revert that change.

AFAIK, we only build from source for compatibility with custom builds of DMS (prior releases before Debian 12 IIRC would have used the Dovecot CE repo package instead, but Debian 12 only has access to Dovecot 2.4 from that which is incompatible with DMS until v16 which will upgrade to Debian 13).

So if a build from the upstream repo as we had been doing before with :edge doesn't work, something else has changed that prevents a reproducible build. You could try the Debian package instead in that case (there is a notable version difference however).

Have you considered Apache Solr? Some DMS users have had various issues raised about Xapian, and found Solr worked better for them. It does require a slight tweak to DMS and a separate container for Solr itself, this is all documented here in our docs if you are unable to resolve the Xapian issue.

---

Debug: Skipping module doveadm_fts_plugin, because dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so:
undefined symbol: fts_user_get_language_list

I'm short on time, but if this is easy to reproduce, it might be beneficial to get into our test suite. It's possible that something about the change in how we build the plugin prevented that expected symbol from being available 😓

There's also been talk about dropping Xapian in favor of Flatcurve once Dovecot 2.4 lands in DMS with v16 release (Flatcurve is bundled with Dovecot by default IIRC and is meant to be a better choice?).

---

Is this not the command that should do it?
docker exec mailserver doveadm index -A -q \*

AFAIK this is a community contributed feature, so none of the active maintainers may be familiar enough to assist. It would be good to ensure it's resolved before DMS v16 is released though 😅

[bocan]

I'm afraid I've walked myself into this with bad timing and now I need to see it through.

I'm a long time docker-mailserver user. It started with me thinking yesterday about the amount of spam getting through rspamd and how much worse it's been getting over the last year. Wondered what version rspamd was and if it was greatly behind so I started digging into it to figure that out. Maybe even run rspamd outside docker mailserver. Then I spotted the xapian package. "What is that?", I wondered. Looked it up. Ah! That explains why my search is literally useless with mailboxes my size. Next thing I know I looked out the window and note the sun is starting to come up!

In the next hour or so I'll build the debian 13 branch. See if it works with that by default. Whether it does or not, I think I might need to cherry pick bits out of #4357 and #4557 if they're not in the new branch? I'll test that too if possible.

[polarathene]

It started with me thinking yesterday about the amount of spam getting through rspamd and how much worse it's been getting over the last year.

Wondered what version rspamd was and if it was greatly behind so I started digging into it to figure that out.

Each image release has a fixed version of rspamd that was the latest release at the time. But rebuilding any release of DMS will use the latest release at build time, since upstream rspamd project does not provide a repo with older releases that we can pin to (nor are they interested in supporting that).

I've contemplated switching to the Debian package with Debian 13, so that we can at least ensure by default a predictable version.

I had raised a concern with upstream about if they release a major version with breaking changes how that'd affect our builds, at the time I think they said that wouldn't happen (but IIRC debian 12 now would build with Rspamd 4.x instead of the 3.x release we had).

---

Maybe even run rspamd outside docker mailserver.

That should be doable, I think you'd just want to add user-patches.sh that updates our Postfix config to use rspamd?

docker-mailserver/target/scripts/startup/setup.d/security/rspamd.sh

Lines 157 to 165 in 7290055

	# Adjust Postfix's configuration files. We only need to append Rspamd at the end of
	# `smtpd_milters` in `/etc/postfix/main.cf`.
	function __rspamd__setup_postfix() {
	__rspamd__log 'debug' "Adjusting Postfix's configuration"
	postconf 'rspamd_milter = inet:localhost:11332'
	# shellcheck disable=SC2016
	_add_to_or_update_postfix_main 'smtpd_milters' '$rspamd_milter'
	}

Just be aware that TCP IP/port in a container is reachable by any other container in the same connected network, ports don't need to be published for that. If that's a security concern for you, you can presumably listen on a unix socket instead and share that between the Rspamd and DMS containers.

[bocan]

Excellent news! Just the vanilla debian-13 branch works with xapian, but the config/dovecot.cf has to be changed from:

plugin {
  fts = xapian
  fts_xapian = partial=3 full=20 verbose=1
  fts_autoindex = yes
  fts_enforced = yes
}

service indexer-worker {
  vsz_limit = 1GB
  extra_groups = docker
}

protocol imap {
  mail_plugins = $mail_plugins fts fts_xapian
}

protocol lmtp {
  mail_plugins = $mail_plugins fts fts_xapian
}

To:

mail_plugins {
  fts = yes
  fts_xapian = yes
}

language "en" {
  default = yes
}

fts xapian {
  partial = 3
  verbose = 1
  lowmemory = 500
  maxthreads = 4
}

service indexer-worker {
  vsz_limit = 1GB
  extra_groups = docker
}

If anyone ends up catching debian-13 up with main, let me know and I'll rebuild and retest. I'm going to stay on that branch on my mail server for now unless anything terrible rattles out of it. I don't use LDAP or anything exotic and the logs look good all round.

[polarathene]

Excellent news! Just the vanilla debian-13 branch works with xapian

Oh, well that's good to hear. Still I'm curious what went wrong with the :edge release.

If anyone ends up catching debian-13 up with main, let me know and I'll rebuild and retest.

Just subscribe to the PR for it and you'll be notified once that's merged 👍

I've been blocking it until I've had time to go through the changes properly, but hoping to get that sussed out ASAP.

You can see a list of commits that the master branch has landed which haven't been carried over to debian-13 branch here, one of those was the Xapian PR I mentioned that switched from the source repo to a Debian repo for building, so debian-13 doesn't have that yet.

---

I know you've tried to hunt down the breakage affecting :edge, but do you know if the failure you encountered is easy to reproduce?

Does it just need Xapian enabled? I could look into reproducing it with a test mail and then running docker exec mailserver doveadm index -A -q \* if that presumably produces the error you encountered.

[bocan]

I'll try and see if I can figure it out properly in :edge tonight. All I know so far, is that what's there doesn't appear to be compatible at all with the 2.3 version of Dovecot - which doesn't make sense if it's compiled.

I think it's easy to replicate by adding this to your config/dovecot.cf:

plugin {
  fts = xapian
  fts_xapian = partial=3 full=20 verbose=1
  fts_autoindex = yes
  fts_enforced = yes
}

service indexer-worker {
  vsz_limit = 1GB
  extra_groups = docker
}

protocol imap {
  mail_plugins = $mail_plugins fts fts_xapian
}

protocol lmtp {
  mail_plugins = $mail_plugins fts fts_xapian
}

Then trying to index all your mail from scratch:

docker exec mailserver doveadm index -A \*

That command should have a lot of of output on a mailbox of any reasonable size...

But on edge it won't even print anything.

It would be so much faster to get to the bottom of this if I could build native on my macbook as arm64 instead of building amd64 and shifting it somewhere. 😩

[polarathene]

It would be so much faster to get to the bottom of this if I could build native on my macbook as arm64 instead of building amd64 and shifting it somewhere. 😩

I don't own a mac, but DMS has build support for aarch64 / arm64, so it shouldn't be an issue? I know custom Dovecot builds (if enabled) don't offer arm64 support however, but I don't think that'd affect the Xapian compilation.

---

While DMS doesn't implement it, it is often quite possible to define a Dockerfile that minimises the need for a native arch runner or slow emulation via QEMU to build for a different target arch.

I find this easiest with Fedora as a base image and using it's dnf --forcearch=<arch> --installroot=/rootfs (I don't have the exact command and args on me, but this will install only the minimal packages needed for a new base image at /rootfs, which you then COPY to a FROM scratch stage afterwards).

In that same Fedora build stage, you can cross-compile with tools like zig cc/zig c++ or other languages that can use zig for any c/c++ build deps (zig can also lower the glibc requirement, which can increase portability). The Fedora package manager also helps avoid building by supplying devel packages of libs for linking against. If necessary, those linked libs can be copied along with the compiled binary, and patchelf can be used to set DT_RUNPATH/DT_RPATH to better enforce an isolated set of libs to run with instead of system libs.

That approach is a bit overkill for most and involves more complexity, but it absolutely works for fast cross-compilation. However it's much simpler to leverage image layer caching along with build cache mounts (in RUN) as appropriate, which can greatly minimize overhead of QEMU for subsequent builds at least.

[bocan]

Ok! Stand down. Figured it out.

TLDR:

Sorry, it was me, not the edge image.
There's nothing wrong with :edge as far as I can tell. I just missed the top line when I copy/pasted from the example config

I'll write a small tomb here to explain this in case anyone else stumbles across it in future.

What I did to test:

What I did was quickly get Claude Code to vibe-code a test harness. From start to finish it:

1. Builds (or reuses) the image. By default it builds the docker-mailserver image fresh for linux/amd64, streaming the full build log to a file so I could watch it. It also lets me point it at an existing image (like mailserver/docker-mailserver:edge) and skip the build.
2. Starts a throwaway mail server. It spins up a minimal container with most services (ClamAV, Rspamd, etc) turned off for speed, and mounts a small Dovecot config snippet that switches the Xapian backend on.
3. Creates a test account and sends itself an email. The email body contains a unique random word, so later I can search for something I know is in exactly one message.
4. Triggers indexing. It runs doveadm index to tell Dovecot to build the Xapian search index for that mailbox.
5. Checks the results and prints a verdict. It looks at two things that have to be true for FTS to genuinely work, and reports each as a gate:
   - Gate A: the plugin loaded cleanly. It scans the logs for real plugin load failures (ABI or missing symbol errors).
   - Gate B: an index was actually written to disk. It confirms the Xapian database files (the xapian-indexes folder and its .glass files) really exist and are not empty.

It's here if of any use to anyone ever again: https://gist.github.com/bocan/af9691d4589f48089bd237979f718474

First, I built directly from master, ran the harness, and it confirmed everything worked and there was no ABI mismatch.

So, I thought, maybe it's a problem with the :edge imgage. Pulled that and ran it. And it worked too!

I Dig Deeper...

Why it looked like an ABI problem

When full text search wasn't working, the logs showed this line:

  Skipping module doveadm_fts_plugin, because dlopen() failed: ...
  undefined symbol: fts_user_get_language_list

"undefined symbol" is the classic fingerprint of an ABI mismatch, which is when a plugin was compiled against a different version of Dovecot than the one actually running, so the two no longer fit together.

The message names the FTS plugin specifically, and search really was broken, so it felt reasonable to read this as "the Xapian plugin is built wrong." That is the conclusion I jumped to, and it is why I raised this discussion. 😞

Why that was a false alarm

That message is harmless, and it shows up even in a perfectly healthy, working image. Here is what is really happening.

Dovecot loads its plugins in several passes. Some plugins depend on others. On an early pass, a plugin whose helper hasn't loaded yet will fail with "undefined symbol," get set aside, and then load successfully on a later pass once its dependency is in place. Dovecot knows this is normal, which is why the very same log line ends with its own reassurance: "(this is usually intentional, so just ignore this message)."

I confirmed this by running the exact same command against a clean, untouched :edge image. It printed the identical "undefined symbol" line, then a moment later printed "Module loaded" for that same plugin, and full text search worked fine. So the message was noise, not the cause.

What the real cause was

The actual problem was a small configuration scoping mistake. The Xapian plugin was switched on like this:

plugin {
  fts = xapian
  fts_xapian = partial=3 full=20 verbose=1
  fts_autoindex = yes
  fts_enforced = yes
}

service indexer-worker {
  vsz_limit = 1GB
  extra_groups = docker
}

protocol imap {
  mail_plugins = $mail_plugins fts fts_xapian
}

protocol lmtp {
  mail_plugins = $mail_plugins fts fts_xapian
}

That enables it for the IMAP protocol (what mail clients use) and the LMTP protocol (how new mail is delivered), but not globally. The catch is that the indexing command, doveadm index, does not run as IMAP or LMTP. It runs in its own separate "doveadm" context, and that context only sees the global setting, where the plugin was never added.

So when doveadm index -A '*' was run to index all the existing mail, it had no Xapian backend loaded at all. It quietly did nothing and returned instantly, which is exactly the "returns without saying a word" behaviour that was seen. The large backlog of existing mail never got indexed, so searches came back empty. New mail arriving after that point would get indexed automatically (because LMTP did have the plugin), but the old mail needed that manual index run, and the manual run was silently a no op.

I proved this directly: with that config, Dovecot reports the plugin active for imap and lmtp but the doveadm context sees only quota, and a full index run wrote zero index files.

The fix

I set the plugin globally instead of per protocol, by putting this at the top level of the config (outside any protocol block):

  mail_plugins = $mail_plugins fts fts_xapian     # <-- global; this is the fix

  plugin {
    fts = xapian
    fts_xapian = partial=3 full=20 verbose=1
    fts_autoindex = yes
    fts_enforced = yes
  }

  service indexer-worker {
    vsz_limit = 1GB
  }
  
  # Protocol blocks not needed!  I tested this

With that in place, the doveadm context loads Xapian too, doveadm index -A '*' actually builds the index, and search works.

Why upgrading to the debian-13 / dovecot 2.4 branch appeared to fix it

It wasn't an ABI fix, because there was no ABI problem. Rebuilding onto the other branch needed a correctly rewrote config, which happened to put the plugin where the indexer could see it. The same result is achieved on :edge just by moving that one mail_plugins line to the global scope.

[polarathene]

Thanks for looking further into this and sharing an update about the actual cause and fix ❤️

I'd need to look into it a bit more to verify if that's what we should have been advising instead of scoping it to a protocol. (EDIT: Nevermind, misunderstanding we already advise the "fix" 😅)

I understand why it's the solution, but I still like to confirm there aren't any caveats from doing so (sometimes a solution doesn't take into account other scenarios and regresses them or introduces new problems that aren't immediately obvious).

Especially when relying on AI assist to vibe code things, I encourage caution (in general.. not just specific to this experience since as smart as some AI agents are, they're also still very much capable of misleading guidance).

[polarathene]

There's nothing wrong with :edge as far as I can tell. I just missed the top line when I copy/pasted from the example config

Oh... I had just looked at that and realized we weren't advising the protocol scoping 😓 My bad.

So how did the protocol blocks get introduced? Was that something Claude had done? Or you had added yourself manually while trying to troubleshoot from the missing mail_plugins line?

[bocan]

It was me in that case. Without the first line, it still starts, but other things break so I added those blocks from another example I found elsewhere. :-(