[RFC] Add Solr FullTextSearch to dovecot

[Nebukadneza]

What type of PR?

RFC / Feature

I’m not overly happy with some parts:

• Copying the default lang-files, stopwords, all that shizzle in is ugly. This is what solr creates when initializing, and if we want to avoid a bunch of ugly when-do-i-need-to-initialize-statehandling-hacks, this is the best option i’ve found. Maybe someone with more solr knowledge can chime in?
• Running solr on alpine can be subject to discussion. It looked easier than the official container, and it fits our scheme of having everything self-fiddled on alpine better … but it’s another set of Dockerfiles and startup logic to maintain …

As this is RFC, I didn’t adapt setup yet, so if anyone wants to test it, this is the compose part:

  imap:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-1.6}
    restart: always
    env_file: mailu.env
    volumes:
      - "/mailu/mail:/mail"
      - "/mailu/overrides:/overrides"
    depends_on:
      - front
      - solr

  solr:
    image: mailu/solr:latest
    restart: always
    volumes:
      - "/mailu/solr:/opt/solr/server/solr/dovecot/data"

What does this PR do?

Add a alpine-based SOLR server, to be able to enable the respective FTS plugin in dovecot. This could supersede the broken-since-ages lucene FTS engine.

Related issue(s)

• Mention an issue like: Re-enable FTS when it proves stable #751

Prerequistes

• In case of feature or enhancement: documentation updated accordingly
• Unless it's docs or a minor change: add changelog entry file.

[mergify]

Thanks for submitting this pull request.
Bors-ng will now build test images. When it succeeds, we will continue to review and test your PR.

bors try

Note: if this build fails, read this.

[bors]

try

Build succeeded

• continuous-integration/travis-ci/push

[micw]

Shouldn't this be part of roundcube, not of dovecot?
Edit: Understood it better after reading the changes.

[micw]

Is squat fts really broken or just deprecated?

Adding solr adds much complexity and increases memory requirements a lot (from 1 to 3 gb?). If we add it, I think adding the original image is better than maintaining an own. It's similar to redis, "just" a tool that needs to run along with mailu.

We could also consider to use https://github.com/grosjo/fts-xapian which is directly integrated in dovecot.

[tonobo]

i suggest to only add the fts-solr package by default. so i would be able to add further fts stuff if i want to, but it wouldn't be configured by default. what do you think about it?

[kaiyou]

It sounds fine. Your PR is still interesting however. I wonder how we could easily make this less ressource hungry, or at least optional without introducing too much complexity.

[tonobo]

@kaiyou fts-elastic would be cooler, but there is actually no apk available for this :D

[kaiyou]

If it is in the official source, it is probably a matter of contributing the build script. We already contributed to most dovecot apk, I guess it is only reasonable we work on this :)

[tonobo]

its not official but mentioned on the dovecot site and well supported i think. checkout https://github.com/filiphanes/fts-elastic

[tonobo]

i personally prefer elasic over solr because i'm more familiar with elasticsearch, but thats the only reason 😄

[micw]

Can we provide support for multiple fst implementations and use the least resource hungry as a default? What happens without fst, would this mean, server-side mail search won't work at all?

[kaiyou]

Proposed moto: simplicity. Let's start and stick with one first. When we are confident it runs fine, let's think about supporting many 😅

I agree with Elastic, and it should be somewhat straightforward to provide. Also, setup will be easier.

[ajgon]

Ouch, for whatever reason GH search didn't return this PR when I checked for SOLR, so I added my own here - #1297 .

This configuration is based on my custom mailu build with solr, which I use for almost a year, and didn't have any issues with it - so if it helps, feel free to take some parts from it :)

[Nebukadneza]

Hi,

so, I think I’m going to close this one, as i think that #1297 is cleaner code-wise. Also, the issues raised in this discussion about whether solr is the right option also has a good point — I think we should go with a lean, simple and small solution to fit the scope of mailu.