Fatal: write(indexer) failed: Resource temporarily unavailable

[grosjo]

From @ams001

Each time I start an index, I get the following:

> doveadm index -q -u fts-test@example.com \*

doveadm(fts-test@example.com): Info: FTS Xapian: Starting with partial=3 full=20 attachments=0 verbose=1
doveadm(fts-test@example.com): Fatal: write(indexer) failed: Resource temporarily unavailable

but indexer-worker does run in the background. It does this now on every mailbox. Is this really an error? Either way, it needs addressing.

[grosjo]

@ams001 Are you getting this only from doveadm or also in normal indexing ?

[ams001]

@grosjo I've only seen it in doveadm and only with fts-xapian.

[grosjo]

@ams001 Dovecot teams point me to https://wiki.dovecot.org/SocketUnavailable

Can you verify your own setting ? (especially ulimut et imap settings)

[ams001]

client limit was not set. I have now set to 1. ulimit -n is 65535. I'm currently running other tests but will see if client_limit=1 as they suggest makes a difference between runs.

Thanks

[ams001]

I have just shutdown and restarted dovecot imap, same error, and nothing in the logs about client_limit or process_limit. However, because I am clearing the index prior to reindexing, I do see errors about fts-xapian being unable to open indexer files for folders within an account.

Error: FTS Xapian: Can not open RO index (Folder) /var/vmail/example.com/aaaa/xapian-indexes/db_7f5af7ba291b2df1a11d573bdb55d7e9 : DatabaseOpeningError - No such file or directory
Error: FTS Xapian: Can not open RO index (Folder) /var/vmail/example.com/aaaa/xapian-indexes/db_7f5af7ba291b2df1a11d573bdb55d7e9 : DatabaseOpeningError - No such file or directory

2x per folder.

This makes sense because everything has been deleted, and after seeing the error, the directory exists - presumably having been created. This should be a warning or info message though that a new index for a folder has been created because one did not exist previously.

Could this error be causing the temporary error?

[grosjo]

This is a bug indeed. I fixed it latest git. (you are great at finding bugs ! :D )
Please try again

[ams001]

The latest git removed the errors and provided an info message that the folder/directory was being created, as per your fix. Thanks.

Unfortunately the

Fatal: write(indexer) failed: Resource temporarily unavailable

message remains when initiating a reindex:

doveadm index -u aaaaa@example.com -q '*'

Oddly, it does not always do this when running the index in the foreground (i.e. without -q). I wonder if this is a race condition of some kind?

[grosjo]

This is my settings:

service imap-login {
        chroot = login
        service_count = 0
}

protocol imap {
        imap_client_workarounds = delay-newmail tb-extra-mailbox-sep
        imap_max_line_length = 655360
        mail_max_userip_connections = 256
        mail_plugins = $mail_plugins imap_zlib
}

Maybe the "max_userip_conn.." makes the difference ?

[grosjo]

If your server uses 'systemd', make sure "LimitNOFILE=65535" is uncommented

[ams001]

If your server uses 'systemd', make sure "LimitNOFILE=65535" is uncommented

As per previous comment open files limit is 65535. The above is how I achieved it.

I also briefly tested your settings but that give the same error every time I flushed and recreated the index for a user's account (all folders).

However, when I switched back to solr (restarting everything of course), for the first user's reindex, I got the same error back from doveadm -q. All other reindex requests returned no error. That's enough to say its not necessarily fts-xapian specific, just that whatever the cause, fts-xapian triggers it every time. Nothing in the logs to indicate why :-(

[grosjo]

You shall mention that problem to Dovecot team

[grosjo]

I close this issue as it is not related specifically to this plugin but to a wider scope