[bug]: Fails to start when Elasticsearch is down/starting up

[sirrkitt]

What happened?

When using Elasticsearch for the FTS store, if Elasticsearch hasn't finished starting up, or if it is down, Stalwart disables all of its stores and then you can't use it. If you restart Stalwart once Elasticsearch is back to accepting connections, then it starts up and all is fine.

I did notice that if Elasticsearch becomes unavailable while Stalwart is already running, it just gives an error message and continues working (minus the FTS).

I'm just wondering if it'd be possible for Elasticsearch to not prevent Stalwart from starting up, if for whatever reason Elasticsearch isn't ready by the time Stalwart boots up.

I attached in the log area, below, an example of when Elasticsearch is down, preventing Stalwart from starting up.

I also attached an example of when Elasticsearch goes down while Stalwart is already running, which lets Stalwart to continue running, minus the FTS capabilities.

How can we reproduce the problem?

I can reproduce the problem by doing the following steps:

Version

v0.6.x

What database are you using?

RocksDB

What blob storage are you using?

RocksDB

Where is your directory located?

Internal

What operating system are you using?

Docker

Relevant log output

Elasticsearch down, trying to start Stalwart:


Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.456868Z  INFO common::manager::boot: Starting Stalwart Mail Server v0.7.0...                                                                           Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.518278Z ERROR smtp::reporting::scheduler: Failed to read from store: Internal Error: No store configured context="queue" event="error"                 Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.518636Z  WARN jmap: Failed to unpack webadmin bundle. event="error" error=InternalError("No store configured")
Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.518836Z ERROR jmap::services::index: Failed to iterate over index emails context="fts_index_queued" event="error" reason=InternalError("No store configured")
Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.518923Z ERROR utils::config: Build error for key "store.fts": Internal Error: ElasticSearch error: error sending request for url (https://localhost:9200/stalwart_email): error trying to connect: tcp connect error: Connection refused (os error 111)                                                                                                                 Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.518947Z ERROR utils::config: Build error for key "storage.*": One or more stores are missing, disabling all stores                                     Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.518967Z ERROR utils::config: Failed to parse setting "storage.fts": Full-text store "fts" not found                                                    Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.518973Z  INFO common::listener::listen: Starting listener id="http" protocol=Http bind.ip="::" bind.port=8080 tls=false                                Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.519048Z ERROR smtp::queue::spool: Failed to read from store: Internal Error: No store configured context="queue" event="error"                         Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.519082Z  INFO common::listener::listen: Starting listener id="https" protocol=Http bind.ip="::" bind.port=443 tls=true
Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.519124Z  INFO common::listener::listen: Starting listener id="imap" protocol=Imap bind.ip="::" bind.port=143 tls=false                                 Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.519182Z  INFO common::listener::listen: Starting listener id="imaptls" protocol=Imap bind.ip="::" bind.port=993 tls=true                               Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.519237Z  INFO common::listener::listen: Starting listener id="sieve" protocol=ManageSieve bind.ip="::" bind.port=4190 tls=false                        Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.519291Z  INFO common::listener::listen: Starting listener id="smtp" protocol=Smtp bind.ip="::" bind.port=25 tls=false                                  Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.519353Z  INFO common::listener::listen: Starting listener id="submission" protocol=Smtp bind.ip="::" bind.port=587 tls=false
Apr 12 03:17:13 mail stalwart[25547]: 2024-04-12T10:17:13.519409Z  INFO common::listener::listen: Starting listener id="submissions" protocol=Smtp bind.ip="::" bind.port=465 tls=true
















Stalwart already running, Elasticsearch goes down:


Apr 12 03:23:24 mail stalwart[25951]: 2024-04-12T10:23:24.083303Z ERROR jmap::services::index: Failed to index email in FTS index context="fts_index_queued" event="error" account_id=0 document_id=4296 reason=InternalError("ElasticSearch error: error sending request for url (https://localhost:9200/stalwart_email/_doc): error trying to connect: tcp connect error: Connection refused (os error 111)")
Apr 12 03:23:24 mail stalwart[25951]: 2024-04-12T10:23:24.084617Z ERROR jmap::services::index: Failed to index email in FTS index context="fts_index_queued" event="error" account_id=0 document_id=4297 reason=InternalError("ElasticSearch error: error sending request for url (https://localhost:9200/stalwart_email/_doc): error trying to connect: tcp connect error: Connection refused (os error 111)")
Apr 12 03:23:24 mail stalwart[25951]: 2024-04-12T10:23:24.086160Z ERROR jmap::services::index: Failed to index email in FTS index context="fts_index_queued" event="error" account_id=0 document_id=4298 reason=InternalError("ElasticSearch error: error sending request for url (https://localhost:9200/stalwart_email/_doc): error trying to connect: tcp connect error: Connection refused (os error 111)")

Code of Conduct

• I agree to follow this project's Code of Conduct