🐛: Domain listing lists no domains and says "Not found" for a domain

[akosiaris]

What happened?

I find myself in a weird position. Domain listing for domains doesn't work, see screenshot

I can not create the domain that the UI says it can not find, as I get back

I can create other domains, being unable to delete them afterwards as they aren't listed.

I can create and delete accounts in the Accounts tab. Those accounts can use email addresses in the domain that is erroring out.

I receive emails normally in all accounts and all domains configured in this instance (at least that I can tell of)

I am not sure how to proceed with debugging this further, any kind of help would be appreciated. Current config file (minus passwords and other more instance specific information I omitted out) is:

    directory.internal.store = "postgres"
    directory.internal.type = "internal"
    server.http.allowed-endpoint = 200
    server.http.hsts = false
    server.http.permissive-cors = false
    server.http.use-x-forwarded = true
    server.listener.http.bind = "[::]:8080"
    server.listener.http.protocol = "http"
    server.listener.imaptls.bind = "[::]:993"
    server.listener.imaptls.protocol = "imap"
    server.listener.imaptls.tls.implicit = true
    server.listener.sieve.bind = "[::]:4190"
    server.listener.sieve.protocol = "managesieve"
    server.listener.smtp.bind = "[::]:25"
    server.listener.smtp.protocol = "smtp"
    server.listener.submission.bind = "[::]:587"
    server.listener.submission.protocol = "smtp"
    server.listener.submissions.bind = "[::]:465"
    server.listener.submissions.protocol = "smtp"
    server.listener.submissions.tls.implicit = true
    server.max-connections = 8192
    server.socket.backlog = 1024
    server.socket.nodelay = true
    server.socket.reuse-addr = true
    server.socket.reuse-port = true
    storage.blob = "postgres"
    storage.data = "postgres"
    storage.directory = "internal"
    storage.fts = "postgres"
    storage.lookup = "postgres"
    store.postgres.compression = "lz4"
    store.postgres.database = "stalwart"
    store.postgres.host = "postgres-postgresql.default.svc.cluster.local"
    store.postgres.pool.max-connections = 10
    store.postgres.port = 5432
    store.postgres.purge.frequency = "0 3 *"
    store.postgres.read-from-replicas = true
    store.postgres.timeout = "15s"
    store.postgres.tls.allow-invalid-certs = false
    store.postgres.tls.enable = false
    store.postgres.type = "postgresql"
    store.postgres.user = "stalwart"
    tracer.log.ansi = false
    tracer.log.enable = true
    tracer.log.level = "info"
    tracer.log.path = "/opt/stalwart-mail/logs"
    tracer.log.prefix = "stalwart.log"
    tracer.log.rotate = "daily"
    tracer.log.type = "console"
    metrics.prometheus.enable = true
    certificate."certmanager".cert = "%{file:/opt/stalwart-mail/tls/tls.crt}%"
    certificate."certmanager".private-key = "%{file:/opt/stalwart-mail/tls/tls.key}%"
    certificate."certmanager".default = true

Version is 0.11.8.

How can we reproduce the problem?

I wish I could tell you how to reproduce this. The only thing that seems relevant is that I create a new account, which ended up somehow being listed an Individual under the Type column in the domain listed. Feeling weirded out, I deleted that account and here I am. That account had an email in the domain that is now erroring out.

Version

v0.11.x

What database are you using?

PostgreSQL

What blob storage are you using?

PostgreSQL

Where is your directory located?

Internal

What operating system are you using?

Linux

Relevant log output

Logs have only INFO lines, nothing immediately obviously related to what I report here.

Code of Conduct

• I agree to follow this project's Code of Conduct

[akosiaris]

Yes, but Postgres is not being used as a directory per

directory.internal.type = "internal"

[stiivo]

Yes, but Postgres is not being used as a directory per

directory.internal.type = "internal"

Sorry, i was too fast.

[akosiaris]

Adding some more information. Enabling debug logging and navigating to /manage/directory/domains spews this in the logs

2025-05-07T13:37:47Z DEBUG HTTP request URL (http.request-url) listenerId = "http", localPort = 8080, remoteIp = 10.244.179.72, remotePort = 57234, remoteIp = XXX.XXX.XXX.XXX, url = "/api/principal?limit=10&page=1&types=domain&"
2025-05-07T13:37:47Z DEBUG Managed resource not found (manage.not-found) listenerId = "http", localPort = 8080, remoteIp = 10.244.179.72, remotePort = 57234, key = "uname.gr"

I can reproduce with a curl call

$ curl 'https://<domain>/api/principal?limit=10&types=individual&page=1&' -H 'authorization: Bearer <redacted>'
{"error":"notFound","item":"uname.gr"}

Varying the types parameter in the curl request, e.g. using individual, list, group returns results just fine.

[mdecimus]

Perhaps you already have another principal called uname.gr? Even if it is not a domain, principal names have to be unique across types.

[akosiaris]

Not that I can see of. I don't have any groups, mailing lists, roles, tenants, api keys, or auth clients. Just accounts and domains. If there is some way to dump all principals, e.g. via the CLI, happy to share a listing. I did try to get that info via just hitting /api/principal without any query parameters with a cURL call, but same error.

[akosiaris]

Trying to debug this a bit more, I started trying the /api/principal/<principal> looking into the various principals I had created. The principal before all of this happened was named devnull. The API returned this

{
  "data": {
    "id": 31,
    "type": "domain",
    "name": "devnull",
    "quota": 0,
    "members": 0
  }
}

On a hunch, I delete this principals using a curl -X DELETE call to the /api/principal/devnull endpoint (using admin bearer token).

And the error has now changed. Instead of {"error":"notFound","item":"uname.gr"} I now get {"error":"notFound","item":""}. I continue receiving/sending emails just fine for what is worth. This is midly dumbfounding to me to be honest.

[akosiaris]

I should note I am pretty sure I had created that principal as an account, not a domain

[akosiaris]

That's fine, thanks for circling back on this.

No, I haven't been able to resolve it. I am unsure what other path to pursue to be honest. I ve tried making sense of the database schema and while I have now a basic grasp, I am unable to figure out if something is corrupted in the d (directory per my reading of

stalwart/crates/store/src/lib.rs

Line 121 in c19d2ce

pub const SUBSPACE_DIRECTORY: u8 = b'd';

)

Any ideas about what other avenue to pursue would be appreciated.

[mdecimus]

The database schema has been recently documented and, if you want to explore the directory subspace, you can use the database console.

Also, if you quickly start a Stalwart instance with RocksDB, can you consistenly reproduce the issue?

[akosiaris]

This has been very helpful and has allowed me to partially reproduce the issue in a clean installation with rocksdb, meaning that this isn't specific to PostgreSQL.

Instructions. Note that I use podman here, but it really doesn't matter whether podman, docker, containerd or otherwise.

$ mkdir /tmp/stalwart ; cd /tmp/stalwart
$ podman run --replace -it -p 8080:8080 -v `pwd`:/opt/stalwart-mail --name stalwart docker.io/stalwartlabs/mail-server:v0.11.8
✅ Configuration file written to /opt/stalwart-mail/etc/config.toml
🔑 Your administrator account is 'admin' with password 'isGq23LTCw'.
Hit ctrl-C
$ podman run --replace -it -p 8080:8080 -v `pwd`:/opt/stalwart-mail --name stalwart --entrypoint /bin/bash docker.io/stalwartlabs/mail-server:v0.11.8
root@4374f9701dd0:/opt/stalwart-mail# stalwart-mail --config /opt/stalwart-mail/etc/config.toml --console
Stalwart Mail Server v0.11.8 Data Store CLI

Enter commands (type 'help' for available commands).
> put d\x00foo.bar \x1b\x07
Putting key: [100, 0, 102, 111, 111, 46, 98, 97, 114]
> exit
Exiting...
root@4374f9701dd0:/opt/stalwart-mail# exit
$ podman run --replace -it -p 8080:8080 -v `pwd`:/opt/stalwart-mail --name stalwart docker.io/stalwartlabs/mail-server:v0.11.8

Navigate to http://localhost:8080/manage/directory/domains and witness the error.

Now as to why that key ended up with the \x1b\x07 value, I don't know, but it is what my domain has in the environment where I experience this. Creation of random other domains has not yet ended up in an entry with that value but I also haven't yet made sense of the the value itself.

My thanks again for circling back on this!

[mdecimus]

What I meant was finding a way to reproduce the issue through the webadmin or REST API. If you use the database CLI you can easily corrupt any kind of entry :-)

[akosiaris]

Agreed, this is why I said "partially" and pointed out I don't know how the corruption happened. But you got me unstuck and now I know at least what's corrupted, something I didn't know before. I 'll see what I can do to figure out how that value got corrupted.

I think there's also a bug here in the form of "If a single principal's value is corrupted, it is impossible to list all the other principals of the same type".

[akosiaris]

I ve upgraded my Postgres database by a dump and restore. And... The problem is no more. I have no idea what on earth, I am gonna give up on this one. Thanks for all your help.