Another DMS GUI #4728
Replies: 1 comment · 6 replies
|
what's the problem in relying on setup.sh? I was told it's fine as you don;t have to reinvent the wheel. is you goal to become a standalone ui that can be attached to any other mailserver project? |
All reactions
-
👍 1
Compatibility issues can also arise if the upstream (DMS) makes breaking changes (underlying format) that are handled in a non-breaking manner by it's first-party support transparently (such as the Just an example, given no such changes have happened for quite some time (it's planned in the future if ever refactoring to offer a proper API service for integration). I do understand the reasoning to bypass going through first-party, but compatibility can break both ways 😅 As for speed, you'd need to be doing a large amount of operations or some blocking call for that to be valid. Most configuration modifications through our CLI should be fairly quick, but applying updates can depend upon the change detection service (of which LDAP is exempt of for accounts vs file based account provisioner which is where your issue is probably actually at). Change Detection polls at a fixed rate on monitored config files, and if any have been updated, it'll sync those over to the actual internal service config locations (where DMS assumes full authority to manage and thus users should be cautious to directly manipulate). LDAP has some subtle caveats IIRC, DMS has lacked maintainers for the support and getting such issues resolved has been slow. Just something to be mindful about, which may not be apparent early on.
You can do this when integrating into DMS to call |
All reactions
-
❤️ 1
|
I'll be upfront, I don't think you are correct. Appreciate the help though! I'll try to break it down as much as I understand it, I very much welcome to be corrected when I am wrong. 1. The setup script is inherently slow, listing emails takes literally a second if your mailserver has more than 10 adresses & aliases. This is fact. As you said, the LDAP side of DMS lacks maintainers to get issues fixed. At the risk of sounding arrogant - it doesn't seem like much will change on the LDAP side soon. Maybe I'll find some edge cases in DMS and try to fix them myself - I don't mind jumping into the deep end to contribute if it'll make my implementation work or if I can see a general improvement. If you'd ever want to implement a proper API service, my HTTP/LDAP server would be able to run directly in the DMS container (at which point it could also just use the SSL certs provided to DMS). Listening on a unix socket would be a one line change. Implementing an API for an already existing web backend from that point on would be trivial. |
All reactions
-
👍 1
I see that you have a view that appears to be reproducing this line of information with references like If you had a valid complaint of a significant amount of time, I'd ask you to be a bit more specific with the account vs alias count... "literally a second" isn't excessive for the intended DMS audience (not enterprise/SaaS levels of accounts which would not be suited for the file-based provisioner we have) interacting with a CLI command. I have no doubt that the operation can be done more efficiently, but let's be clear:
Doing the equivalent for the file-based provisioner is totally viable, but the legacy file formats we have are not ideal/convenient, whilst a breaking change away from that needs to take the existing user base into consideration for migration.
Sorry, my previous comment was regarding using Yes you're right with LDAP it's a bit different, as the source of accounts and aliases is no longer the responsibility of DMS, you've instead outsourced to LDAP. In that regard if you're not interacting with DMS, then any breaking changes to DMS related to our LDAP integration is irrelevant to your project being compatible, it'd be the same as with any other LDAP user needing to adjust DMS. NOTE:
Basic overview:
The collapsed content I had typed up earlier, but I think the above covers this point well enough. Original reply (collapsed for brevity)Yes... DMS handles this integration on your behalf so you don't run into issues like occurred here when changes need to be made and mistakes are made by those that don't understand this area of the project. I'm one of the very few that has contributed in DMS that helps resolve such problems when reported, or catches changes in PR reviews that the other maintainers would not recognize can unintentionally break or diverge support with LDAP for example. By relying on someone like me to prevent LDAP support getting worse (which it had done over the years prior to my involvement accumulating tech debt from lack of maintenance support), that is very much like relying on our DMS may just be integration of these third-party services like Postfix and Dovecot, with some convenience in config management, but hopefully you can see from the linked PR that it's not always as simple as it seems 😓 (granted if you only have yourself as the deployment to be concerned about, and with the availability of AI assistance, these days you can probably do quite fine managing it on your own, but you might be delegating some trust to AI to do things properly/securely, which tbh it doesn't always deliver on as I've provided support to users numerous times that relied upon AI tooling that failed them). I did mention our LDAP support has caveats btw, and that you've likely not run into them just yet (it really depends what features a user of DMS utilizes). It should not impact your project specifically, but I'm just making you aware that LDAP was a community contributed feature to DMS long ago, it is not as well tested and depending on how a user configures it on their end, there can be some issues. The default file-based provisioner support on the other hand is in better shape.
As noted in the first point you raised, no doubt such operations can be done faster but for a typical DMS deployment I don't perceive 1 second as slow 🤷♂️ When scaling, there are worse issues tied to the change detection triggers that are known to be slow due to naive processing when a single account or alias is added/modified/deleted. That concern would be justified, and switching to an alternative provisioner such as LDAP avoids that problem. FWIW, timing of such processing can be dependent on the environment itself. I get less than 400ms for DMS running `compose.yaml`services:
dms:
image: ghcr.io/docker-mailserver/docker-mailserver:15.1
hostname: mail.example.test
configs:
- source: dms-accounts
target: /tmp/docker-mailserver/postfix-accounts.cf
- source: dms-aliases
target: /tmp/docker-mailserver/postfix-virtual.cf
configs:
dms-accounts:
content: |
john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
john.doe-1@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
john.doe-2@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
john.doe-3@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
john.doe-4@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
john.doe-5@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
john.doe-6@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
john.doe-7@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
john.doe-8@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
john.doe-9@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
dms-aliases:
content: |
john@example.test john.doe@example.test
hello@example.test john.doe@example.test
world@example.test john.doe@example.test
hello-world@example.test john.doe@example.test
1@example.test john.doe@example.test
2@example.test john.doe@example.test
3@example.test john.doe@example.test
x@example.test john.doe@example.test
y@example.test john.doe@example.test
z@example.test john.doe@example.test$ docker compose up -d --force-recreate
$ docker compose exec dms bash
# Give it a moment to configure and start DMS:
$ time setup email list
* john.doe@example.test ( 0 / ~ ) [0%]
[ aliases -> john@example.test, hello@example.test, world@example.test, hello-world@example.test, 1@example.test, 2@example.test, 3@example.test, x@example.test, y@example.test, z@example.test ]
* john.doe-1@example.test ( 0 / ~ ) [0%]
* john.doe-2@example.test ( 0 / ~ ) [0%]
* john.doe-3@example.test ( 0 / ~ ) [0%]
* john.doe-4@example.test ( 0 / ~ ) [0%]
* john.doe-5@example.test ( 0 / ~ ) [0%]
* john.doe-6@example.test ( 0 / ~ ) [0%]
* john.doe-7@example.test ( 0 / ~ ) [0%]
* john.doe-8@example.test ( 0 / ~ ) [0%]
* john.doe-9@example.test ( 0 / ~ ) [0%]
real 0m0.386s
user 0m0.144s
sys 0m0.129s
As pretty much nobody else tackles such but me and my time is very limited, you'd be right 😓
I'm aware of a few with LDAP, I've documented them on the issue tracker or in ad-hoc github discussions. There's also this large list of tasks I add to, but as might be evident, never get around to upstreaming solutions :( I'd appreciate more contributors, but the PRs really need to be minimal with easy to verify reproductions, otherwise it delegates burden onto me as a reviewer to get that confidence to approve 😅 Presently I'm barely able to allocate time to DMS for anything in-depth (like the Debian 13 + Dovecot 2.4 upgrade PR), so if anything the project needs more maintainers but that's been a struggle to find anyone interested in such.
There's a separate repo (inactive) for those wanting to progress an official API in DMS. General stance has been to defer to community and if a community project has had a decent amount of adoption, we could look at officially supporting that or as a reference. You should not need to rely upon LDAP though for account management API support however. We could just as easily go with a DB instead if that were relevant, but since the file provisioner is the default and widely adopted, that's what should be supported out of the box. I'm well aware of how to implement these services, and like mentioned have shared a simple example that used a reverse proxy (Caddy) with a config to map an HTTP API to the # Start-up DMS
$ docker compose up --force-recreate -d
# Add `jane.doe@example.test` via API calling `setup email add ...` from a different container:
$ docker compose run --rm --quiet my-dms-ui curl -fsS -X POST --unix-socket /var/run/dms-api/api.sock \
'http://localhost/api/emails/?address=jane.doe@example.test&secret=some-password'
# List the accounts to verify new account was added:
$ docker compose run --rm --quiet my-dms-ui curl -fsS --unix-socket /var/run/dms-api/api.sock \
'http://localhost/api/emails/'
* john.doe@example.test ( 0 / ~ ) [0%]
* jane.doe@example.test ( 0 / ~ ) [0%]That endpoint is rather simple in handle_path /api/emails/ {
map {method} {subcommand} {cli-args} {
GET "list"
POST "add" "{query.address} {query.secret}"
PUT "update" "{query.address} {query.secret}"
DELETE "del" "{query.options} {query.address}"
}
cgi * /opt/dms-api/bin/proxy-to-cli "email {subcommand}" {cli-args}
}As the link shows, a separate There's definitely better approaches, such as refactoring our file provisioner, and removing the need to route through an executable like our So it's not that implementing the solution is a problem. Like with most things in DMS it's just a lack of contributor time and more importantly reviewer time/expertise. I can only afford so much time, but try to chime in on reviews to block on any concerns as in the past many low quality PRs had been approved and merged which burdened maintaining DMS. The DMS project simply needs more trusted contributors/maintainers 😓 |
All reactions
|
Oh god, I feel like we're going to be tossing words a lot... I don't mind if you wont! ;)
This was populated from the setup script itself. Setup email list also lists the aliases and the quota. I have now stopped using this, the project is going through a lot while stepping away from setup cli. I'd recommend not trying it for a bit haha. I completely agree with you though! Didn't mean to bash DMS in any way, the setup script is perfect for what it is intended for. That just isn't a HTTP server.
Thaaaank you for the heads up! Prod is running static versioning, the mailserver-compose is just for quick prototyping.
This was ran on my local machine with 16 CPU cores.
The code creates 14 email addresses, each with 3 aliases. The slow part however is when the setup script has to retrieve said data and mend all aliases to each email to output it in email list (this is an assumption). But that does not take away from your point. Setup is a CLI utility, it isn't known for its' speed, nor needs it be.
Would tests help you regain some of that confidence? :) I've used DMS for a few years now, so I'm probably here to stay. I get it though. Don't forget to allocate some you time too. |
All reactions
I don't know why a cheap VPS with much less resources would handle a
There is more work for the aliases output with docker-mailserver/target/bin/listmailuser Lines 83 to 106 in e8be731
where docker-mailserver/target/bin/listalias Lines 11 to 18 in e8be731
However I'd still wager that calls to Dovecot are where added latency is probably coming from. Each account will call You could verify by ensuring the quota feature is disabled: docker-mailserver/target/bin/listmailuser Line 55 in e8be731
If you wanted to support such with LDAP, you'd have to likewise query Dovecot AFAIK, but I'm not sure if quota support has been wired up for LDAP within the DMS integration 🤷♂️ It does appear to be unsupported at present: docker-mailserver/target/scripts/startup/setup.d/dovecot.sh Lines 142 to 146 in e8be731
The feature is enabled by default IIRC to avoid a concern with becoming a backscatter source for spammers. Without it Postfix receives mail and accepts it, then delivers to Dovecot that then rejects it or something along those lines on the basis of the mailbox / system no longer having storage capacity to store the new mail. Postfix then acts on that by sending a delivery failure mail back to the return address which a spammer could spoof, effectively using DMS to send spam. With the file provisioner we had a related workaround for aliases to have dummy mailbox accounts in Dovecot to share the same quota, as IIRC when the mail is arriving at Postfix, it has not reached a stage that an alias would be resolved, but outright goes "is this a valid delivery recipient I should handle?" and if it is, it then checks other policies such as "Hey Dovecot, does this recipient address have full quota, or does their mailbox have enough quota spare to accept it?", but for LDAP no such support is implemented (presumably an oversight when the feature landed and lack of LDAP maintenance / familiarity at the time to keep it at feature parity). I know there's a few disparities like that with the LDAP support either not implemented, or not done correctly.
They do help yes! 😁 My experience however has been I am still burdened to grok the feature well enough (or better than the contributor), as the tests don't necessarily mean the contribution was implemented correctly, just that the tests added/changed alongside it pass 😓 (there's a PR open currently to improve The LDAP tests need to be revised IIRC, and I've had an open PR stalled since 2023 where I wanted to overhaul the LDAP config approach. For LDAP, to rewrite the docs properly for DMS I have to allocate time to grok our integration again and the caveats we had, and understand LDAP well enough for testing and troubleshooting, which I recall was a necessity as there was different configurations supported where LDAP auth was inconsistent. Presently, you'd want to wait until DMS v16 (or at least the relevant PR is merged). The test suite needs to finish migrating to the convention set out in 2023, and then from there the plan was to migrate to Docker Compose and bring in proper DNS records. LDAP itself had it's tests switch container images due to the prior image not being maintained, but Bitnami since ceased publishing a free tier LDAP image IIRC, and the prior image we had been using I believe is actively being maintained again in 2026 🤷♂️ (setup and configuring LDAP isn't my thing, I did some tidy up with the LDIF files we use for provisioning the LDAP service, but it can probably be done better) Related to that is our tests using DMS is for the most part maintained, but addressing these long standing concerns is more of an issue with me as a blocker AFAIK. Getting active contributors to help move these improvements along in DMS would be great, but if we're lacking the review bandwidth those contributions get stalled or slip through without proper quality control (which is how we ended up with a variety of bugs in the past). Mail servers are boring and rarely a topic of interest for contributors to know more than they need beyond "works for me" solutions.
I was burning out around 2023 IIRC, but since improved on when to call it a day. That just hasn't helped with my OSS backlog over time 😆 Anyway, back on topic. I agree that a proper API would be great to have instead of our shell scripts with the
No promises that DMS would adopt a Go solution, but if the codebase is easy to grok and maintain then there's a possibility in future. I'd personally pursue a Rust solution, but that could be bias 😝 |
All reactions
-
❤️ 1
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
All the other ones are either slow, use the setup script as a source of truth or are written with a JS backend or by AI. Ew.
Not to mention the command injection vulns I'm seeing, yeesh!
Let's change it up!
A backend implementation for DMS with an internal LDAP server, able to connect to local or distributed databases for account management, written in sweet, sweet Golang.
There's a lot on my to-do list, and the project has just gotten started. Fun!
If you have any ideas or suggestions, feel free to drop in.
The project does not currently hold a license, and I do not want contributions to it, for now at least.
When it is finished, the project will be provided with an MIT license.
docker-mailserver-mailman
All reactions