Running ejabberd through homebrew services

[mull]

UPDATE

Short term workaround (from #3343 (comment)):

brew services stop ejabberd
cd ~
# You might need to confirm that this is really what you want to do
rm .erlang.cookie 
cp /usr/local/var/lib/ejabberd/.erlang.cookie .
brew services start ejabberd
# The following should now work
ejabberdctl start

Environment

• ejabberd version: ejabberd: stable 20.04 (bottled), HEAD
• Erlang version: Erlang (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 10.7.2.2
• OS: MacOS 10.15.6 (Catalina)
• Installed from: homebrew

Configuration (only if needed): grep -Ev '^$|^\s*#' ejabberd.yml

hosts:
  - localhost
loglevel: info
listen:
  -
    port: 5222
    ip: "::"
    module: ejabberd_c2s
    max_stanza_size: 262144
    shaper: c2s_shaper
    access: c2s
    starttls_required: true
  -
    port: 5269
    ip: "::"
    module: ejabberd_s2s_in
    max_stanza_size: 524288
  -
    port: 5443
    ip: "::"
    module: ejabberd_http
    tls: true
    request_handlers:
      /admin: ejabberd_web_admin
      /api: mod_http_api
      /bosh: mod_bosh
      /captcha: ejabberd_captcha
      /upload: mod_http_upload
      /ws: ejabberd_http_ws
  -
    port: 5280
    ip: "::"
    module: ejabberd_http
    request_handlers:
      /admin: ejabberd_web_admin
      /.well-known/acme-challenge: ejabberd_acme
  -
    port: 3478
    transport: udp
    module: ejabberd_stun
    use_turn: true
  -
    port: 1883
    ip: "::"
    module: mod_mqtt
    backlog: 1000
s2s_use_starttls: optional
acl:
  local:
    user_regexp: ""
  loopback:
    ip:
      - 127.0.0.0/8
      - ::1/128
access_rules:
  local:
    allow: local
  c2s:
    deny: blocked
    allow: all
  announce:
    allow: admin
  configure:
    allow: admin
  muc_create:
    allow: local
  pubsub_createnode:
    allow: local
  trusted_network:
    allow: loopback
api_permissions:
  "console commands":
    from:
      - ejabberd_ctl
    who: all
    what: "*"
  "admin access":
    who:
      access:
        allow:
          acl: loopback
          acl: admin
      oauth:
        scope: "ejabberd:admin"
        access:
          allow:
            acl: loopback
            acl: admin
    what:
      - "*"
      - "!stop"
      - "!start"
  "public commands":
    who:
      ip: 127.0.0.1/8
    what:
      - status
      - connected_users_number
shaper:
  normal: 1000
  fast: 50000
shaper_rules:
  max_user_sessions: 10
  max_user_offline_messages:
    5000: admin
    100: all
  c2s_shaper:
    none: admin
    normal: all
  s2s_shaper: fast
modules:
  mod_adhoc: {}
  mod_admin_extra: {}
  mod_announce:
    access: announce
  mod_avatar: {}
  mod_blocking: {}
  mod_bosh: {}
  mod_caps: {}
  mod_carboncopy: {}
  mod_client_state: {}
  mod_configure: {}
  mod_disco: {}
  mod_fail2ban: {}
  mod_http_api: {}
  mod_http_upload:
    put_url: https://@HOST@:5443/upload
  mod_last: {}
  mod_mam:
    assume_mam_usage: true
    default: always
  mod_mqtt: {}
  mod_muc:
    access:
      - allow
    access_admin:
      - allow: admin
    access_create: muc_create
    access_persistent: muc_create
    access_mam:
      - allow
    default_room_options:
      mam: true
  mod_muc_admin: {}
  mod_offline:
    access_max_user_messages: max_user_offline_messages
  mod_ping: {}
  mod_privacy: {}
  mod_private: {}
  mod_proxy65:
    access: local
    max_connections: 5
  mod_pubsub:
    access_createnode: pubsub_createnode
    plugins:
      - flat
      - pep
    force_node_config:
      storage:bookmarks:
        access_model: whitelist
  mod_push: {}
  mod_push_keepalive: {}
  mod_register:
    ip_access: trusted_network
  mod_roster:
    versioning: true
  mod_s2s_dialback: {}
  mod_shared_roster: {}
  mod_stream_mgmt:
    resend_on_timeout: if_offline
  mod_stun_disco: {}
  mod_vcard: {}
  mod_vcard_xupdate: {}
  mod_version:
    show_os: false

Errors from error.log/crash.log

emil@Emils-MacBook-Pro ejabberd % cat error.log
2020-07-27 17:10:26.299074+02:00 [error] <0.556.0> ** Connection attempt from disallowed node '487-ctl-ejabberd@localhost' **

Bug description

Using brew services start ejabberd followed by ejabberctl status gives me the following output:

Failed RPC connection to the node ejabberd@localhost: nodedown

I've added my hostname (Emils-MacBook-Pro.local) in /private/etc/hosts as described in the formula:

127.0.0.1	        localhost     Emils-MacBook-Pro.local
::1               localhost     Emils-MacBook-Pro.local

I can run ejabberd by using ejabberdctl, but not through the brew services. Since homebrew seems to be the official recommendation for installation, perhaps we can figure out what's going on here and fix it (or mention it) in the formula?

The documentation seems to (?) recommend using ejabberdctl for running the server, but when you install it from homebrew you get the impression that I can be managed by brew services. Is there a technical reason why that can't be done, or is it just a bug that we can fix?

ejabberdctl live gives a hint at the problem, possibly? It might be related to the hostname resolution... I've silenced that by setting turn_ip to 127.0.0.1, but it doesn't seem to be helping. I just loose the warning in the live output, but the same connection problem (nodedown) appears.

Erlang/OTP 22 [erts-10.7.2.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe] [dtrace]

Eshell V10.7.2.2  (abort with ^G)
(ejabberd@localhost)1> 2020-07-27 17:18:39.941821+02:00 [info] Loading configuration from /usr/local/etc/ejabberd/ejabberd.yml
2020-07-27 17:18:40.143680+02:00 [info] Configuration loaded successfully
2020-07-27 17:18:40.369435+02:00 [info] Loading modules for localhost
2020-07-27 17:18:40.384235+02:00 [warning] Mnesia backend for mod_mam is not recommended: it's limited to 2GB and often gets corrupted when reaching this limit. SQL backend is recommended. Namely, for small servers SQLite is a preferred choice because it's very easy to configure.
2020-07-27 17:18:40.425696+02:00 [info] Won't auto-announce STUN/TURN service with loopback address: 127.0.0.1:3478 (udp), please specify a public 'turn_ip'
2020-07-27 17:18:40.526822+02:00 [info] Building MQTT cache for localhost, this may take a while
2020-07-27 17:18:40.567646+02:00 [info] Waiting for Mnesia synchronization to complete
2020-07-27 17:18:40.711212+02:00 [warning] No certificate found matching localhost
2020-07-27 17:18:40.711381+02:00 [warning] No certificate found matching conference.localhost
2020-07-27 17:18:40.711544+02:00 [warning] No certificate found matching upload.localhost
2020-07-27 17:18:40.711673+02:00 [warning] No certificate found matching proxy.localhost
2020-07-27 17:18:40.711803+02:00 [warning] No certificate found matching pubsub.localhost
2020-07-27 17:18:40.711907+02:00 [info] ejabberd 20.04 is started in the node ejabberd@localhost in 0.82s
2020-07-27 17:18:40.712103+02:00 [info] Start accepting UDP connections at 0.0.0.0:3478 for ejabberd_stun
2020-07-27 17:18:40.712401+02:00 [info] Start accepting TCP connections at [::]:5222 for ejabberd_c2s
2020-07-27 17:18:40.712439+02:00 [info] Start accepting TCP connections at [::]:5269 for ejabberd_s2s_in
2020-07-27 17:18:40.712665+02:00 [info] Start accepting TCP connections at [::]:5280 for ejabberd_http
2020-07-27 17:18:40.712656+02:00 [info] Start accepting TLS connections at [::]:5443 for ejabberd_http
2020-07-27 17:18:40.712786+02:00 [info] Start accepting TCP connections at [::]:1883 for mod_mqtt
2020-07-27 17:18:40.712793+02:00 [info] Start accepting TCP connections at 127.0.0.1:7777 for mod_proxy65_stream
2020-07-27 17:18:40.762404+02:00 [warning] Option 'turn_ip' is undefined and the server's hostname doesn't resolve to a public IPv4 address, most likely the TURN relay won't be working properly

[mull]

Please redirect me if you consider this a homebrew problem rather than ejabberd related. Since homebrew is now the recommended version of installing it, I figure this is the right place.

[mremond]

Hi,

I think the issue is that when started as a service, ejabberd will be started with a different user and thus will use a different Erlang cookie.
I will check if this is possible to clean this behaviour.

[prefiks]

Hello,

Could you check if using 'sudo ejabberdctl ' helps here?

Generally you need to run that command as user that ejabberd server is using (to have permission to talk to service), but when running it as root that script will change to ejabberd user for you. I am guessing brew start service using different user than what you are using.

[mull]

Sudo doesn't help, unfortunately. I thought the same thing could have been happening. ps aux | grep ejabber produces:

emil@Emils-MacBook-Pro ~ % ps aux | grep ejabber
emil              7509   0.1  0.3  5507724  51288   ??  S    10:35AM   0:04.13 /usr/local/Cellar/erlang@22/22.3.4.4/lib/erlang/erts-10.7.2.2/bin/beam.smp -K true -P 250000 -- -root /usr/local/Cellar/erlang@22/22.3.4.4/lib/erlang -progname erl -- -home /usr/local/var/lib/ejabberd -- -sname ejabberd@localhost -smp auto -mnesia dir "/usr/local/var/lib/ejabberd" -s ejabberd -noshell -noinput --
emil              7516   0.0  0.0  4272740    968   ??  Ss   10:35AM   0:00.00 /usr/local/Cellar/ejabberd/20.04_1/lib/eimp-1.0.14/priv/bin/eimp
emil              7515   0.0  0.0  4272740    976   ??  Ss   10:35AM   0:00.01 /usr/local/Cellar/ejabberd/20.04_1/lib/eimp-1.0.14/priv/bin/eimp

That's ejabberd running through brew services start ejabberd. This is the output of sudo ejabberdctl status

emil@Emils-MacBook-Pro ~ % sudo ejabberdctl status
Password:
WARNING: It is not recommended to run ejabberd as root
Failed RPC connection to the node ejabberd@localhost: nodedown

However.. the --root part above made me think that I'm looking at the wrong process. Curiously, launchctl list | grep ejabber gives me this:

emil@Emils-MacBook-Pro ~ % launchctl list
PID	Status	Label
-	4	homebrew.mxcl.ejabberd

So... maybe there's really nothing running, huh?

emil@Emils-MacBook-Pro ~ % ejabberdctl start
ERROR: The ejabberd node 'ejabberd@localhost' is already running.

I suspect I just don't understand the domain well enough.

Happy to provide further information, just let me know how I can help :) Meanwhile, I'll try to look for more logs, maybe there really is an error message somewhere, but not exactly where we might expected it.

[mull]

That "status 4" indicates that the process was shut down:

# cat /var/log/system/log

Jul 28 11:07:44 Emils-MacBook-Pro com.apple.xpc.launchd[1] (homebrew.mxcl.ejabberd[9881]): Service exited with abnormal code: 4

Not a reason for the exit, but it is a start :)

[prefiks]

This command in using erlang interprocess communication protocol, and to make two node be able to talk to each one, they must share common cookie file, looks like calling that command through brew uses different cookie than when you are calling this directly. Probably something need to be changed in brew service startup process, or have some command that will make it use brew cookie.

[mull]

I'm unfortunately not very familiar with Erlang, and it's the first time I hear about Erlang cookies. Is there a relatively straightforward way for me to find out what the path to the cookie ejabberctl is using? I can tell which one the service uses by looking at the plist file

# cat ~/Library/LaunchAgents/homebrew.mxcl.ejabberd.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>EnvironmentVariables</key>
  <dict>
    <key>HOME</key>
    <string>/usr/local/var/lib/ejabberd</string>
  </dict>
  <key>Label</key>
  <string>homebrew.mxcl.ejabberd</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/opt/ejabberd/sbin/ejabberdctl</string>
    <string>start</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>WorkingDirectory</key>
  <string>/usr/local/var/lib/ejabberd</string>
</dict>
</plist>

The directory /usr/local/var/lib/ejabberd contains an .erlang.cookie file, so that's probably all good. Question is, which one does ejabberctl use? I'm not sure how to figure that out.

[mull]

Well actually.. maybe it's not cookie related. If the service has a status of 4 (meaning it's down), it's because ejabberd isn't starting at all. Let me see if I can enable a more useful logging level to get more information out of it.

[mull]

I think your suspicions are right that this is user account related. Unfortunately I don't know how to proceed on that, but I'm happy to assist & test your ideas

[mremond]

@mull In the short term, you should ensure that the content of the file /usr/local/var/lib/ejabberd/.erlang.cookie is the same than ~/.erlang.cookie. You can copy the first one to your home directory.

[mull]

I can confirm that the short term workaround works! I'll edit the issue for the others who might arrive here. Thank you! If you need help validating the outcome of #3346 let me know when it's ready and I'll give it a try :)

[mremond]

It should be fixed when ejabberd 20.07 is released on homebrew: Homebrew/homebrew-core#58859

[mull]

Thank you for the fix, I can confirm that it works. The only difference in our setup seems to be that the ejabberd_auth_http module comes preinstalled now, so we switched module_install to module_upgrade. Thanks again :)