Unable to compile latest ejabberd for the Elixir project

[Relrin]

Environment

• ejabberd version: 23.10
• Erlang version: Erlang (SMP,ASYNC_THREADS) (BEAM) emulator version 14.2
• OS: Linux (Debian) as Docker image
• Installed from: Hex.pm

Bug description

On an attempt to use a ejabberd as a dependency in the Elixir project, the Mix stops compiling with the following error:

==> ejabberd
Compiling 282 files (.erl)
src/ejabberd_auth.erl:53:14: can't find include lib "xmpp/include/scram.hrl"
%   53| -include_lib("xmpp/include/scram.hrl").
%     |              ^

could not compile dependency :ejabberd, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile ejabberd --force", update it with "mix deps.update ejabberd" or clean it with "mix deps.clean ejabberd"

Steps to reproduce:

1. Create a Dockerfile that uses a Elixir 1.16 (or 1.15). In my case the image was pulled from DockerHub and has pre-installed dev dependencies, like this:

FROM elixir:1.16-slim

RUN apt-get update && apt-get install -y git build-essential libyaml-dev zlib1g-dev libssl-dev libexpat1-dev

COPY src /app
WORKDIR /app

2. Create a new project with the Mix command: mix new ejapp
3. Add ejabberd as a dependency for the project

defmodule Ejapp.MixProject do
...
  defp deps do
    [
     {:ejabberd, "~> 23.10"}
    ]
  end
end

4. Run the mix do deps.get, compile command and see the output.

I also tried to use older versions of ejabberd, but still had no success in compiling it without issues. Initially I was following the official documentation, and the only difference is the used version.

[licaon-kter]

Can you try with HEAD instead?

[Relrin]

Yes, I tried with pulling the dependency from git, but got exactly the same error output

[licaon-kter]

So you did:

• git checkout a57bdfffb715abd1334045bf7b33c113c1b46ca0
• ./autogen.sh
• ./configure

Then tried what?

[Relrin]

In my case, in the project dependencies, was done just a simple change:

defp deps do
    [
      {:ejabberd, git: "https://github.com/processone/ejabberd.git", ref: "a57bdfffb715abd1334045bf7b33c113c1b46ca0"},
    ]
  end

did a clean up, and rebuild the app via the mix do deps.get, compile. With this setup was retrieved the same error as was previously.

So you did:

• git checkout a57bdfffb715abd1334045bf7b33c113c1b46ca0
• ./autogen.sh
• ./configure

Then tried what?

I haven't done it previously. However, I did as soon as this was suggested:

1. Because I pulled the dependency already on this commit, I just switched to the /app/deps/ejabberd directory
2. Executed the ./autogen.sh and it didn't output anything
3. For the ./configure command there was the following output:

checking whether make sets $(MAKE)... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking for a sed that does not truncate output... /bin/sed
checking for erl... /usr/local/bin/erl
checking for erlc... /usr/local/bin/erlc
checking for epmd... /usr/local/bin/epmd
checking for erl... /usr/local/bin/erl
checking for erlc... /usr/local/bin/erlc
checking for Erlang/OTP root directory... /usr/local/lib/erlang
checking for escript... /usr/local/lib/erlang/bin/escript
checking for make... make
checking Erlang/OTP version... ok
configure: creating ./config.status
config.status: creating Makefile
config.status: creating vars.config

4. Went to the root app directory, executed the mix do deps.get, compile command. It pulled the stun dependency, ran a compilation step. However, as a result it didn't finish with a success: got the the same error

[badlop]

Aha! You are right, recent versions of ejabberd have the problem you described. It seems nobody used that possibility recently... Thanks for reporting it!

I've found the problem, fixed this and many other small problems. I've committed those preliminary fixes to a branch in my fork, it would be great if you can test it, check it really works, and report any other problems.

---

It works for me now following this tutorial:
https://www.process-one.net/blog/how-to-use-ejabberd-as-an-elixir-application-dependency/

To use my fork with the proposed fixes, in step 3 setup this in the file mix.exs:

  defp deps do
    [
     {:ejabberd, git: "https://github.com/badlop/ejabberd", branch: "elixir-all-in"}
    ]
  end

Following that tutorial, ejabberd and all dependencies are compiled and start correctly.

---

The default ejabberd configuration in the file config/ejabberd.yml does not have any SSL certificates configured, but requires encryption for clients to login. So, to login with a client, either setup some certificates or disable starttls requirement:

starttls_required: false

---

Stun is now enabled by default, no need to run configure inside ejabberd. In any case, if you want to enable some features, it is possible:

In config/config.exs add the new option vars_config_path, for example:

config :ejabberd,
  vars_config_path: '../../config/vars.config',
  file: "config/ejabberd.yml",
  log_path: 'logs/ejabberd.log'

and write a new file config/vars.config with the options you want to enable or disable, for example:

{odbc, true}.
{mssql, false}.
{mysql, true}.
{pgsql, true}.
{sqlite, true}.
{pam, true}.
{zlib, true}.
{redis, true}.
{stun, true}.
{sip, true}.
{lua, true}.

---

There are still some warnings during compilation that I couldn't solve. Don't worry, ejabberd works correctly:

src/mod_mqtt.erl:19:2: Warning: behaviour p1_server undefined
%   19| -behaviour(p1_server).

[Relrin]

Thank you for your response!

I tried to rebuild an Elixir application with settings mentioned above. And the changes work great to me.

However, right after it I faced with another issue on recompile the application with the --enable-pgsql feature flag. In this case, I did a minor change to the deps setup, like this:

defp deps do
    [
      {
        :ejabberd,
        git: "https://github.com/badlop/ejabberd",
        branch: "elixir-all-in",
        compile: "./autogen.sh && ./configure --enable-pgsql && make"
      },
    ]
  end

And then executed the mix deps.compile ejabberd --force command. It has compiled with no issues, but failed on further start up:

root@294808fe5e74:/app# iex -S mix
FORMATTER ERROR: bad return value
FORMATTER ERROR: bad return value
FORMATTER ERROR: bad return value
FORMATTER ERROR: bad return value
FORMATTER ERROR: bad return value
FORMATTER ERROR: bad return value

Which, I believe, might be related to the #4087 issue, right?

[badlop]

I faced with another issue on recompile the application with the --enable-pgsql feature flag.

As I said in my previous comment, no need to run ./configure. If you want to enable or disable some of its features like pgsql, create a vars.config file with your desired options and tell ejabberd to use it.

compile: "./autogen.sh && ./configure --enable-pgsql && make"

This doesn't seem a good idea, because then the dependencies are downloaded and compiled twice

FORMATTER ERROR: bad return value

I added a fix for this in badlop@615e764 and the fix requires to add --enable-elixir or add --with-rebar=mix

In summary: better use vars_config_path as I explained in the previous comment.

[badlop]

Were you able to finally compile ejabberd correctly?

[Relrin]

I tried today to rebuild an app with the mentioned solution (via vars_config_path), but didn't manage to make it running with no issues.

The setup is quite simple:

• config/config.exs

import Config

config :ejabberd,
  file: "config/ejabberd.yml",
  vars_config_path: "config/ejabberd_features.config",
  log_path: 'logs/ejabberd.log'

• config/ejabberd_features.config is basically adjusted set of feature flags:

{odbc, false}.
{mssql, false}.
{mysql, false}.
{pgsql, true}.
{sqlite, false}.
{pam, true}.
{zlib, true}.
{redis, false}.
{stun, true}.
{sip, true}.
{lua, false}.

And the mix.exs file has the following declared dependencies:

defp deps do
    [
      {
        :ejabberd,
        git: "https://github.com/badlop/ejabberd",
        branch: "elixir-all-in",
      },
    ]
  end

With the such setup, it compiles, but fails on the mix run command with the logs:

Generated messaging_service app
2024-01-08 18:09:07.723 [info] Loading configuration from config/ejabberd.yml
2024-01-08 18:09:08.379 [info] Configuration loaded successfully
2024-01-08 18:09:08.721 [info] Got no NOTIFY_SOCKET, notifications disabled
2024-01-08 18:09:09.576 [info] Loading modules for localhost
2024-01-08 18:09:09.705 [info] Going to offer STUN/TURN service: 172.19.0.3:3478 (udp)
2024-01-08 18:09:09.824 [critical] Failed to start Erlang application 'p1_pgsql': no such file or directory: p1_pgsql.app. This usually means that ejabberd or Erlang was compiled/installed incorrectly.

So, I tried to add append the p1_pgsql dependency (as the {:p1_pgsql, "~> 1.1"} line) to the mix.exs file, but it didn't work out for me. It failed on fetching the deps:

root@cddf9c00bcff:/app# mix deps.get
Resolving Hex dependencies...
Resolution completed in 0.039s
New:
  p1_pgsql 1.1.23
Unchanged:
  base64url 1.0.1
  cache_tab 1.0.30
  eimp 1.0.22
  ezlib 1.0.12
  fast_xml 1.1.49
  fast_yaml 1.0.36
  idna 6.0.0
  jiffy 1.1.1
  jose 1.11.5
  mqtree 1.0.15
  p1_acme 1.0.22
  p1_oauth2 0.6.11
  p1_utils 1.0.25
  pkix 1.0.9
  stringprep 1.0.29
  stun 1.2.10
  unicode_util_compat 0.4.1
  yconf 1.0.15
* Updating xmpp (Hex package)
** (Mix) The lock is missing for package xmpp. This could be because another package has configured the application name for the dependency incorrectly. Verify with the maintainer of the parent application

[badlop]

In my comment I showed:

config :ejabberd,
  vars_config_path: '../../config/vars.config',
  file: "config/ejabberd.yml",
  log_path: 'logs/ejabberd.log'

But you changed that to:

  vars_config_path: "config/ejabberd_features.config",

That option is used by deps/ejabberd/mix.exs and the value is a file that exists in config/vars.config, so ../../in the path is required.

[Relrin]

Right, you were correct. That was my mistake on the configuration side. Now I'm able to compile and run the application.

Thank you for the assistance! :)

[badlop]

Support for vars_config_path has been added in fa3c25a and another fix in 4daeb41