Elixir 1.15 sometimes fails during mix release due to missing module

[AndrewDryga]

Elixir and Erlang/OTP versions

elixir 1.15.2-otp-26
erlang 26.0.2

Operating system

Ubuntu 22.04.2

Current behavior

We started to have issues with mix release that fails once in a while. Restarting the CI job most usually fixes the issue, so it's most likely some sort of race condition during compilation.

For crash errors see:

https://github.com/firezone/firezone/actions/runs/5491388141/jobs/10007864417#step:4:1638
https://github.com/firezone/firezone/actions/runs/5510881208/jobs/10045788818#step:9:1009

25.18 == Compilation error in file lib/web/mailer/auth_email.ex ==
25.18 ** (UndefinedFunctionError) function Web.Mailer.module_info/1 is undefined (module Web.Mailer is not available)
25.18     Web.Mailer.module_info(:exports)
25.18     (elixir 1.15.2) src/elixir_import.erl:143: :elixir_import.get_functions/1
25.18     (elixir 1.15.2) src/elixir_import.erl:106: :elixir_import.calculate/6
25.18     (elixir 1.15.2) src/elixir_import.erl:28: :elixir_import.import/4

you can see other builds that were successful at the same branch without any code changes, eg.:

https://github.com/firezone/firezone/actions/runs/5491969244/jobs/10009045418#step:4:2783 (the latter failed job was triggered on the same codebase by GitHub merge queue checks).

Expected behavior

Consistently compile the module as it was pre-1.15.

[AndrewDryga]

Initially, we blamed the cache so we reset it and then split up the cache for Docker, but it did not help. I also hit compilation races once in a while locally (by running mix test without docker). MIX_ENV=test mix clean resolves it, but I can't find a way to reproduce it reliably so did not report it as a separate issue yet.

One thing to notice is that we use the module name Domain.Mailer.NoopAdapter in the parent umbrella app (domain) as an atom (without defining the module itself) and the actual module is defined and used only by the child (web) app. Is there anything changed in 1.15 around that?

[josevalim]

Thank you. However, without an isolated mechanism to reproduce the error, there isn't much we can do.

One thing to notice is that we use the module name Domain.Mailer.NoopAdapter

It depends. If you are using the module at compile-time, then it can be a problem (for example, a race condition). Elixir v1.15 did not change anything related to that, explicitly, but a race condition may happen when we make parts of the system faster (making it more likely for the race to be encountered). In both cases though, we need an isolated and minimal way to reproduce it. :)

[AndrewDryga]

I will report back if we find a way to reliably reproduce it, so far we can't even do it reliably on CI because the issue self-resolves after a restart.

If you are using the module at compile-time

No, it's just an atom there, so should not be the root cause.

[AndrewDryga]

Looks like this commit fixed the issue: firezone/firezone@1ffd08f. I think we might get a hint from it since it only does two things:

1. ensures that the adapter module (Web.Mailer.NoopAdapter) is always compiled before we try to use it;
2. swoosh dep is started a little more time before the web app is started because the web awaits for domain to start.

We also noticed that the race only happens with the Swoosh adapter and other code is not affected, I checked the Swoosh source, and the only suspicious thing is Swoosh.Mailer which uses an @on_load module attribute that executes the following function: https://github.com/swoosh/swoosh/blob/v1.11.3/lib/swoosh/mailer.ex#L236.

Will report more if I find the root cause.

[josevalim]

I actually think that explains a lot. If the @on_load hook fails, the module is not loaded, which is why it says it cannot find the module. The actual root cause is logged only a couple of lines above:

Error: 23.19 23:30:27.694 [error] Elixir.Web.Mailer.NoopAdapter does not exist

The reason why this happens is because Web.Mailer.NoopAdapter may or may not have been compiled before the mailer is loaded. The race condition has always been there, it is just that Elixir v1.15 compiles faster, so it may make it more likely to happen. To reproduce it consistently, you can add a Process.sleep(10000) before defmodule Web.Mailer.NoopAdapter, making the race more likely to happen.

There are two fixes here:

1. Temporary: define NoopAdapter in the same file and before Web.Mailer, to make sure it is always defined be

2. Move the Swoosh @on_load check to compile-time (such as @before_compile/@after_compile) and use Code.ensure_compiled instead of Code.ensure_loaded, which will guarantee an adapter defined in the same project will be compiled before hand. I don't see the need to check for this at runtime. The way Elixir manages your dependencies, it is safe to assume that if the module is available at compile, it should be available at runtime

I will move the issue to Swoosh. :) Thank you for the follow up!

[josevalim]

Done, please consider sending a PR there too: swoosh/swoosh#792 :)

[princemaple]

Give 1.11.4 a go! @AndrewDryga