Shared hook compilation failure

[omaralsoudanii]

Environment

• Oban Version: 2.14.2
• Oban pro: 0.13.3
• Oban.Web: 2.9.6
• PostgreSQL Version: 15
• Elixir & Erlang/OTP Versions (elixir --version): Elixir 1.14.2 (compiled with Erlang/OTP 25)

Current Behavior

Hello, I'm using a shared hook for Oban Pro worker (https://getoban.pro/docs/pro/0.13.0/Oban.Pro.Worker.html#module-worker-hooks) and attached it to two different worker modules, which works fine until I edit the shared hook module - the compilation fails with:

== Compilation error in file my_worker.ex ==
** (ArgumentError) hooks: unable to load callback module MyApp.shared_hook.
    (oban_pro 0.13.3) lib/oban/pro/worker.ex:507: Oban.Pro.Worker.init_stage!/2
    (elixir 1.14.2) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
    (elixir 1.14.2) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
    my_worker.ex:6: (module)

MyApp.shared_hook is a module that implements after_process/2. And inside my_worker.ex I'm using the shared hook:

use Oban.Pro.Worker,
    hooks: [MyApp.shared_hook]

Digging through Oban code, I found the following snippet when initializing the hooks in Oban.Pro.Stages.Hooks.validate_opt/1:

cond do
        not Code.ensure_loaded?(module) ->
          {:error, "unable to load callback module #{inspect(module)}"}

        not function_exported?(module, :after_process, 2) ->
          {:error, "#{inspect(module)} doesn't implement after_process/2"}

        true ->
          :ok
      end

Code.ensure_loaded? is returning false whenever a change happens to that file, recompiling fixes the issue. I adjusted the above code to also ensure that the modules passed are complied, which seems to be fixing the issue:

      Code.ensure_compiled!(module)

      cond do
        not Code.ensure_loaded?(module) ->
          {:error, "unable to load callback module #{inspect(module)}"}

        not function_exported?(module, :after_process, 2) ->
          {:error, "#{inspect(module)} doesn't implement after_process/2"}

        true ->
          :ok
      end

I'm not sure if the above fix is needed, or if I'm using the shared hooks incorrectly.

Expected Behavior

Using a shared hook shouldn't result in a compilation failure when editing the shared hook file.

[sorentwo]

I don't believe an Oban module should use ensure_compiled!, since that can interfere with module compilation order and possibly cause deadlocks. However, something funny is happening here, and we'll get to the bottom of it.

I'm curious, is shared_hook really the module name? It's not a standard-looking module like MyApp.SharedHook?

[sorentwo]

One other thing—is this happening in dev with code reloading, or from running a standalone mix compile?

[omaralsoudanii]

@sorentwo the module name is a standard-looking name like MyApp.SharedHook.
Indeed, something is funny happening here. In the beginning, I was looking into a different solution to this issue by trying to:

• Ensure the file permission is correct
• Rename the module
• Place the module somewhere else in the app
• Remove the explicit hooks attribute and use the implicit way for each module that is supposed to use the shared hook, which works but requires undesirable code duplication.

One other thing—is this happening in dev with code reloading, or from running a standalone mix compile?

It happens in both scenarios and sometimes randomly on GitHub CI which runs:

• mix deps.get
• mix deps.compile

Another weird fix that I forgot to mention on dev with code reloading is if I remove the hooks attribute and then start the server (Phoneix framework), then my app works without the compilation error. After that, I re-add the hooks attribute and start the server which runs without the compilation error.

[sorentwo]

Ok. On CI it would still have progressive compilation. I believe what's happening is that when the old version of the module initializes. I'm investigating and trying to recreate the problem still.

[sorentwo]

The issue was that there wasn't a compilation dependency between the hooks and the worker, but the worker tried verifying hook modules during compilation. It's fixed for the next release.