Production Release: Function Mix.Compilers.ApplicationTracer.trace/2 is undefined

[Exadra37]

Proof of Concept

https://github.com/Exadra37/elixir-library-domo-bug

Environment

• Elixir version (elixir -v):

$ ./bin/tasks remote
Erlang/OTP 23 [erts-11.1.6] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1]

Interactive Elixir (1.11.3) - press Ctrl+C to exit (type h() ENTER for help)

• Domo version (mix deps | grep domo | head -1):

$ mix deps | grep domo | head -1
* domo 1.0.1 (Hex package) (mix)

• TypedStruct version (mix deps | grep typed_struct | head -1):

$ mix deps | grep typed_struct | head -1
* typed_struct 0.2.1 (Hex package) (mix)

I run it in development and in production with the docker image from the Phoenix releases docs:

https://hexdocs.pm/phoenix/releases.html#containers

Versions used to build the docker image:

ELIXIR_VERSION=1.11.3
ERLANG_OTP_VERSION=23.2.2
ALPINE_VERSION=3.12.1

Actual behavior

The error from a production release:

 09:20:32.459 | error | module=gen_server function=error_info/7 line=943  | GenServer #PID<0.2551.0> terminating
app_1  | ** (UndefinedFunctionError) function Mix.Compilers.ApplicationTracer.trace/2 is undefined (module Mix.Compilers.ApplicationTracer is not available)
app_1  |     Mix.Compilers.ApplicationTracer.trace({:alias_reference, [line: 45], NaiveDateTime}, #Macro.Env<aliases: [], context: nil, context_modules: [TypeIt.Progress], file: "/app/lib/type_it/lib/progress.ex", function: nil, functions: [{Kernel, [!=: 2, !==: 2, *: 2, ...]}], lexical_tracker: #PID<3.316.0>, line: 42, macro_aliases: [], macros: [{Domo, ...}, {...}], module: TypeIt.Progress, requires: [...], ...>)
app_1  |     (elixir 1.11.3) src/elixir_env.erl:36: :elixir_env."-trace/2-lc$^0/1-0-"/3
app_1  |     (elixir 1.11.3) src/elixir_env.erl:36: :elixir_env.trace/2
app_1  |     (elixir 1.11.3) lib/macro.ex:1440: Macro.do_expand_once/2
app_1  |     (elixir 1.11.3) lib/macro.ex:1610: Macro.expand_until/2
app_1  |     (domo 1.0.1) lib/domo/type_spec_matchable/remote_type.ex:11: Domo.TypeSpecMatchable.RemoteType.expand/2
app_1  |     (domo 1.0.1) lib/domo/type_contract.ex:642: Domo.TypeSpecMatchable.Any.match_spec?/3
app_1  |     (tasks 0.1.0) lib/type_it/lib/progress.ex:42: TypeIt.Progress.TypeChecker.__field_error/1

The Typed Struct code:

defmodule TypeIt.Progress do

  use Domo

  @all_states %{
    backlog: "Backlog",
    todo: "Todo",
    doing: "Doing",
    pending: "Pending",
    done: "Done",
    archived: "Archived",
  }

  @states Map.keys(@all_states)

  typedstruct do
    field :state, :backlog | :todo | :doing | :pending | :done | :archived
    field :title, String.t()
    field :since, NaiveDateTime.t()
  end

  def default(), do: new_for!(:todo)

  def next_state(:backlog), do: :todo
  def next_state(:todo), do: :done
  def next_state(:done), do: :todo

  def new_for!(state), do: new!(state: state, title: @all_states[state], since: NaiveDateTime.utc_now())
  def new_for!(state, since: since), do: new!(state: state, title: @all_states[state], since: since)
  def new_for!(state, title: title), do: new!(state: state, title: title, since: NaiveDateTime.utc_now())

  def states() do
    @states
  end

  def all() do
    @all_states
  end
end

Expected behavior

In production is throwing the reported error that crashes the app, but in development it works ok.

[Exadra37]

In the Elixir forum Jose Valim kindly took it's time to evaluate this issue, and he have given me this reponse:

I assume the issue is around this line:

Domo/lib/domo/type_checker_generator.ex

Line 128 in 0fb448a

caller_env = Macro.escape(caller_env)

It is trying to save a compilation environment to use it to expand things at runtime. Not only this slow, it won’t work as you noticed. :slight_smile: They should do whatever expansion they need at compile time and remove the runtime expand calls.

[IvanRublev]

Hey,
The version of Domo v1.2.0 is released. It resolves all necessary types at compile time. It was long planned, so this issue is a good cause to do this. Please, find the archive of the proof of concept repository with the updated library fixing the problem.
elixir-library-domo-bug-fix-1.2.0.zip

I checked the TypeIt.Progress it works as well.

To adapt to Domo v1.2.0

• add :domo compiler to the project mix.exs configuration file
• change the call to constructor function name from new!/1 -> new/1

It seems that you going to import struct's data from JSON. In this case, you can execute Progress.ensure_type!(progress) as the final step of import to verify all fields against struct type. Or Progress.ensure_type_ok(progress) to get the result as an error tuple instead of the exception.

This version v1.2.0 can increase project compilation times to approximately 25% because it resolves all types at compile time and generates code. Please, evaluate how it works for you. If you have a code base with many structures I'll be glad to get to know how compilation times are changed, you can email me at ivan at ivanrublev.me.

[Exadra37]

To adapt to Domo v1.2.0

add :domo compiler to the project mix.exs configuration file

You may want to clarify this to be like suggested in the error Mix.compilers() ++ [:domo]:

== Compilation error in file lib/tasks/todos/todo_type.ex ==
** (CompileError) lib/tasks/todos/todo_type.ex:28: Domo should be included after the :elixir in the compilers list in the project's configuration mix.exs file because it launches the second-pass of the compilation to resolve remote types that are in project BEAM files.
The mix.exs should have project/0 function returning a list with the following key compilers: Mix.compilers() ++ [:domo] where the :domo location is after the :elixir compiler.
    (domo 1.2.0) lib/domo/raises.ex:31: Domo.Raises.raise_absence_of_domo_compiler!/3
    (domo 1.2.0) expanding macro: Domo.__using__/1
    lib/tasks/todos/todo_type.ex:28: Tasks.Todos.TodoType (module)
    (elixir 1.11.3) expanding macro: Kernel.use/1
    lib/tasks/todos/todo_type.ex:28: Tasks.Todos.TodoType (module)
    (elixir 1.11.3) lib/kernel/parallel_compiler.ex:314: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7

[Exadra37]

To adapt to Domo v1.2.0

add :domo compiler to the project mix.exs configuration file
change the call to constructor function name from new!/1 -> new/1

You may want to add a third step:

• Add Use TypedStruct after Use Domo

I am now curious to know the reason behind the need to explicitly require now the typed struct, do you mind to briefly explain?

[Exadra37]

I use Domo to type structs that I often use as default values in Ecto schemas, but that is not possible to do anymore:

== Compilation error in file lib/tasks/todos/todo.ex ==
** (RuntimeError) The Tasks.Todos.TodoType.TypeEnsurer module implementation should be generated by Domo. Please, ensure that :domo compiler is included after the :elixir in the compilers list in the project's configuration mix.exs file.

    lib/tasks/todos/todo_type.ex:3: Tasks.Todos.TodoType.TypeEnsurer.ensure_type!/1
    lib/tasks/todos/todo_type.ex:3: anonymous fn/2 in Tasks.Todos.TodoType.new/1
    (stdlib 3.14) maps.erl:233: :maps.fold_1/3
    lib/tasks/todos/todo_type.ex:3: Tasks.Todos.TodoType.new/1
    lib/tasks/todos/todo.ex:17: (module)
    (stdlib 3.14) erl_eval.erl:680: :erl_eval.do_apply/6
    (elixir 1.11.3) lib/kernel/parallel_compiler.ex:314: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7

The module:

defmodule Tasks.Todos.TodoType do

  use Domo
  use TypedStruct

  typedstruct do
    field :type, :professional | :personal
    field :title, String.t()
  end

  def default(), do: new(type: :professional, title: "Professional")

end

Ecto schema:

  schema "todos" do
    field :title, :string, null: false
    field :description, :string, default: nil
    field :type, :map, default: Tasks.Todos.TodoType.default()

    timestamps()
  end

I understand why, because by this time the :domo compiler has not done its pass yet.

So, this will be a limitation of the current approach or can I do it differently?

[Exadra37]

The Domo library 1.2.0 fixes the issue I reported for a production release, but for development creates another issue:

The ProgressType.TypeEnsurer module implementation should be generated by Domo. Please, ensure that :domo compiler is included after the :elixir in the compilers list in the project's configuration mix.exs file. 

Steps to reproduce:

unzip .elixir-library-domo-bug-fix-1.2.0.zip -d domo-bug-fix 
cd domo-bug-fix
cp .env.example .env
sudo docker-compose run --rm --service-ports dev
mix deps.get
mix compile
iex -S mix phx.server

Now visit http://localhost:4000 and you will see the error.

[IvanRublev]

Hey,

I understand why, because by this time the :domo compiler has not done its pass yet.
So, this will be a limitation of the current approach or can I do it differently?

That's the limitation of the current approach. The generated new/1 can't be used in macro calls yet.

The Domo library 1.2.0 fixes the issue I reported for a production release, but for development creates another issue

Thanks, let's see what can be done.

[IvanRublev]

Turns out that after adding the following line to config.exs file, Phoenix builds and runs domo modules:

config :domo_bug, DomoBugWeb.Endpoint,
  reloadable_compilers: [:phoenix] ++ Mix.compilers() ++ [:domo],

And the iex -S mix phx.server serves pages correctly.

Same time, the Phoenix hot code reload doesn't work for these type ensurer modules completely. Meaning that if one modifies the structure's fields they need to rerun iex -S mix phx.server, otherwise the verification works against the old type.

Currently, domo type ensurer modules recompile each time when project compilation runs.

Let's see what can be done to address these.

[Exadra37]

Meaning that if one modifies the structure's fields they need to rerun iex -S mix phx.server, otherwise the verification works against the old type.

If we are already using iex -S mix phx.server then doing a recompile should suffix 🤔

[IvanRublev]

It seems that Phoenix doesn't respect recompile, unfortunately.

[IvanRublev]

The updated version https://hex.pm/packages/domo/1.2.1 has the full support of Phoenix hot-reload.

Please, update the name of the compiler from :domo -> :domo_compiler in the following files:

# in mix.exs
def project do
  [
    compilers: [:phoenix] ++ Mix.compilers() ++ [:domo_compiler],
  ]
end


# and in config.exs
config :domo_bug, DomoBugWeb.Endpoint,
  reloadable_compilers: [:phoenix] ++ Mix.compilers() ++ [:domo_compiler],

and do mix clean --deps && mix compile to recompile the project.

Now you can start the server with iex -S mix phx.server

After change of the type in the struct or in any level deep dependant struct, Phoenix hot-reloads the module with updated new/1 function correctly 😄

[Exadra37]

It seems that is now working properly.

You can try it by yourself at https://taskit.exadra37.com

Thanks for your work and fixes :)

Can I buy you a coffee?

[Exadra37]

Can I buy you a coffee?

Do you want to add the button the repo or send me a link to pay you a coffee?

https://ko-fi.com/
https://www.buymeacoffee.com/

or maybe just enable Github Sponsors?

[IvanRublev]

Hey, thanks. I'm very glad that the library working for you!
Thanks for the coffee. I'll think about what can be done.