generate one new module and one atom per compilation during runtime

[redink]

elixir/lib/elixir/src/elixir_code_server.erl

Line 132 in d8a533d

{list_to_atom("elixir_compiler_" ++ integer_to_list(I)), I}.

Environment

$ iex
Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> 

Our use case

We using beam as a specific cache for database, we need compile some given modules to load the latest data from db to beam in interval time during runtime. Using:

Kernel.ParallelCompiler.files_to_path(source_files, beam_path)

Current behavior

create one new module and one atom per compilation, the atom just like:

elixir_compiler_8
elixir_compiler_7
elixir_compiler_6
elixir_compiler_5
elixir_compiler_4
elixir_compiler_3
elixir_compiler_2

and the module just like:

=mod:elixir_compiler_0
Current size: 0
Old size: 5193
Old attributes: 836C00000001680264000376736E6C000000016E1000E98FE57BEC2C0B4635218BB1ED55D6CF6A6A
Old compilation info: 836C0000000368026400076F7074696F6E736C0000000164000E6E6F7761726E5F6E6F6D617463686A680264000776657273696F6E6B0005372E312E326802640006736F757263656B005D2F6F70742F747562692F636D735F736572766963652F6C69622F706F6C6963795F656E67696E652D342E312E35302F7372632F6C69622F706F6C6963795F656E67696E652F67656E2F67656E5F7365726965735F706F6C6963792E65786A
=mod:elixir_compiler_12
Current size: 0
Old size: 9748
Old attributes: 836C00000001680264000376736E6C000000016E1000DAC1E389C40641D4E01D37111B6418976A6A
Old compilation info: 836C0000000368026400076F7074696F6E736C0000000164000E6E6F7761726E5F6E6F6D617463686A680264000776657273696F6E6B0005372E312E326802640006736F757263656B00602F6F70742F747562692F636D735F736572766963652F6C69622F706F6C6963795F656E67696E652D342E312E35302F7372632F6C69622F706F6C6963795F656E67696E652F67656E2F67656E5F636F6E7461696E65725F706F6C6963792E65786A
=mod:elixir_compiler_4
Current size: 0
Old size: 9027
Old attributes: 836C00000001680264000376736E6C000000016E10009D8CADBE367736B8F3FAFE93BDEF7E5C6A6A
Old compilation info: 836C0000000368026400076F7074696F6E736C0000000164000E6E6F7761726E5F6E6F6D617463686A680264000776657273696F6E6B0005372E312E326802640006736F757263656B00512F6F70742F747562692F636D735F736572766963652F6C69622F706F6C6963795F656E67696E652D342E312E35302F7372632F6C69622F706F6C6963795F656E67696E652F736C696365732F6E362E65786A

just because there is limitation for module amount in Erlang, the Elixir/Erlang node will be crash when the module amount reach the limitation, the crash dump slogan is:

no more index entries in module_code (max=65536)

the detailed information is:

=hash_table:module_code
size: 51437
used: 48890
objs: 65536
depth: 3
=index_table:module_code
size: 65536
limit: 65536
entries: 65536

Attempt

I tried to modify the Erlang vm args just as ERL_MAX_PORTS, BUT I failed.

Because the limitation for module amount can't modify in Erlang, it is hard-code in:

https://github.com/erlang/otp/blob/master/erts/emulator/beam/module.c#L38

Thanks.

[redink]

I noticed the logic for retrieve_compiler_module and return_compiler_module are complete. It will retrieve the compiler module when compile, and will return after compile.

dispatch(Module, Fun, Args, Purgeable, I, E) ->
  Res = Module:Fun(Args),
  code:delete(Module),
  if Purgeable ->
      code:purge(Module),
      return_compiler_module(I);
     true ->
       ok
  end,
  {Res, E}.

elixir/lib/elixir/src/elixir_compiler.erl

Line 91 in d8a533d

ok

But, why not return when can't purge the module ?

[josevalim]

We can't return because we can't purge the module. If we return a non-purged module, it will be redefined later on, which then would cause the purging possibly breaking existing code. You rather need to figure out why your modules are not considered as purgeable. You can get the beam from the disk and call this:

beam_lib:chunks(Binary, [labeled_locals])

If you have labelled locals, i.e. local anonymous functions in the module body, then they can't be purged.

[redink]

Thanks for reply.

Just want to know, why anonymous functions will affect the purge ?

( I only know that anonymous functions maybe lead failed when hot upgrade, but I don't know why clearly.

[josevalim]

iex(1)> :beam_lib.chunks(Test, [:labeled_locals])

You need to run the command above for the :elixir_compiler_1 modules and so on. If the internal modules have functions, it means they can spawn processes and purging those modules mean they will crash.

If you can find an :elixir_compiler_X that has labelled locals, you can then call :elixir_compiler_X.__FILE__ or :elixir_compiler_X.__MODULE__ and it will tell you if a file or module are generating the unpurgeable modules. If you paste them here, I can help you figure out why they can't be purged.

[redink]

I can find an :elixir_compiler_X, but I can't call functions :elixir_compiler_X.__FILE__ and :elixir_compiler_X.__MODULE__, because :elixir_compiler_X is not available.

Just like:

iex(1)> ReconTrace.calls([{:beam_lib, :_, fn _ -> :return end}], 100)
26
iex(2)> Kernel.ParallelCompiler.files_to_path(["test.ex"], "./")

22:51:41.714382 <0.1445.0> :beam_lib.chunks(<<70, 79, 82, 49, 0, 0, 8, 60, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 1, 160, 0,
  0, 0, 32, 17, 101, 108, 105, 120, 105, 114, 95, 99, 111, 109, 112, 105, 108,
  101, 114, 95, 48, 10, 95, 95, 77, 79, 68, 85, 76, ...>>, [:labeled_locals])
 
22:51:41.726182 <0.1445.0> :beam_lib.chunks/2 --> {:ok,
 {:elixir_compiler_0, [labeled_locals: [{:"-__MODULE__/1-fun-0-", 1, 10}]]}}
[OtherModule]
iex(3)> 
nil
iex(4)> :elixir_compiler_0.module_info
** (UndefinedFunctionError) function :elixir_compiler_0.module_info/0 is undefined (module :elixir_compiler_0 is not available)
    :elixir_compiler_0.module_info()

I think this due to:

elixir/lib/elixir/src/elixir_compiler.erl

Line 86 in d8a533d

code:delete(Module),

[josevalim]

@redink oh, you can try removing that line and running make erlang and then run against your local Elixir.

[josevalim]

@fishcakez suggested a less conservative approach for purging modules which will likely solve your problem. Please give master a try and let us know!

[redink]

during compilation, the state of elixir code server is:

iex(58)> :sys.get_state(:erlang.whereis(:elixir_code_server))
{:elixir_code_server, %{
   "/usr/local/bin/mix" => true
 }, {[7, 4], 8}, %{
   #Reference<0.3414257687.589824008.147734> => M1,
   #Reference<0.3414257687.589824008.147738> => M2,
   #Reference<0.3414257687.590086151.234246> => M3,
   #Reference<0.3414257687.590610436.53835> => M4,
   #Reference<0.3414257687.591396866.54944> => M5,
   #Reference<0.3414257687.591396866.54960> => M6,
   #Reference<0.3414257687.591396866.188095> => M7,
   #Reference<0.3414257687.591396869.7197> => M8
 }, %{#PID<0.586.0> => :ok}}

after compile, the state is:

iex(84)> :sys.get_state(:erlang.whereis(:elixir_code_server))
{:elixir_code_server, %{
   "/usr/local/bin/mix" => true
 }, {[1, 0, 2, 3, 6, 5, 4, 7], 8}, %{}, %{}}

after some times retry, the counter of mod_pool didn't increase.

so, I think it works.

Thanks all of you again.

[josevalim]

The approach we chose here is causing some false positives. We will have to revert this or come up with another strategy.