New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iex --erl "-mode embedded" crashes #7208

Closed
fxn opened this Issue Jan 14, 2018 · 16 comments

Comments

Projects
None yet
3 participants
@fxn
Contributor

fxn commented Jan 14, 2018

TL;DR

See a summary of the resolution in this comment.

Environment

Elixir:

$ elixir --version
Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Elixir 1.7.0-dev (b959126) (compiled with OTP 20)

OS is macOS High Sierra (10.13.2).

Current behavior

$ iex --erl "-mode embedded"
2018-01-14 11:31:19 crash_report
    initial_call: {supervisor_bridge,user_sup,['Argument__1']}
    pid: <0.179.0>
    registered_name: []
    error_info: {error,undef,[{'Elixir.IEx.CLI',start,[],[]},{user_sup,start_user,3,[{file,"user_sup.erl"},{line,100}]},{user_sup,init,1,[{file,"user_sup.erl"},{line,49}]},{supervisor_bridge,init,1,[{file,"supervisor_bridge.erl"},{line,80}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
    ancestors: [kernel_sup,<0.165.0>]
    message_queue_len: 0
    messages: []
    links: [<0.166.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 190
2018-01-14 11:31:19 supervisor_report
    supervisor: {local,kernel_sup}
    errorContext: start_error
    reason: {undef,[{'Elixir.IEx.CLI',start,[],[]},{user_sup,start_user,3,[{file,"user_sup.erl"},{line,100}]},{user_sup,init,1,[{file,"user_sup.erl"},{line,49}]},{supervisor_bridge,init,1,[{file,"supervisor_bridge.erl"},{line,80}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
    offender: [{pid,undefined},{id,user},{mfargs,{user_sup,start,[]}},{restart_type,temporary},{shutdown,2000},{child_type,supervisor}]
2018-01-14 11:31:19 crash_report
    initial_call: {application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}
    pid: <0.164.0>
    registered_name: []
    error_info: {exit,{{shutdown,{failed_to_start_child,user,{undef,[{'Elixir.IEx.CLI',start,[],[]},{user_sup,start_user,3,[{file,"user_sup.erl"},{line,100}]},{user_sup,init,1,[{file,"user_sup.erl"},{line,49}]},{supervisor_bridge,init,1,[{file,"supervisor_bridge.erl"},{line,80}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,134}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
    ancestors: [<0.163.0>]
    message_queue_len: 1
    messages: [{'EXIT',<0.165.0>,normal}]
    links: [<0.163.0>,<0.162.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 243
2018-01-14 11:31:19 std_info
    application: kernel
    exited: {{shutdown,{failed_to_start_child,user,{undef,[{'Elixir.IEx.CLI',start,[],[]},{user_sup,start_user,3,[{file,"user_sup.erl"},{line,100}]},{user_sup,init,1,[{file,"user_sup.erl"},{line,49}]},{supervisor_bridge,init,1,[{file,"supervisor_bridge.erl"},{line,80}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},{kernel,start,[normal,[]]}}
    type: permanent
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,user,{undef,[{'Elixir.IEx.CLI',start,[],[]},{user_sup,start_user,3,[{file,\"user_sup.erl\"},{line,100}]},{user_sup,init,1,[{file,\"user_sup.erl\"},{line,49}]},{supervisor_bridge,init,1,[{file,\"supervisor_bridge.erl\"},{line,80}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,365}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{kernel,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,user,{undef,[{'Elixir.IEx.CLI',start,[],[]},{user_sup,start_user,3,[{file,"user_sup.

Crash dump is being written to: erl_crash.dump...done

Expected behavior

IEx starts just fine, or yields a controlled error message if running in embedded mode is not possible for some reason.

@josevalim

This comment has been minimized.

Member

josevalim commented Jan 14, 2018

@fxn

This comment has been minimized.

Contributor

fxn commented Jan 14, 2018

Why does it work in vanilla erl?

@josevalim

This comment has been minimized.

Member

josevalim commented Jan 14, 2018

@fxn

This comment has been minimized.

Contributor

fxn commented Jan 14, 2018

Let me see if I get it.

I'll explain the mental model I have built so far, and you tell me where is the error :).

I had understood that embedded eager loads the modules according to the boot script. That does not mean the boot script needs to have all of them listed. Also, with that description in hand, it does not follow there has to be a release involved to be able to load other modules or applications.

For example, let's write this Erlang application:

-module(foo).
-on_load(yo/0).

yo() ->
  io:format("yo!~n").

with resource file

{application, foo,
  [{modules, [foo]}]}.

Throw those two files in the current directory (which plays the role of ebin), and compile foo.erl.

OK, you can start that successfuly in embedded mode:

$ erl -pa . -mode embedded
Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V9.2  (abort with ^G)
1> application:start(foo).
ok

It works!

BTW, that seems to confirm application:load/1 does not load the application modules even in embedded mode, they seem to be truly orthogonal concepts (for anyone reading, that's a separate related topic we have been commenting offline.)

@fishcakez

This comment has been minimized.

Member

fishcakez commented Jan 14, 2018

That does not mean the boot script needs to have all of them listed

embedded will not automatically load a module when one of its functions is called when the module is not loaded, instead it will error immediately. It is expected that every module is loaded already. Try to call foo:yo(). in that last Erlang shell and it will get undef error.

@fxn

This comment has been minimized.

Contributor

fxn commented Jan 14, 2018

But the shell is functional and you can use the module:

$ erl -pa . -mode embedded
Erlang/OTP 20 [erts-9.2] ...

Eshell V9.2  (abort with ^G)
1> code:load_file(foo).
yo!
{module,foo}

To me, interactive/embedded seems a pretty simple concept that is quite independent of anything else, and that only assumes the existence of a boot script. Any boot script, the generated in a release, or the stock ones, doesn't matter.

Seeing how the system behaves my hypothesis (I have not read the source code) is that embedded means:

  • Preload whatever the boot script says to be preloaded.
  • Prevent the error handler from falling back to the code server.

And that's pretty much it.

The ERTS boots and seems to be functional (except for interactive mode, of course).

So, the question is: Shouldn't IEx behave like erl and launch? And, if there is a technical reason for which that is not possible, shouldn't the error be controlled because a user may expect it to work?

@josevalim

This comment has been minimized.

Member

josevalim commented Jan 14, 2018

Embedded is actually only about "prevent the error handler from falling back to the code server". You can run in interactive or embedded mode regardless of the presence of a bootscript. It is just that, without a bootscript, embedded mode is quite annoying as you need to do everything by hand. :) The bootscript automates the annoying parts.

The solution to this issue would be for Elixir to ship with its own bootscripts. One for Elixir and another for IEx. But my concern is that we may end-up making all Elixir commands slower because now we decided to preload the standard library as a whole only to avoid issues with embedded mode.

So for now I would rather continue treating Elixir and IEx as regular applications that, if you want to run in embedded mode, you need to release them yourself. Similar to your own applications.

From the Elixir root, you could do:

$ erl -pa lib/*/ebin
Erlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.0  (abort with ^G)
1> application:start(elixir).
{error,{not_started,compiler}}
2> application:start(compiler).
ok
3> application:start(elixir).
ok
4> application:start(iex).
ok
%% load all modules and then hit Ctrl+G
User switch command
 --> s 'Elixir.IEx'
 --> c
Interactive Elixir (1.7.0-dev) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

Maybe our discourse will change once we have releases as part of Elixir. So we will see.

@fishcakez

This comment has been minimized.

Member

fishcakez commented Jan 14, 2018

But my concern is that we may end-up making all Elixir commands slower because now we decided to preload the standard library as a whole only to avoid issues with embedded mode.

Fortunately this concern isn't valid because we can write a boot script that has the right behavior based on interactive/embedded mode. There is an instruction that effectively stops further modules being loaded in interactive mode but they will be loaded in embedded mode.

@josevalim

This comment has been minimized.

Member

josevalim commented Jan 14, 2018

@fishcakez oh, so I guess we should start chopping. :D Although I don't think we should start this before we have releases in Elixir... does it sound sane?

@fishcakez

This comment has been minimized.

Member

fishcakez commented Jan 14, 2018

I don't think we should either because come to think of it we can't write this boot script for multiple versions of OTP, we would have some form of release structure so that we always used the same erts runtime once Elixir is built. Almost certainly that means shipping Elixir as a release.

@fishcakez

This comment has been minimized.

Member

fishcakez commented Jan 14, 2018

ER not the same ERTS runtime, the same application versions of the built in OTP apps because we would need to know the paths (includes version) and the module list.

@fxn

This comment has been minimized.

Contributor

fxn commented Jan 14, 2018

You can run in interactive or embedded mode regardless of the presence of a bootscript.

Ah, I thought a boot script was required for ERTS to launch.

@fishcakez

This comment has been minimized.

Member

fishcakez commented Jan 14, 2018

Ah, I thought a boot script was required for ERTS to launch.

It is. The bootscript is always run but in interactive mode once the essentials are loaded the remaining load instructions are ignored. Other instructions still run the same though.

@fxn

This comment has been minimized.

Contributor

fxn commented Jan 14, 2018

Cool guys! So guess we can close this one by now.

By the way, I have no practical need for running IEx in embedded mode other than to inspect loading/starting applications for the sake of studying the concepts and procedures. It was a good exercise anyway to understand why it does not work.

Thanks for the discussion ❤️.

@fxn fxn closed this Jan 14, 2018

@josevalim

This comment has been minimized.

Member

josevalim commented Jan 14, 2018

I don't think we should either because come to think of it we can't write this boot script for multiple versions of OTP

Ah yes. The bootscript includes which modules to load and that will depend on the OTP version. :S

@fxn

This comment has been minimized.

Contributor

fxn commented Jan 18, 2018

Summary of the thread for the archives:

You can run erl -mode embedded because Erlang/OTP is distibuted as a release and its boot script loads all is needed to run the shell and have a normal runtime. Interactive loading is disabled, but, as a curiosity, you can still modify the code path with -pa and friends and load modules by hand using l(module) or code:load_file(module).

As of this writing, Elixir is not distibuted as a release, not to couple it with a specific Erlang/OTP version. When the ERTS is started, the boot script of the Erlang/OTP release is executed. Since that one preloads none of the Elixir core modules, and interactive mode is disabled, IEx just cannot run because the VM knows nothing about the needed Elixir modules.

To complement, let's also say that Elixir applications distributed as releases run by default in embedded mode. That is possible because Elixir and IEx are among the OTP applications bundled with the release and the boot script generated for the release loads them in consequence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment