Pythox.Janitor exited after using ~PY

[wingyplus]

Reproduce repo wingyplus/sqlglot_ex#1.

After I convert Pythox.eval into ~PY and run it with iex -S mix, the Pythonx throws an error that Pythonx.Janitor process exited:

$ iex -S mix
Erlang/OTP 27 [erts-15.2.3] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] 
[jit]

==> pythonx
c++ -shared -fPIC -fvisibility=hidden -std=c++17 -Wall -Wextra -Wno-unused-parameter -Wno-
comment -I/nix/store/n0xcdacdnm62wv2x1w75w1jhjhsqz1fy-erlang-27.3/lib/erlang/erts-15.2.3/i
nclude -I/Users/thanabodee/src/github.com/wingyplus/sqlglot_ex/deps/fine/include -O3 -unde
fined dynamic_lookup -flat_namespace /Users/thanabodee/src/github.com/wingyplus/sqlglot_ex
/deps/pythonx/c_src/python.cpp /Users/thanabodee/src/github.com/wingyplus/sqlglot_ex/deps/
pythonx/c_src/pythonx.cpp -o /Users/thanabodee/src/github.com/wingyplus/sqlglot_ex/_build/
dev/lib/pythonx/priv/libpythonx.so
Compiling 10 files (.ex)
Generated pythonx app
==> sqlglot_ex
Compiling 2 files (.ex)
[pythonx] whereis(Pythonx.Janitor) failed. This is unexpected and a Python object will not
 be deallocated
               [pythonx] whereis(Pythonx.Janitor) failed. This is unexpected and a Python 
object will not be deallocated
                              [pythonx] whereis(Pythonx.Janitor) failed. This is unexpecte
d and a Python object will not be deallocated
                                             [pythonx] whereis(Pythonx.Janitor) failed. Th
is is unexpected and a Python object will not be deallocated
                                                            [pythonx] whereis(Pythonx.Jani
tor) failed. This is unexpected and a Python object will not be deallocated
                                                                           [pythonx] where
is(Pythonx.Janitor) failed. This is unexpected and a Python object will not be deallocated

[pythonx] whereis(Pythonx.Janitor) failed. This is unexpected and a Python object will not
 be deallocated
               [pythonx] whereis(Pythonx.Janitor) failed. This is unexpected and a Python 
object will not be deallocated
                              [pythonx] whereis(Pythonx.Janitor) failed. This is unexpecte
d and a Python object will not be deallocated
                                             [pythonx] whereis(Pythonx.Janitor) failed. Th
is is unexpected and a Python object will not be deallocated
                                                            [pythonx] whereis(Pythonx.Jani
tor) failed. This is unexpected and a Python object will not be deallocated
                                                                           [pythonx] where
is(Pythonx.Janitor) failed. This is unexpected and a Python object will not be deallocated

[pythonx] whereis(Pythonx.Janitor) failed. This is unexpected and a Python object will not
 be deallocated
               [pythonx] whereis(Pythonx.Janitor) failed. This is unexpected and a Python 
object will not be deallocated

== Compilation error in file lib/sql_glot.ex ==
** (exit) exited in: GenServer.call(Pythonx.Janitor, :ping, 5000)
    ** (EXIT) no process: the process is not alive or there's no process currently associa
ted with the given name, possibly because its application isn't started
    (elixir 1.18.3) lib/gen_server.ex:1121: GenServer.call/3
    (pythonx 0.4.5-dev) lib/pythonx.ex:242: Pythonx.eval/3
    (pythonx 0.4.5-dev) lib/pythonx/ast.ex:22: Pythonx.AST.scan_globals/1
    (pythonx 0.4.5-dev) expanding macro: Pythonx.sigil_PY/2
    lib/sql_glot.ex:23: SQLGlot.transpile/4

[josevalim]

We should improve the exception but ~PY is meant to be used by notebook scripts mostly, because it requires a Python environment to run at compile-time.

[jonatanklosko]

Oh, we already have a specific error for using ~PY at compile time, but we raise it if the interpreter is not initialized (which it should not be). However it doesn't work as expected because of this line, which explicitly initializes the interpreter at compile time (though it would only do that on the first compilation anyway).

[josevalim]

@jonatanklosko should we check if the Janitor process is running instead or if the application is started?

[jonatanklosko]

Yeah, the reason why the result is so confusing is because Pythonx is initialized (unexpected) and the application is not started. If it was started it would actually work, though this should still not be encouraged.

So perhaps we raise this error:

pythonx/lib/pythonx.ex

Lines 317 to 321 in bf63ea0

	raise RuntimeError,
	"using ~PY sigil requires the Python interpreter to be already initialized. " <>
	"This sigil is designed for dynamic evaluation environments, such as IEx or Livebook. " <>
	"If that is your case, make sure you initialized the interpreter first, otherwise " <>
	"use Pythonx.eval/2 instead. For more details see Pythonx.sigil_PY/2 docs"

When Pythonx is is not initialized or when not started?

[josevalim]

When not started, because we need processes from its sup tree for it to work.

[jonatanklosko]

Right, I mentioned "not initialized", because that's what we already do, we need to do both :D

[wingyplus]

To confirm my understanding, my library should do:

1. Move uv_init to inside a function (or process, maybe).
2. Don't use ~PY inside a function (or anything that evaluate at compile time).

Is that correct?

[josevalim]

Check out the application usage here, you can use configuration for uv_init:

https://github.com/livebook-dev/pythonx?tab=readme-ov-file#usage-application

However, if you have a library, then indeed calling it on application start would be best.

[wingyplus]

Thanks @josevalim 🙏