spinedb_api not found in embedded Julia Console

[spine-o-bot]

In GitLab by @DillonJ on Nov 13, 2020, 16:33

Hi folks,

I wonder could you please help me diagnosing an issue.

I created a Julia kernel in toolbox using the same project and Julia version as my VS code REPL. However, I can't run SpineOpt from toolbox at all. spinedb_api is not found in the toolbox's Julia console but it is in the REPL and they are both using the same Julia project and the same embedded Conda environment.

I also tried using the Julia executable instead of a kernel and get the very same problem

In the REPL (embedded in VS Code)

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.2 (2020-05-23)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> Base.julia_cmd()
`'C:\Julia\bin\julia.exe' -Cnative '-JC:\Julia\lib\julia\sys.dll' -g1`

julia> Base.active_project()
"C:\\Users\\jodyd\\.julia\\environments\\v1.4\\Project.toml"

julia> using PyCall

julia> os=pyimport("os")
PyObject <module 'os' from 'C:\\Users\\jodyd\\.julia\\conda\\3\\lib\\os.py'>

julia> os.__file__
"C:\\Users\\jodyd\\.julia\\conda\\3\\lib\\os.py"

julia> os.system("conda env list")
# conda environments:
#
base                  *  C:\Users\jodyd\.julia\conda\3

0

julia> pyimport("spinedb_api")
PyObject <module 'spinedb_api' from 'C:\\Users\\jodyd\\AppData\\Roaming\\Python\\Python36\\site-packages\\spinedb_api\\__init__.py'>

In the console:

Jupyter QtConsole 4.5.5
Julia: A fresh approach to technical computing.

Base.julia_cmd()
Out[1]: `'C:\Julia\bin\julia.exe' -Cnative '-JC:\Julia\lib\julia\sys.dll' -g1`

Base.active_project()
Out[2]: "C:\\Users\\jodyd\\.julia\\environments\\v1.4\\Project.toml"

using PyCall

os=pyimport("os")
Out[4]: PyObject <module 'os' from 'C:\\Users\\jodyd\\.julia\\conda\\3\\lib\\os.py'>

os.__file__
Out[5]: "C:\\Users\\jodyd\\.julia\\conda\\3\\lib\\os.py"

os.system("conda env list")
# conda environments:
#
base                  *  C:\Users\jodyd\.julia\conda\3

Out[6]: 0

pyimport("spinedb_api")
PyError (PyImport_ImportModule

The Python package spinedb_api could not be imported by pyimport. Usually this means
that you did not install spinedb_api in the Python version being used by PyCall.

PyCall is currently configured to use the Julia-specific Python distribution
installed by the Conda.jl package.  To install the spinedb_api module, you can
use `pyimport_conda("spinedb_api", PKG)`, where PKG is the Anaconda
package the contains the module spinedb_api, or alternatively you can use the
Conda package directly (via `using Conda` followed by `Conda.add` etcetera).

Alternatively, if you want to use a different Python distribution on your
system, such as a system-wide Python (as opposed to the Julia-specific Python),
you can re-configure PyCall with that Python.   As explained in the PyCall
documentation, set ENV["PYTHON"] to the path/name of the python executable
you want to use, run Pkg.build("PyCall"), and re-launch Julia.

) <class 'ModuleNotFoundError'>
ModuleNotFoundError("No module named 'sqlalchemy'",)
  File "C:\Users\jodyd\.julia\conda\3\lib\site-packages\spinedb_api\__init__.py", line 12, in <module>
    from .database_mapping import DatabaseMapping
  File "C:\Users\jodyd\.julia\conda\3\lib\site-packages\spinedb_api\database_mapping.py", line 19, in <module>
    from .database_mapping_query_mixin import DatabaseMappingQueryMixin
  File "C:\Users\jodyd\.julia\conda\3\lib\site-packages\spinedb_api\database_mapping_query_mixin.py", line 21, in <module>
    from sqlalchemy import false, distinct, func, or_


Stacktrace:
 [1] pyimport(::String) at C:\Users\jodyd\.julia\packages\PyCall\BcTLp\src\PyCall.jl:547
 [2] top-level scope at In[7]:1

and finally, the same commands executed using the Julia exe in toolbox

`
'C:\Julia\bin\julia.exe' -Cnative '-JC:\Julia\lib\julia\sys.dll' -g1`
C:\Users\jodyd\.julia\environments\v1.4\Project.toml
# conda environments:

#

base                  *  C:\Users\jodyd\.julia\conda\3
ERROR:
LoadError:
PyError (PyImport_ImportModule

The Python package spinedb_api could not be imported by pyimport. Usually this means...

Is it because toolbox is using python's spinedb_api and Julia want's to use it's private conda's spine_dbapi and there is a conflict?
Any insights @PekkaSavolainen @manuelma @soininen ?

[spine-o-bot]

In GitLab by @DillonJ on Nov 13, 2020, 16:34

changed the description

[spine-o-bot]

In GitLab by @DillonJ on Nov 13, 2020, 16:35

changed the description

[spine-o-bot]

In GitLab by @DillonJ on Nov 13, 2020, 16:43

changed the description

[spine-o-bot]

In GitLab by @DillonJ on Nov 13, 2020, 16:43

changed the description

[spine-o-bot]

In GitLab by @PekkaSavolainen on Nov 13, 2020, 17:08

Here's my output from Julia Console. I may be missing something here but at least importing spinedb_api worked.

My guess is that the PyCall is pointing to the wrong Python. You can try the new configuration assistant to fix that (updated about two hours ago in issue #873). (File->Configuration Assistants...->SpineOpt.jl). But before that, you need to go and pick the Python kernel referring to the Conda env you want to use for PyCall in Settings->Tools.

Jupyter QtConsole 4.5.4
Julia: A fresh approach to technical computing.

Base.julia_cmd()

Out[1]: `'C:\Julia-1.4.0\bin\julia.exe' -Cnative '-JC:\Julia-1.4.0\lib\julia\sys.dll' -g1`

Base.active_project()
Out[2]: "C:\\data\\SpineToolboxProjects\\Julia Project\\Project.toml"

using PyCall
┌ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
└ @ Base loading.jl:1260

os=pyimport("os")
Out[4]: PyObject <module 'os' from 'C:\\Users\\ttepsa\\.julia\\conda\\3\\envs\\toolbox_36\\lib\\os.py'>

os.__file__
Out[5]: "C:\\Users\\ttepsa\\.julia\\conda\\3\\envs\\toolbox_36\\lib\\os.py"

os.system("conda env list")
'conda' is not recognized as an internal or external command,
operable program or batch file.
Out[6]: 1

pyimport("spinedb_api")
Out[7]: PyObject <module 'spinedb_api' from 'C:\\Users\\ttepsa\\.julia\\conda\\3\\envs\\toolbox_36\\lib\\site-packages\\spinedb_api\\__init__.py'>

[spine-o-bot]

In GitLab by @DillonJ on Nov 13, 2020, 17:11

Thanks @PekkaSavolainen
But doesn't the output of the commands I ran show that both are using the same Python?

E.g.

From REPL (working, finds spinedb_api):

julia> os.__file__
"C:\\Users\\jodyd\\.julia\\conda\\3\\lib\\os.py"

And from the toolbox's Julia console (not working, doesn't find spinedb_api):

os.__file__
Out[5]: "C:\\Users\\jodyd\\.julia\\conda\\3\\lib\\os.py"

[spine-o-bot]

In GitLab by @PekkaSavolainen on Nov 13, 2020, 17:18

What's the output of PyCall.python on both?

[spine-o-bot]

In GitLab by @DillonJ on Nov 13, 2020, 17:20

Toolbox console:

PyCall.python
Out[12]: "C:\\Users\\jodyd\\.julia\\conda\\3\\python.exe"

VS Code REPL:

julia> PyCall.python
"C:\\Users\\jodyd\\.julia\\conda\\3\\python.exe"

[spine-o-bot]

In GitLab by @PekkaSavolainen on Nov 13, 2020, 18:35

In your VS code REPL you have this

julia
julia> pyimport("spinedb_api")
PyObject <module 'spinedb_api' from 'C:\\Users\\jodyd\\AppData\\Roaming\\Python\\Python36\\site-packages\\spinedb_api\\__init__.py'>

So, spinedb_api that it imports is coming from AppData/Roaming/Python python. This probably means that your VS code REPL is adding the directory C:\\Users\\jodyd\\AppData\\Roaming\\Python\\Python36\\ to the PYTHONPATH at some point.

In Toolbox Julia Console:

julia
pyimport("spinedb_api")
PyError (PyImport_ImportModule

The Python package...

) <class 'ModuleNotFoundError'>
ModuleNotFoundError("No module named 'sqlalchemy'",)
  File "C:\Users\jodyd\.julia\conda\3\lib\site-packages\spinedb_api\__init__.py", line 12, in <module>
    from .database_mapping import DatabaseMapping
  File "C:\Users\jodyd\.julia\conda\3\lib\site-packages\spinedb_api\database_mapping.py", line 19, in <module>
    from .database_mapping_query_mixin import DatabaseMappingQueryMixin
  File "C:\Users\jodyd\.julia\conda\3\lib\site-packages\spinedb_api\database_mapping_query_mixin.py", line 21, in <module>
    from sqlalchemy import false, distinct, func, or_


Stacktrace:
 [1] pyimport(::String) at C:\Users\jodyd\.julia\packages\PyCall\BcTLp\src\PyCall.jl:547
 [2] top-level scope at In[7]:1

It's trying to import spinedb_api from your Conda environment site-packages (as it probably should?!). And it actually finds some outdated version there.

[spine-o-bot]

In GitLab by @PekkaSavolainen on Nov 13, 2020, 19:01

I just found this from https://github.com/JuliaPy/PyCall.jl

Troubleshooting

Here are solutions to some common problems:

By default, PyCall doesn't include the current directory in the Python search path. If you want to do that (in order to load a Python module from the current directory), just run pushfirst!(PyVector(pyimport("sys")."path"), "").

Here's also the discussion about this problem:
JuliaPy/PyCall.jl#48

So, the solution here depends on where do you want to import spinedb_api from?

[spine-o-bot]

In GitLab by @DillonJ on Nov 13, 2020, 20:23

Ah, ok - this makes sense - I think I dev'd spinedb_api at some point... that must be it!!

Thanks for that @PekkaSavolainen, really appreciate it

[spine-o-bot]

In GitLab by @PekkaSavolainen on Nov 27, 2020, 14:46

assigned to @PekkaSavolainen