SystemError: Parent module '' not loaded, cannot perform relative import

[nbro]

I am trying to create a html documentation for a module with the following command:

pdoc --html --overwrite huffman.py 

But I am receiving the following error:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/bin/pdoc", line 458, in <module>
    module = imp.load_source('__pdoc_file_module__', fp, f)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/imp.py", line 172, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 693, in _load
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 662, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/Users/cell/Desktop/ands/algorithms/greedy/huffman.py", line 56, in <module>
    from ...ds.MinPriorityQueue import *
SystemError: Parent module '' not loaded, cannot perform relative import    

I know that the problem is related to the fact that I am doing a relative import, which doesn't work because huffman.py is not being imported as I programmed it to be imported. So, currently, I am using the following code:

if __name__ == "__main__":
    from os import path, sys
    BASE_DIR = path.dirname(path.dirname(path.dirname(__file__)))  # ands
    sys.path.append(BASE_DIR)
    from ds.MinPriorityQueue import *
else:
        from ...ds.MinPriorityQueue import *

Which is already a mess, because I am trying to handle both when this huffman.py file (or module) is run as main or imported from outside ands, which is the parent of the parent of the package where huffman.py is situated. To solve the problem, I did the following:

from os import path, sys
BASE_DIR = path.dirname(path.dirname(path.dirname(__file__)))  # ands
sys.path.append(BASE_DIR)

if __name__ == "__main__":
    from ds.MinPriorityQueue import *
else:
    try:
        from ...ds.MinPriorityQueue import *
    except SystemError:
        from ds.MinPriorityQueue import *

Unfortunately this is far from being simple, considered that what I want to achieve should be very simple.

Any suggestions to simplify my code?

[BurntSushi]

You might try fiddling with PYTHONPATH instead (modifying sys.path is essentially never a good solution and it should never be recommended in a project's README). Relative imports should be similarly barred. If for some reason pdoc cannot work with relative imports, then I doubt I will personally spend time hacking workarounds for it because it's not idiomatic Python.

Let me know if that doesn't help and I'll take a closer look.

[nbro]

@BurntSushi I would also understand that we should not mess with sys.path, on the other hand relative imports exist for a reason, and in my opinion should be used, if they work.Regarding fiddling with PYTHONPATH, I am not sure I have ever done it and if it would be more useful or not... It should work no matter if I am on my computer or someone is on is own computer...

[BurntSushi]

on the other hand relative imports exist for a reason, and in my opinion should be used, if they work

It is not a goal of pdoc to support non-idiomatic Python code. It is too much work and too complicated.

I'm happy to help you get pdoc working, but let's not make this a discussion about pdoc supporting non-idiomatic Python.

[nbro]

@BurntSushi No problem, anyway I've never mentioned that I wanted pdoc to support non-idiomatic Python. And by the way, how can I know what's idiomatic for Python by the way, if you know? And what does it mean to be idiomatic?

[BurntSushi]

@Dossan I don't know everything about idiomatic Python and there is unfortunately no canonical source. Nevertheless, relative imports are bad. Here are some other tips: https://docs.python.org/2/howto/doanddont.html

[nbro]

@BurntSushi Honestly, apart from the fact that sometimes relative imports do not work in some environments, like with pdoc, I've still not understood why they are bad, even after reading what you suggested. For example, in my case, for example I have the following structure:

ds/
    MinPriorityQueue.py
    __init__.py
    ...
algorithms/
    ...
    greedy/
        __init__.py
        huffman.py
        ...

In huffman.py I need to import the class MinPriorityQueue.py, and there's no way except from using relative imports or appending to sys.path. Correct me if I am wrong. I am still not familiar with setuptools and how to create an installable package, but I don't understand how can I just use absolute imports in such an situation...Yes, it would be somehow possible if I am able to tell Python that my main directory from which to start to import is ands, but I haven't found a way yet to do it. I mean, all imports would then be absolute with respect to the main directory ands...

I know this is another problem, and you don't have to answer. Thanks anyway!

[BurntSushi]

Honestly, apart from the fact that sometimes relative imports do not work in some environments, like with pdoc, I've still not understood why they are bad, even after reading what you suggested.

Not sure what else to add, then. Google around for "python relative import" and you'll see a whole bunch of issues. They make code harder to read and result in all sorts of weird import problems (as evidenced by googling). Python's import semantics are complicated enough as it is!

In huffman.py I need to import the class MinPriorityQueue.py, and there's no way except from using relative imports or appending to sys.path.

That's wrong. You should be able to do from ds import MinPriorityQueue or import ds.MinPriorityQueue as MinPriorityQueue. If you can't, then your project isn't really structured right.

I am still not familiar with setuptools and how to create an installable package, but I don't understand how can I just use absolute imports in such an situation...Yes, it would be somehow possible if I am able to tell Python that my main directory from which to start to import is ands, but I haven't found a way yet to do it. I mean, all imports would then be absolute with respect to the main directory ands...

The idiomatic way to build Python projects is to use a setup.py which will make it pip installable, and then work inside a virtualenv. You might try a tutorial to get started: https://www.dabapps.com/blog/introduction-to-pip-and-virtualenv-python/ --- And here's something else that covers setup.py specifically: https://pythonhosted.org/an_example_pypi_project/setuptools.html

Unfortunately, pip and Python's packaging situation in general is pretty intimidating. It's a big barrier to entry IMO. You might need to read several tutorials to really grok it. I don't blame you for having avoided it. Nevertheless, it's what we're stuck with.

[nbro]

@BurntSushi I would definitely stop using relative imports, but I am not able to make my scripts run without ImportErrors using absolute imports without appending paths to sys.path in every module.

That's wrong. You should be able to do from ds import MinPriorityQueue or import ds.MinPriorityQueue as MinPriorityQueue. If you can't, then your project isn't really structured right.

As far as I understood, doing from ds import MinPriorityQueue in huffman.py would only work if the path to ds is the sys.path, but if I run my module as a script (or as __main__), the only path that will be in sys.path (apart from the usual ones) is .../algorithms/greedy (specifically in my case: /Users/cell/Desktop/ands/ands/algorithms/greedy). Now, I need to import a module that is in the package ds which is in the second ands (folder), but the path in not in sys.path. Is the only solution to the problem adding the path to ds manually?

Since I have a lot of modules in a similar situation (complex but logical imports), this would be really tedious to do in all my submodules that need to import classes from other subpackages from parent packages.

You said I have my project not well setup, do you know a guide where it explains how to deal with these complex situations of importing modules?

The idiomatic way to build Python projects is to use a setup.py which will make it pip installable, and then work inside a virtualenv

I knew already about virtualenv, but I am not seeing any advantage for my case, except from isolating the dependencies, which in my case are simply tabulate.

As far as I have understood, creating a setup.py file would only help me in making my whole package installable, but for now that's not a real problem, unless setting up that file would make my imports work smoothly well. So far I have not figured out anything about setup.py that would ease the process of importing modules and packages around within the modules of my project, but maybe it does something...

People keep saying "avoid relative imports", and I would really avoid them, but at the end to make a simply module of 50-10 lines of code to run both as a script and as an imported module it takes other 10 lines of code with many hacks.

You might need to read several tutorials to really grok

Believe me, I've gone through many tutorials and answers to questions on SO, but none of them tells me how to deal with complex imports that I need to do. It would be really nice to just be able to do absolute imports after specifying the main directory of your project. All imports would be absolute with respect to that main directory. This would be really a nice feature, which would never cause problems!!!

[BurntSushi]

As far as I have understood, creating a setup.py file would only help me in making my whole package installable, but for now that's not a real problem, unless setting up that file would make my imports work smoothly well.

It would make absolute imports work. If your Python package is installed, then it is accessible in a standard sys.path location. That's pretty much the whole point of having an installable package.

People keep saying "avoid relative imports", and I would really avoid them, but at the end to make a simply module of 50-10 lines of code to run both as a script and as an imported module it takes other 10 lines of code with many hacks.

The Python module system and its import semantics are complicated You're only hope for thriving in the ecosystem, including using other tools, is to follow ecosystem conventions. By operating outside of a setup.py and having a pip installable package, you are going against the grain of those conventions. You will make life harder on yourself. Indeed, you will wind up in situations where "avoiding relative paths means hacking at sys.path."

I've linked to a few tutorials. I'd recommend starting there. There is no simple set of commands I can give you. You need to:

1. Read, understand and produce a setup.py file for your project. My guess is that you can start with something very simple. You could, for example, look at how pdoc is setup. Or one of many other Python projects on PyPI.
2. Learn how to use pip.
3. Learn how to use virtualenv.
4. Apply it to your project, e.g., virtualenv me && source me/bin/activate && pip install -e ..

You're other choice is to ignore ecosystem conventions and do things your own way. I'm afraid that may make pdoc more inconvenient to use, however, I have presented an easy work-around for you.

[nbro]

@BurntSushi Thank you very much! I really going to have a deeper look at setup.py (and setuptools, etc) and more at idiomatic conventions, because I am really tired of doing all these "hacks".

Actually, I already know more or less how to work with a virtualenv, but not exactly with setting up the setup.py. I will have a closer look at your suggested tutorial, etc, and I hope that imports will no more be a problem.

Actually, I really like this pdoc tool, so I guess I am really going to follow your suggestions :)

[BurntSushi]

@Dossan No problem, good luck. I wish you the best!

[nbro]

@BurntSushi I have set up a setup.py and did what you suggested, including working on a virtualenv, and now after the installation of my package, imports are no more a problem, because of course the path to the main folder of my project is on sys.path (because it is installed to site-packages). Now, I would have a question. I tried also to install my package globally doing

python3.5 setup.py install

from the root directory of my project, and it produces the following folders in the root folder of my project:

ands.egg-info
build
dist

I am still not 100% sure of what's happening when I do pythonX setup.py install, but as far as I understood it should first build my package and then install it. I am not sure here what "building" actually does or means, but I guess it creates a compacted package or something similar... The problem is that I have had a looked at the site-packages folder and almost all other packages have more than one related file or folder. For example, tabulate has

tabulate.py
tabulate-0.7.5-py3.5.egg-info

Whereas in my case I have just the following:

 ands-0.0.1a1-py3.5.egg 

Would you know why? Apparently, even if I remove the files that are produced in the root directory after python3.5 setup.py install, I can still import ands normally, so apparently those files are not important..

I know you are not obligated to answer, and thanks anyway!

[nbro]

@BurntSushi Ahm, forgot to say, now the produced documentation with pdoc is what we expected ;)

[BurntSushi]

@Dossan We're really getting into the weeds here, and such details of Python's build system are far beyond my own comprehension. I could take guesses, and they may be right, but I'd rather not. I note that Python's ecosystem has been through several iterations and package formats over the years, and I have long since lost track of what's actually going on. I would recommend seeking out help through other channels, because I alone am quite a poor resource!

I'm glad pdoc is working well though! :-)