improve failure modes for failed imports

[nbro]

I am trying to generate all at once the docs for a project, which contains some packages and subpackages, with modules, and so on. For example, I have a package called sorting, and I would like to generate recursively docs for it and for its modules or subpackages.

Currently I tried:

pdoc --html --html-dir docs sorting

But it produces an empty file, apparently only for the package itself sorting. Any ideas on how to produce recursively the docs for a whole project? I have seen other issues, but not sure if they are talking about the same thing. Of course I could create a script to do it for me, but if pdoc already provides it, it would be nice.

[BurntSushi]

It's hard to say without knowing the structure of sorting. pdoc is built to document the public interface of a module. If the public interface includes sub-modules, then those should get documented.

If you'd like to force the issue, you might try passing --all-submodules.

[nbro]

I tried with --all-modules but it still produces the following index.html under docs/sorting

Actually sorting is a package and not a module, so I don't understand why the produced html's title says "module" after "sorting". Anyway, the idea is that I would like to automate the creation of the documentation recursively, so that I don't have to do it manually for every package or module or subpackage or submodule. I think that if this feature is not present, it would be really nice to add it. Based on the Python's project, i.e. by specifying a __init__.py file for example, you should be able to understand if files are packages or subpackages or modules, etc, and generate the docs recursively...

[BurntSushi]

@Dossan That is what happens. Look at the source code: https://github.com/BurntSushi/pdoc/blob/master/pdoc/__init__.py#L598-L614

It's hard for me to help you further because I don't have a reproducible example. If I had that, I might be able to tell you exactly what's wrong.

[nbro]

@BurntSushi Ok, so this is my project structure:

I am inside a folder (which is also a package) called algorithms, which contains other subpackages, and one of them is called sorting, which has some modules inside it. I would like to create a documentation for the package sorting and all its modules. My current directory is therefore inside algorithms:

algorithms/
    __init__.py
    ...
    sorting/
        ├── README.md
        ├── __init__.py
        ├── bubblesort.py
        ├── heapsort.py
        ...

As I said in the question, I am using the following command from the terminal (I repeat inside algorithms and not sorting):

pdoc --html --html-dir docs sorting

or

pdoc --html --html-dir docs --all-submodules sorting

(both produce the same thing)

Again, what's produced is a folder inside algorithms called docs, which contains another folder called sorting, which contains a file called index.html, which is empty. Ask if you need more information...

Thanks for any help!

[BurntSushi]

It's working fine for me here.

(pdoc)[andrew@Liger algorithms] tree
.
├── __init__.py
└── sorting
    ├── bubblesort.py
    ├── heapsort.py
    └── __init__.py

1 directory, 4 files
(pdoc)[andrew@Liger algorithms] pwd
/home/andrew/tmp/pdoc/91/algorithms
(pdoc)[andrew@Liger algorithms] tree
.
├── __init__.py
└── sorting
    ├── bubblesort.py
    ├── heapsort.py
    └── __init__.py

1 directory, 4 files
(pdoc)[andrew@Liger algorithms] cat sorting/__init__.py 
class Sorting(object):
    pass
(pdoc)[andrew@Liger algorithms] cat sorting/bubblesort.py 
class BubbleSort(object):
    pass
(pdoc)[andrew@Liger algorithms] cat sorting/heapsort.py 
class HeapSort(object):
    pass
(pdoc)[andrew@Liger algorithms] pdoc --html --html-dir docs sorting
(pdoc)[andrew@Liger algorithms] tree docs/
docs/
└── sorting
    ├── bubblesort.m.html
    ├── heapsort.m.html
    └── index.html

1 directory, 3 files

Note that --all-submodules wasn't required since pdoc picked up on the public interface automatically.

The next step is to probably show code in your Python files.

[nbro]

@BurntSushi I have just done a commit and push to the github repo here:

https://github.com/dossan/ands

You find algorithms immediately as a subpackage of ands...

[BurntSushi]

I was able to reproduce your issue. TL;DR - The exact circumstances that cause sys.path to be modified are unknown by me. Your specific issue can be fixed by setting PYTHONPATH=..

The problem, as I identified it, was that (for example) __import__('sorting.bubblesort') was failing inside of pdoc, but running python -c '__import__("sorting.bubblesort")' executed successfully. It seems that the specific issue is that in the former case, sys.path does not contain the current directory where as in the latter case it does. I narrowed it down to simply printing the contents of sys.path:

(pdoc) [andrew@Serval algorithms] cat sys-path
#!/usr/bin/env python

import sys
print(sys.path)

Executing this in your algorithms directory shows why imports work through executing the interpreter normally:

(pdoc) [andrew@Serval algorithms] ./sys-path
['/tmp/ands/algorithms', '/tmp/pdoc', '/home/andrew/src/python/virts/pdoc/lib/python35.zip', '/home/andrew/src/python/virts/pdoc/lib/python3.5', '/home/andrew/src/python/virts/pdoc/lib/python3.5/plat-linux', '/home/andrew/src/python/virts/pdoc/lib/python3.5/lib-dynload', '/usr/lib64/python3.5', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-linux', '/home/andrew/src/python/virts/pdoc/lib/python3.5/site-packages']

I then copied sys-path to /home/andrew/src/python/virts/pdoc/bin, which is precisely the location that the pdoc script lives, and then re-ran it:

(pdoc) [andrew@Serval algorithms] cp ./sys-path ~/src/python/virts/pdoc/bin/
(pdoc) [andrew@Serval algorithms] /home/andrew/src/python/virts/pdoc/bin/sys-path
['/home/andrew/src/python/virts/pdoc/bin', '/tmp/pdoc', '/home/andrew/src/python/virts/pdoc/lib/python35.zip', '/home/andrew/src/python/virts/pdoc/lib/python3.5', '/home/andrew/src/python/virts/pdoc/lib/python3.5/plat-linux', '/home/andrew/src/python/virts/pdoc/lib/python3.5/lib-dynload', '/usr/lib64/python3.5', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-linux', '/home/andrew/src/python/virts/pdoc/lib/python3.5/site-packages']

Note the changed sys.path! I personally don't grok the algorithm that produces sys.path. The behavior seems funny here, but I'm willing to believe something to do with virtualenv is doing this explicitly for probably good reasons. And in general, the changed sys.path is probably a good thing, because it reduces conflicts with the current working directory. Unfortunately, in pdoc's case, this is exactly what we don't want.

Setting PYTHONPATH=. will fix your specific problem, because it will cause the current working directory to appear in sys.path somewhere. This now works:

(pdoc) [andrew@Serval algorithms] pdoc --overwrite --html --html-dir docs sorting
(pdoc) [andrew@Serval algorithms] tree docs/
docs/
└── sorting
    └── index.html

1 directory, 1 file
(pdoc) [andrew@Serval algorithms] PYTHONPATH=. pdoc --overwrite --html --html-dir docs sorting
(pdoc) [andrew@Serval algorithms] tree docs
docs
└── sorting
    ├── bubblesort.m.html
    ├── heapsort.m.html
    ├── index.html
    ├── insertionsort.m.html
    ├── mergesort.m.html
    ├── quicksort.m.html
    ├── selectionsort.m.html
    └── util.m.html

1 directory, 8 files

I note that these issues are trivially avoided if you use an idiomatic Python project structure. In this case, that would mean having a setup.py and doing development in a virtualenv (pip install -e . makes this convenient).

The major failing of pdoc here is that tracking down this problem was hard. There are no clear signals that something has gone wrong or an import has failed. This is because pdoc specifically suppresses failed imports, but perhaps that is the wrong behavior. I did it that way initially during development because there are so many Python packages that try to execute code or print things to stdout on import, but it seems rather unfortunate to sacrifice good failure modes just for projects without good hygiene. (As ridiculous as it may sound, one package was trying to read from sys.stdin on import, which of course will cause pdoc to hang indefinitely!)

As such, I've modified your issue title to address the bad failure mode. Thanks!