Generate docs of project fails with virtual environment dependencies

[ogallagher]

Adapted from the pdocs programmed recursive documentation generation example, I have a script that runs the following methods:

tl_util.py

import pdoc

# documentation

def pdoc_module_path(m:pdoc.Module, ext:str='.html', dir_path:str=DOCS_PATH):
    return os.path.join(
        dir_path, 
        *regex.sub(r'\.html$', ext, m.url()).split('/'))
# end pdoc_module_path

def document_modules(mod:pdoc.Module) -> Generator[Tuple[pdoc.Module,str],None,None]:
    """Generate documentation for pdoc-wrapped module and submodules.
    
    Args:
        mod = pdoc.Module instance
    
    Yields tuple:
        module
        
        cleaned module html
    """
    
    yield (
        mod, 
        mod.html().replace('\u2011','-').replace(u'\xa0', u' ')
    )
    
    for submod in mod.submodules():
        yield from document_modules(submod)
# end document_modules

main.py

from typing import *
import pdoc
import os
import logging
from logging import Logger
import tl_util

log:Logger = logging.getLogger(__name__)

# omitted code

def document(dir_path:str=DOCS_PATH):
    """Recursively generate documentation using pdoc.
    
    Adapted from 
    [pdoc documentation](https://pdoc3.github.io/pdoc/doc/pdoc/#programmatic-usage).
                         
    Args:
        dir_path = documentation output directory; default=`algo_trader.DOCS_PATH`
    """
    
    ctx = pdoc.Context()
    
    modules:List[pdoc.Module] = [
        pdoc.Module(mod)
        for mod in [
            '.' # this script resides within the package that I want to create docs for
        ]
    ]
    
    pdoc.link_inheritance(ctx)
    
    for mod in modules:
        for submod, html in tl_util.document_modules(mod):
            # write to output location
            ext:str = '.html'
            filepath = tl_util.pdoc_module_path(submod, ext, dir_path)
            
            dirpath = os.path.dirname(filepath)
            if not os.access(dirpath, os.R_OK):
                os.makedirs(dirpath)
            
            with open(filepath,'w') as f:
                if ext == '.html':
                    try:
                        f.write(html)
                    except:
                        log.error(traceback.format_exc())
                elif ext == '.md':
                    f.write(mod.text())
            # close f
            
            log.info('generated doc for {} at {}'.format(
                submod.name,
                filepath))
        # end for module_name, html
    # end for mod in modules
# end document

if __name__ == '__main__':
    # omitted logic...
    document()

My project filesystem is like this:

my_package/
    env/
        Lib/
            site-packages/
                <installed dependencies, including pdoc3>
    main.py
    tl_util.py

Below is the error that I currently get:

(env) PS C:\<path>\my_package python .\main.py --document
=== Program Name ===

set logger <RootLogger root (DEBUG)> to level 10
C:\<path>\my_package\env\lib\site-packages\pdoc\__init__.py:643: UserWarning: Module <Module 'my_package
.env.Lib.site-packages.dateutil'> doesn't contain identifier `easter` exported in `__all__`
  warn("Module {!r} doesn't contain identifier `{}` "
Traceback (most recent call last):
  File “.\main.py", line 608, in <module>
    main()
  File “.\main.py", line 469, in main
    document()
  File “.\main.py", line 399, in document
    modules:List[pdoc.Module] = [
  File “.\main.py", line 400, in <listcomp>
    pdoc.Module(mod)
  File "C:\Users\Owen\Documents\my_package\env\lib\site-packages\pdoc\__init__.py", line 708, in __init__
    m = Module(import_module(fullname),
  File "C:\Users\Owen\Documents\my_package\env\lib\site-packages\pdoc\__init__.py", line 708, in __init__
    m = Module(import_module(fullname),
  File "C:\Users\Owen\Documents\my_package\env\lib\site-packages\pdoc\__init__.py", line 708, in __init__
    m = Module(import_module(fullname),
  [Previous line repeated 1 more time]
  File "C:\Users\Owen\Documents\my_package\env\lib\site-packages\pdoc\__init__.py", line 646, in __init__
    obj = inspect.unwrap(obj)
UnboundLocalError: local variable 'obj' referenced before assignment

The referenced installed package __init__.py file for dateutil is as follows:

# -*- coding: utf-8 -*-
try:
    from ._version import version as __version__
except ImportError:
    __version__ = 'unknown'

__all__ = ['easter', 'parser', 'relativedelta', 'rrule', 'tz',
           'utils', 'zoneinfo']

I’ve confirmed that the relevant virtual environment has been activated. I’ve also confirmed that all modules in the dateutil package are where I expect them to be, like so:

dateutil/
    __init__.py
    easter.py
    parser.py
    relativedelta.py
    rrule.py
    tz/
        ...
    utils.py
    zoneinfo/
        ...
    ...

Why is the attempt to document the dateutil dependency failing this way? If documentation of dependencies is not supported, how should I skip them?

Additional info

• pdoc version: 0.9.2

[kernc]

The dateutil thing seems to be just a warning. The real breaking issue is line 646:

pdoc/pdoc/__init__.py

Lines 640 to 646 in 0658ff0

	try:
	obj = getattr(self.obj, name)
	except AttributeError:
	warn("Module {!r} doesn't contain identifier `{}` "
	"exported in `__all__`".format(self.module, name))
	if not _is_blacklisted(name, self):
	obj = inspect.unwrap(obj)

where rhs obj is undefined.

Maybe there's a little continue missing in the except block. 🤔 Can you try that?

[ogallagher]

@kernc Thanks for your quick reply. I did try adding a continue statement where you suggested and execution was able to continue. However, I am quickly realizing that trying to recursively generate docs for all the external dependencies is a nightmare, as some now require new dependencies for their development/private code, and some specify different versions for different versions of python, the earlier of which fail to import for me because I’m using Python 3.

For example, mako requires babel (which I don’t normally need from a usage standpoint), which requires lingua, which requires chameleon, which fails to import because chameleon/py25.py has statements only compatible with Python 2.

With this in mind, I’m requesting guidance for just skipping everything in my virtual environment env/ folder when generating pdoc documentation. I’m aware of the __pdoc__ dictionary, but it seems that I can’t use something like a wildcard to skip modules that I know will fail or that I don’t want to include, like:

from pdoc import __pdoc__

__pdoc__['env'] = False
# or
__pdoc__['env.*'] = False

So far, all I can think of is to move my env/ folder outside of the package folder that I want to document.

[kernc]

move my env/ folder outside of the package folder

I assume that's how everyone else does it.

project_dir/     # <-- git root
    env/
    package/     # <-- actual named, released package
        main.py
        t1_util.py
   ...

Alternatively, appropriately positioned (i.e. in my_package/__init__.py):

__pdoc__ = {}
__pdoc__['env'] = False

(defined anew; not imported) should prevent descending further into env.

[ogallagher]

Alternatively, appropriately positioned (i.e. in my_package/init.py):

__pdoc__ = {}
__pdoc__['env'] = False

(defined anew; not imported) should prevent descending further into env.

@kernc Perfect, thanks! This is what I needed to know. My documentation generation now works as I’d hoped, adding the proper pdoc excludes and also monkey-patching the pdoc.Module constructor. I definitely suggest adding this patch to the next release.

[trimeta]

I'm experiencing a similar issue, also centered around the section with the "Module {!r} doesn't contain identifier {} exported in __all__" warning:

pdoc/pdoc/__init__.py

Lines 679 to 688 in 4aa70de

	if hasattr(self.obj, '__all__'):
	for name in self.obj.__all__:
	try:
	obj = getattr(self.obj, name)
	except AttributeError:
	warn(f"Module {self.module!r} doesn't contain identifier `{name}` "
	"exported in `__all__`")
	if not _is_blacklisted(name, self):
	obj = inspect.unwrap(obj)
	public_objs.append((name, obj))

Ultimately this problem stems from the try/except block being obviated by the blacklist test: if the except triggers, then in addition to the AttributeError being caught and turned into a warning, the blacklist test references a variable (obj) that only exists if the try block was successful, so if you went into the except block you now error out at the blacklist test.

The solution here is to wrap the blacklist test in an else block of the try/except: then it would only run if the try block succeeded. Unless the desired behavior is to always error out if a module referenced in __all__ cannot be imported as an attribute of the base module, in which case the try/except block is unnecessary.