Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.9.0 errors for imports related to type declarations #245

Closed
brizental opened this issue Aug 27, 2020 · 8 comments
Closed

v0.9.0 errors for imports related to type declarations #245

brizental opened this issue Aug 27, 2020 · 8 comments
Labels
bug Something isn't working

Comments

@brizental
Copy link

brizental commented Aug 27, 2020

Expected Behavior

Should build the documentation normally

Actual Behavior

Doesn't build and errors with a NameError

Steps to Reproduce

I have managed to isolate this into a small example. Consider this:

foo.py

from typing import Set, TYPE_CHECKING

if TYPE_CHECKING:
    from .bar import BarClass

class Foo:
    _baz: Set["BarClass"] = set()

bar.py

from foo import Foo

class BarClass:
    _woop = Foo()

If you try to create the docs for these files, calling pdoc3 --html foo bar, you will get NameError: name 'BarClass' is not defined.

I tried adding if typing.TYPE_CHECKING conditional around this line:

pdoc/pdoc/__init__.py

Lines 906 to 907 in 532d9a3

# Add members for which type annotations exist
names.update(_get_type_hints(obj).keys())

and it works. I'd be happy to send a PR with the fix.

Additional info

  • pdoc version: 0.9.0
  • python version: 3.8.3
@ghost
Copy link

ghost commented Aug 27, 2020

Can reproduce with attrs.

from __future__ import annotations

import typing

import attr

if typing.TYPE_CHECKING:
    import datetime

...

@attr.s(eq=True, hash=False, init=True, kw_only=True, slots=True, weakref_slot=False)
class ActivityTimestamps:
    """The datetimes for the start and/or end of an activity session."""

    start: typing.Optional[datetime.datetime] = attr.ib(repr=True)
    """When this activity's session was started, if applicable."""

    end: typing.Optional[datetime.datetime] = attr.ib(repr=True)
    """When this activity's session will end, if applicable."""

Offending code appears to be in 0eb6777 that is raising this:

Traceback (most recent call last):
  File "docs/patched_pdoc.py", line 9, in <module>
    main()
  File "docs/patched_pdoc.py", line 5, in main
    cli.main()
  File "/home/me/code/hikari/.nox/pdoc/lib/python3.8/site-packages/pdoc/cli.py", line 537, in main
    modules = [pdoc.Module(module, docfilter=docfilter,
  File "/home/me/code/hikari/.nox/pdoc/lib/python3.8/site-packages/pdoc/cli.py", line 537, in <listcomp>
    modules = [pdoc.Module(module, docfilter=docfilter,
  File "/home/me/code/hikari/.nox/pdoc/lib/python3.8/site-packages/pdoc/__init__.py", line 640, in __init__
    self.doc[name] = Class(name, self, obj)
  File "/home/me/code/hikari/.nox/pdoc/lib/python3.8/site-packages/pdoc/__init__.py", line 952, in __init__
    for _name, obj in _getmembers_all(self.obj)
  File "/home/me/code/hikari/.nox/pdoc/lib/python3.8/site-packages/pdoc/__init__.py", line 907, in _getmembers_all
    names.update(_get_type_hints(obj).keys())
  File "/usr/lib/python3.8/typing.py", line 1232, in get_type_hints
    value = _eval_type(value, base_globals, localns)
  File "/usr/lib/python3.8/typing.py", line 270, in _eval_type
    return t._evaluate(globalns, localns)
  File "/usr/lib/python3.8/typing.py", line 518, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
NameError: name 'datetime' is not defined

I would possibly argue that pdoc needs to be setting typing.TYPE_CHECKING to True for each module being imported anyway, since pdoc is pretty much playing the role of a type checker that only reads the type hints, but doesn't validate them.

Another solution may be to parse the AST and work out what imports are available. in the scope you are observing annotations from. That way, you could link to other modules without having to physically import them as part of parsing the file. This would mean that side effects that the use of typing.TYPE_CHECKING was created to avoid, such as circular dependency issues, could be avoided if it became an issue, since you could resolve the type hints after the module had completed importing.

ghost pushed a commit to hikari-py/hikari that referenced this issue Aug 27, 2020
Currently failing to build from pdoc issue pdoc3/pdoc#245, so this
will need to be updated when that issue gets resolved or reverted.
ghost pushed a commit to hikari-py/hikari that referenced this issue Aug 27, 2020
Currently failing to build from pdoc issue pdoc3/pdoc#245, so this
will need to be updated when that issue gets resolved or reverted.
@davfsa
Copy link
Contributor

davfsa commented Aug 27, 2020

@kernc mind adding the bug label and assigning this to me? I will have a look

@kernc
Copy link
Member

kernc commented Aug 27, 2020

Simply wrapping the lines:

pdoc/pdoc/__init__.py

Lines 906 to 907 in 532d9a3

# Add members for which type annotations exist
names.update(_get_type_hints(obj).keys())

in if False would just bring back issue #210. Setting typing.TYPE_CHECKING is required in your cases. I think where it's set now should certainly cover the CLI-usage case.

If someone can confirm the master is fixed before a patch release pronto, that'd be awesome. 😁

@davfsa
Copy link
Contributor

davfsa commented Aug 27, 2020

What you did doesnt really fix it, since the whole point of typing.TYPE_CHECKING is for it to fix circular imports. In our project, this has caused them popping up (from libs we depend on) when trying to generate the documentation:

Traceback (most recent call last):
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/pdoc/__init__.py", line 216, in import_module
    module = importlib.import_module(module_path)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/davfsa/coding/hikari/hikari/__init__.py", line 44, in <module>
    from hikari.applications import *
  File "/home/davfsa/coding/hikari/hikari/applications.py", line 42, in <module>
    from hikari import files
  File "/home/davfsa/coding/hikari/hikari/files.py", line 59, in <module>
    import aiohttp.client
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/aiohttp/__init__.py", line 6, in <module>
    from .client import BaseConnector as BaseConnector
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/aiohttp/client.py", line 32, in <module>
    from . import hdrs, http, payload
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/aiohttp/http.py", line 7, in <module>
    from .http_parser import HeadersParser as HeadersParser
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/aiohttp/http_parser.py", line 24, in <module>
    from .http_writer import HttpVersion, HttpVersion10
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/aiohttp/http_writer.py", line 10, in <module>
    from .abc import AbstractStreamWriter
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/aiohttp/abc.py", line 26, in <module>
    from .web_request import BaseRequest, Request
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/aiohttp/web_request.py", line 32, in <module>
    from .abc import AbstractStreamWriter
ImportError: cannot import name 'AbstractStreamWriter' from partially initialized module 'aiohttp.abc' (most likely due to a circular import) (/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/aiohttp/abc.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docs/patched_pdoc.py", line 9, in <module>
    main()
  File "docs/patched_pdoc.py", line 5, in main
    cli.main()
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/pdoc/cli.py", line 537, in main
    modules = [pdoc.Module(module, docfilter=docfilter,
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/pdoc/cli.py", line 537, in <listcomp>
    modules = [pdoc.Module(module, docfilter=docfilter,
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/pdoc/__init__.py", line 590, in __init__
    module = import_module(module)
  File "/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/pdoc/__init__.py", line 218, in import_module
    raise ImportError('Error importing {!r}: {}: {}'
ImportError: Error importing 'hikari': ImportError: cannot import name 'AbstractStreamWriter' from partially initialized module 'aiohttp.abc' (most likely due to a circular import) (/home/davfsa/coding/hikari/.nox/pdoc/lib/python3.8/site-packages/aiohttp/abc.py)

I think what @nekokatt suggested is the best approach for this. I think this might need a rewrite of how you get typehints

kernc added a commit that referenced this issue Aug 27, 2020
Don't set `typing.TYPE_CHECKING` as it can result in circular imports.
See #245 (comment)

Instead, fallback to raw annotation strings when other,
eval'ing methods fail.

Ref: #245
@kernc
Copy link
Member

kernc commented Aug 27, 2020

@davfsa Ack. How about now with 77b83a9?

kernc added a commit that referenced this issue Aug 27, 2020
Don't set `typing.TYPE_CHECKING` as it can result in circular imports.
See #245 (comment)

Instead, fallback to raw annotation strings when other,
eval'ing methods fail.

Ref: #245
@davfsa
Copy link
Contributor

davfsa commented Aug 27, 2020

Seems to work better now but I found a bug that I will probably open as another issue:

import attrs
import typing

from hikari.utilities import attr_extensions

@typing.final
@attr_extensions.with_copy
@attr.s(frozen=True, kw_only=True)
class ConsolePalette:
    """Data class containing printable escape codes for colouring console output."""

    default: str = attr.ib(default="")
    bright: str = attr.ib(default="")
    underline: str = attr.ib(default="")
    invert: str = attr.ib(default="")
    red: str = attr.ib(default="")
    green: str = attr.ib(default="")
    yellow: str = attr.ib(default="")
    blue: str = attr.ib(default="")
    magenta: str = attr.ib(default="")
    cyan: str = attr.ib(default="")
    white: str = attr.ib(default="")
    bright_red: str = attr.ib(default="")
    bright_green: str = attr.ib(default="")
    bright_yellow: str = attr.ib(default="")
    bright_blue: str = attr.ib(default="")
    bright_magenta: str = attr.ib(default="")
    bright_cyan: str = attr.ib(default="")
    bright_white: str = attr.ib(default="")
    framed: str = attr.ib(default="")
    dim: str = attr.ib(default="")

seems to cause

blue AttributeError type object 'ConsolePalette' has no attribute 'blue'
bright AttributeError type object 'ConsolePalette' has no attribute 'bright'
bright_blue AttributeError type object 'ConsolePalette' has no attribute 'bright_blue'
bright_cyan AttributeError type object 'ConsolePalette' has no attribute 'bright_cyan'
bright_green AttributeError type object 'ConsolePalette' has no attribute 'bright_green'
bright_magenta AttributeError type object 'ConsolePalette' has no attribute 'bright_magenta'
bright_red AttributeError type object 'ConsolePalette' has no attribute 'bright_red'
bright_white AttributeError type object 'ConsolePalette' has no attribute 'bright_white'
bright_yellow AttributeError type object 'ConsolePalette' has no attribute 'bright_yellow'
cyan AttributeError type object 'ConsolePalette' has no attribute 'cyan'
default AttributeError type object 'ConsolePalette' has no attribute 'default'
dim AttributeError type object 'ConsolePalette' has no attribute 'dim'
framed AttributeError type object 'ConsolePalette' has no attribute 'framed'
green AttributeError type object 'ConsolePalette' has no attribute 'green'
invert AttributeError type object 'ConsolePalette' has no attribute 'invert'
magenta AttributeError type object 'ConsolePalette' has no attribute 'magenta'
red AttributeError type object 'ConsolePalette' has no attribute 'red'
underline AttributeError type object 'ConsolePalette' has no attribute 'underline'
white AttributeError type object 'ConsolePalette' has no attribute 'white'
yellow AttributeError type object 'ConsolePalette' has no attribute 'yellow'

@kernc
Copy link
Member

kernc commented Aug 27, 2020

Any tracebacks? Better open a separate issue. 👍 Seems remotely related, but could also just be a decorator issue.

Seems to work better now

Good enough for a release, you think? 🙏 That, or the alternative to revert 0eb6777 which has caused it all.

@davfsa
Copy link
Contributor

davfsa commented Aug 27, 2020

Looks good to me, also, found the issue that I pointed above looks to be with our lib, not pdoc 😅

ghost pushed a commit to hikari-py/hikari that referenced this issue Aug 28, 2020
Currently failing to build from pdoc issue pdoc3/pdoc#245, so this
will need to be updated when that issue gets resolved or reverted.

Added missing slot to BotAware.

Froze isort to v5.4.2

Reformatted traits.
@kernc kernc added the bug Something isn't working label Sep 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Development

Successfully merging a pull request may close this issue.

3 participants