AttributeError: module 'importlib' has no attribute 'util' with python-markdown 3.4 on macOS/Windows

[mikepqr]

With python3.9 on macOS:

$ python3.9 -m venv venv
$ source venv/bin/activate
$ pip install markdown
Collecting markdown
  Using cached Markdown-3.4-py3-none-any.whl (93 kB)
Collecting importlib-metadata>=4.4; python_version < "3.10"
  Using cached importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Collecting zipp>=0.5
  Using cached zipp-3.8.1-py3-none-any.whl (5.6 kB)
Installing collected packages: zipp, importlib-metadata, markdown
Successfully installed importlib-metadata-4.12.0 markdown-3.4 zipp-3.8.1
WARNING: You are using pip version 20.2.3; however, version 22.1.2 is available.
You should consider upgrading via the '/Users/mike/tmp/resume.md/venv/bin/python3.9 -m pip install --upgrade pip' command.
$ python
Python 3.9.4 (default, Apr 16 2021, 21:18:07)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import markdown
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mike/tmp/resume.md/venv/lib/python3.9/site-packages/markdown/__init__.py", line 22, in <module>
    from .core import Markdown, markdown, markdownFromFile
  File "/Users/mike/tmp/resume.md/venv/lib/python3.9/site-packages/markdown/core.py", line 27, in <module>
    from .preprocessors import build_preprocessors
  File "/Users/mike/tmp/resume.md/venv/lib/python3.9/site-packages/markdown/preprocessors.py", line 29, in <module>
    from .htmlparser import HTMLExtractor
  File "/Users/mike/tmp/resume.md/venv/lib/python3.9/site-packages/markdown/htmlparser.py", line 29, in <module>
    spec = importlib.util.find_spec('html.parser')
AttributeError: module 'importlib' has no attribute 'util'
>>>

With python3.10 on macOS:

$ python3.10 -m venv 3.10
$ source 3.10/bin/activate
$ pip install markdown
Collecting markdown
  Using cached Markdown-3.4-py3-none-any.whl (93 kB)
Installing collected packages: markdown
Successfully installed markdown-3.4
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/Users/mike/tmp/resume.md/3.10/bin/python3.10 -m pip install --upgrade pip' command.
$ python
Python 3.10.3 (main, Mar 25 2022, 22:16:41) [Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import markdown
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mike/tmp/resume.md/3.10/lib/python3.10/site-packages/markdown/__init__.py", line 22, in <module>
    from .core import Markdown, markdown, markdownFromFile
  File "/Users/mike/tmp/resume.md/3.10/lib/python3.10/site-packages/markdown/core.py", line 27, in <module>
    from .preprocessors import build_preprocessors
  File "/Users/mike/tmp/resume.md/3.10/lib/python3.10/site-packages/markdown/preprocessors.py", line 29, in <module>
    from .htmlparser import HTMLExtractor
  File "/Users/mike/tmp/resume.md/3.10/lib/python3.10/site-packages/markdown/htmlparser.py", line 29, in <module>
    spec = importlib.util.find_spec('html.parser')
AttributeError: module 'importlib' has no attribute 'util'

pip install "markdown<3.4" works, so this is perhaps a regression in the 3.4 release?

[mikepqr]

Note this is also affecting GitHub actions on macOS and Windows (but not Ubuntu): https://github.com/mikepqr/resume.md/actions/runs/2678077124.

[facelessuser]

What version of importlib-metadata are you using, we can see here that Python Markdown requires: importlib-metadata>=4.4 for Python < 3.10.

[mikepqr]

These are installations into fresh virtual environments with pip install python-markdown. I therefore get whatever packages markdown declares as dependencies.

As you can see above, that results in:

Successfully installed importlib-metadata-4.12.0 markdown-3.4 zipp-3.8.1

in python 3.9, and just

Successfully installed markdown-3.4

in python 3.10.

Note also that I get the same error on python3.10, which has no importlib-metadata dependency.

[facelessuser]

I wonder if something changed in 4.12.0...I'm on 4.11.3 on Python 3.10 and I wasn't experiencing any issues. Tests were run on Py3.10 as well with no issues: https://github.com/Python-Markdown/markdown/runs/7359376261?check_suite_focus=true. If you downgrade importlib-metadata to 4.11.X do you still get the issue?

[mikepqr]

Away from test environment now, will check later. But downgrading importlib-metadata on 3.9 won't help on python 3.10. It seems like the current release of python-markdown is broken out of the box on 3.10 on macOS and Windows. Looks like the CI suite only covers Linux?

[facelessuser]

Hmm, I'm running macOS and Python 3.10 importlib-metadata 4.12.0 and Python Markdown 3.4 with no issues. I cannot seem to reproduce...

[facelessuser]

Is this running with a brew python or an official python distribution?

[mikepqr]

Locally it happens to be running in pyenv-installed fresh builds of official python 3.9.6 and 3.10.3, and fresh virtual environments. brew is not involved.

But please note I am seeing the exact same error in Github actions on macOS and Windows.

[facelessuser]

I understand, but there is something else going on here. I should be able to reproduce this issue on macOS...I'll have to spend some time this evening and see isolate things in a virtual environment. I should note that I'm on Python 3.10.4. Not sure if that makes a difference either.

[mikepqr]

Thanks! I will also try to narrow this down more later.

[waylan]

The code which is raising this error has not changed in two years and was part of the 3.3.0 release. There were no changes in any release from 3.3.1 through 3.3.7. If it works in any of those but not in 3.4.0, then the issue is something other than our code.

When you do pip install "markdown<3.4", which version do you get?

[mikepqr]

3.3.7.

[mikepqr]

Make an empty repo with workflow to attempt to bisect.

Here's what happens with pip install markdown (i.e. import markdown @ 3.4.0 fails on macOS and Windows with AttributeError: module 'importlib' has no attribute 'util').

Here's the same action with python-markdown pinned to 3.3.7 (works on all tested platforms).

[facelessuser]

I also have plenty of Windows actions run with Python 3.9 and 3.10 with the version right before Markdown 3.4: https://github.com/facelessuser/pymdown-extensions/actions/runs/2652028392. So I am hesitant to think there is an issue.

[waylan]

we can see here that Python Markdown requires: importlib-metadata>=4.4 for Python < 3.10.

Actually, we don't require importlib-metadata at all for Python 3.10, only for any previous Python version. It was my understanding that by default, Python 3.10 was supposed to match importlib-metadata version 4.4 or later. Perhaps that is not true in some cases. Although, it seems strange that an import path for a standard lib would only exist on some platforms.

[facelessuser]

Sorry, I may have stated it backward.

[waylan]

@mikepqr if you install importlib-metadata>=4.4 in python 3.10 does that resolve the issue for you?

[facelessuser]

I can confirm on macOS that util does exist. I just don't have 3.9 on this machine anymore to test that.

Python 3.10.4 (v3.10.4:9d38120e33, Mar 23 2022, 17:29:05) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util
<module 'importlib.util' from '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/util.py'>
>>>

[mikepqr]

I also have plenty of Windows actions run with Python 3.9 and 3.10 with the version right before Markdown 3.4

Do these work with markdown 3.4 though?

[mikepqr]

@mikepqr if you install importlib-metadata>=4.4 in python 3.10 does that resolve the issue for you?

I can check later, but I feel like 3.9 vs. 3.10 is probably a red herring, given the error is the same on both versions.

[facelessuser]

Do these work with markdown 3.4 though?

We'll know shortly: https://github.com/facelessuser/pymdown-extensions/actions/runs/2678476922

[mikepqr]

is this relevant, i.e. are you trying to access importlib.util as an attribute of importlib, rather than by importing importlib.util?

[mikepqr]

Looks like you are?

[facelessuser]

It's a submodule, nothing is wrong with that:

Python 3.10.4 (v3.10.4:9d38120e33, Mar 23 2022, 17:29:05) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util.find_spec
<function find_spec at 0x108703d00>

[facelessuser]

Actions passed on py3.9 and py3.10 on Windows. I don't test in CI on mac, but as I stated, I cannot reproduce locally on my mac.

[mikepqr]

🤷🏻

Configured the action to just access importlib.util.find_spec. Same error.

And locally in fresh environments:

$ python
Python 3.9.6 (default, Jul  5 2021, 13:23:11)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util.find_spec
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'

$ python
Python 3.10.3 (main, Mar 25 2022, 22:16:41) [Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util.find_spec
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'

[facelessuser]

There is something very unique about your environment and how you are doing things. I have many, many projects that rely on Markdown for their documentation. I've demonstrated that PY3.10 does in fact access util just fine along with current actions indicating that to be the case as well.

I do not deny you are having an issue, but I cannot reproduce it in any way. You appear to be an outlier. Maybe try using actions/setup-python@v2 for Python actions 🤷🏻.

[mikepqr]

It's a submodule, nothing is wrong with that

I don't think that's generally true. It depends on how importlib/__init__.py is set up. You get importlib.util in your namespace iff that __init__.py explicitly gives it to you:

$ tree
.
└── package
    ├── __init__.py
    └── util.py

2 directories, 3 files
$ cat package/__init__.py
$ python3
Python 3.9.12 (main, Mar 26 2022, 15:51:15)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import package
>>> package.util
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'package' has no attribute 'util'
>>> import package.util
>>> package.util
<module 'package.util' from '/Users/mleewilliams/tmp/package/package/util.py'>

vs.

$ cat package/__init__.py
import package.util

$ python3
Python 3.9.12 (main, Mar 26 2022, 15:51:15)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import package
>>> package.util
<module 'package.util' from '/Users/mleewilliams/tmp/package/package/util.py'>
>>>

I'm wondering if a change in importlib/__init__.py explains what we're seeing?

[mikepqr]

There is something very unique about your environment and how you are doing things.

eh, I'm validating this behavior by doing pip install markdown; python -c "import markdown" with the default version of python provided in the Windows and macOS Github CI runners, and I found out about this problem because a user reported it to me.

Maybe try using actions/setup-python@v2 for Python actions

I will try later.

[mikepqr]

Following the advice on https://bugs.python.org/issue41958 that describes import importlib; importlib.util.find_spec as user error:

$ git clone https://github.com/Python-Markdown/markdown/
Cloning into 'markdown'...
remote: Enumerating objects: 10709, done.
remote: Counting objects: 100% (507/507), done.
remote: Compressing objects: 100% (237/237), done.
remote: Total 10709 (delta 357), reused 360 (delta 265), pack-reused 10202
Receiving objects: 100% (10709/10709), 3.41 MiB | 15.57 MiB/s, done.
Resolving deltas: 100% (7001/7001), done.

$ cd markdown/

$ python3 -m venv venv

$ source venv/bin/activate

$ python
Python 3.9.12 (main, Mar 26 2022, 15:51:15)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import markdown
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mleewilliams/tmp/markdown/markdown/__init__.py", line 22, in <module>
    from .core import Markdown, markdown, markdownFromFile
  File "/Users/mleewilliams/tmp/markdown/markdown/core.py", line 27, in <module>
    from .preprocessors import build_preprocessors
  File "/Users/mleewilliams/tmp/markdown/markdown/preprocessors.py", line 29, in <module>
    from .htmlparser import HTMLExtractor
  File "/Users/mleewilliams/tmp/markdown/markdown/htmlparser.py", line 29, in <module>
    spec = importlib.util.find_spec('html.parser')
AttributeError: module 'importlib' has no attribute 'util'
>>>

$ vim markdown/htmlparser.py

$ git diff
diff --git a/markdown/htmlparser.py b/markdown/htmlparser.py
index 7ca858e..3512d1a 100644
--- a/markdown/htmlparser.py
+++ b/markdown/htmlparser.py
@@ -20,7 +20,7 @@ License: BSD (see LICENSE.md for details).
 """

 import re
-import importlib
+import importlib.util
 import sys

$ python
Python 3.9.12 (main, Mar 26 2022, 15:51:15)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import markdown
>>>

[waylan]

Okay, I can reproduce the error sometimes, but not every time.

>>> import importlib
>>> importlib.util
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
>>> from importlib import util
>>> importlib.util
<module 'importlib.util' from 'C:\\Users\\wlimberg\\AppData\\Local\\Programs\\Python\\Python38\\lib\\importlib\\util.py'>

A few things to note:

1. This is Python 3.8 on Windows. Not one of the reported problem environments (but the only one I have access to at the moment).
2. I can run Markdown just fine in this environment.
3. The error only occurs if I do the imports in a certain order. Once util is imported directly, then it can be referenced from importlib.util, but not before.

However, I can do this with no errors:

from importlib.util import find_spec, module_from_spec

@mikepqr I'm wondering if this change will work for you:

diff --git a/markdown/htmlparser.py b/markdown/htmlparser.py
index 7ca858e..cfdc49c 100644
--- a/markdown/htmlparser.py
+++ b/markdown/htmlparser.py
@@ -20,14 +20,14 @@ License: BSD (see LICENSE.md for details).
 """

 import re
-import importlib
 import sys

+from importlib.util import find_spec, module_from_spec

 # Import a copy of the html.parser lib as `htmlparser` so we can monkeypatch it.
 # Users can still do `from html import parser` and get the default behavior.
-spec = importlib.util.find_spec('html.parser')
-htmlparser = importlib.util.module_from_spec(spec)
+spec = find_spec('html.parser')
+htmlparser = module_from_spec(spec)
 spec.loader.exec_module(htmlparser)
 sys.modules['htmlparser'] = htmlparser

It seems to work for me.

I can't understand why this makes a difference. And weirdly, now I suddenly can't get Markdown to work without the change when I import it from within a Python session. However, when I call Markdown from the command line (python -m markdown), I get no error.

[mikepqr]

Okay, I can reproduce the error sometimes, but not every time.

I think the difference you're seeing is because import importlib imports importlib/__init__.py (which does not (always?) add util to its namespace) and from importlib import util imports importlib/util.py. See this comment for more.

The error only occurs if I do the imports in a certain order. Once util is imported directly, then it can be referenced from importlib.util, but not before.

The order is not the issue. It's whether or not you do import importlib.util (or, equivalently, from importlib import util).

importlib/__init__.py is very, very complicated, and I can't figure out what it's doing at all. It's certainly doing some weird stuff. But importing importlib.util directly circumvents this complexity entirely so yes, that diff would fix it for me.

So would the diff I posted, which matches what Markdown already does in its setup.py.

I suggest you add that change, assuming it doesn't break already working environments?! (Although I totally understand if you want to get to the root cause of this before landing.)

[waylan]

diff --git a/markdown/htmlparser.py b/markdown/htmlparser.py
index 7ca858e..3512d1a 100644
--- a/markdown/htmlparser.py
+++ b/markdown/htmlparser.py
@@ -20,7 +20,7 @@ License: BSD (see LICENSE.md for details).
 """

 import re
-import importlib
+import importlib.util
 import sys

That also resolves it for me. There is some weird funny-business going on here. Why do I need the fix from the Python prompt, but not when calling a script from the command line? And why did this suddenly start happening today, but I've never encountered it before. And why does downgrading Markdown make a difference when there is no difference to the offending code? So weird.

[waylan]

So would the diff I posted, which matches what Markdown already does in its setup.py.

Well, I guess that should be the route we take to fix it then. I still have no idea why the error didn't appear before now or why it suddenly started appearing now.

[mikepqr]

Me neither!

[facelessuser]

Well, this has been a weird and interesting bug. Nice to see a resolution.

[mikepqr]

Well, I can't promise the fix doesn't break currently working environments, given that it seems like none of us understand the problem! But the fix unblocks me, so I'm all for it ;-)

[mitya57]

Given that Python documentation has an example of importing importlib.util, I don't think it can break any currently working environment. Of course, unless someone relied on us raising AttributeError :)

[mikepqr]

My builds are green again. Thanks folks!