Install of mkdocs fails with nltk 3.5

[klaascuvelier]

Hi,

First of all, thank you for mkdocs. We're using this in our company and are very happy to have it.

I noticed that the installation of mkdocs started failing for us yesterday. After some investigation I found out that the nltk lib had a version bump (3.5) yesterday and that's were it goes wrong.

I was able to use the previous nltk version (3.4.5) by adding it directly to our ci file

pages:
  image: python:3.7-alpine
  stage: 🥾 Bootstrap
  script:
    - pip install nltk==3.4.5
    - pip install mkdocs
    - pip install mkdocs-material
    - mkdocs build

This solves the issue for us.

I am not really used to python and its ecosystem, so perhaps I am overseeing an actual "fix".
But if this is a really issue, I guess the 3.5 nltk version should be blacklisted from the install?

Output of the error

$ pip install mkdocs
18 Collecting mkdocs
19   Downloading mkdocs-1.1-py2.py3-none-any.whl (6.4 MB)
20 Collecting click>=3.3
21   Downloading click-7.1.1-py2.py3-none-any.whl (82 kB)
22 Collecting Jinja2>=2.10.1
23   Downloading Jinja2-2.11.1-py2.py3-none-any.whl (126 kB)
24 Collecting Markdown>=3.2.1
25   Downloading Markdown-3.2.1-py2.py3-none-any.whl (88 kB)
26 Collecting PyYAML>=3.10
27   Downloading PyYAML-5.3.1.tar.gz (269 kB)
28 Collecting tornado>=5.0
29   Downloading tornado-6.0.4.tar.gz (496 kB)
30 Collecting livereload>=2.5.1
31   Downloading livereload-2.6.1-py2.py3-none-any.whl (23 kB)
32 Collecting lunr[languages]==0.5.6
33   Downloading lunr-0.5.6-py2.py3-none-any.whl (36 kB)
34 Collecting MarkupSafe>=0.23
35   Downloading MarkupSafe-1.1.1.tar.gz (19 kB)
36 Requirement already satisfied: setuptools>=36 in /usr/local/lib/python3.8/site-packages (from Markdown>=3.2.1->mkdocs) (46.1.1)
37 Collecting six
38   Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
39 Collecting future>=0.16.0
40   Downloading future-0.18.2.tar.gz (829 kB)
41 Collecting nltk>=3.2.5; extra == "languages"
42   Downloading nltk-3.5.zip (1.4 MB)
43 Collecting joblib
44   Downloading joblib-0.14.1-py2.py3-none-any.whl (294 kB)
45 Collecting regex
46   Downloading regex-2020.4.4.tar.gz (695 kB)
47 Collecting tqdm
48   Downloading tqdm-4.45.0-py2.py3-none-any.whl (60 kB)
49 Building wheels for collected packages: PyYAML, tornado, MarkupSafe, future, nltk, regex
50   Building wheel for PyYAML (setup.py): started
51   Building wheel for PyYAML (setup.py): finished with status 'done'
52   Created wheel for PyYAML: filename=PyYAML-5.3.1-cp38-cp38-linux_x86_64.whl size=44617 sha256=e51f74894350a12a20b5ed28d79e6bf74a837020d884ed06a1e0a0ca03ec28fb
53   Stored in directory: /root/.cache/pip/wheels/13/90/db/290ab3a34f2ef0b5a0f89235dc2d40fea83e77de84ed2dc05c
54   Building wheel for tornado (setup.py): started
55   Building wheel for tornado (setup.py): finished with status 'done'
56   Created wheel for tornado: filename=tornado-6.0.4-cp38-cp38-linux_x86_64.whl size=415149 sha256=5b834e80e999d01b7fb8a67d6ccbf419fae7fb0920b9a8ddcf2d0fade7357613
57   Stored in directory: /root/.cache/pip/wheels/88/79/e5/598ba17e85eccf2626eab62e4ee8452895636cd542650d450d
58   Building wheel for MarkupSafe (setup.py): started
59   Building wheel for MarkupSafe (setup.py): finished with status 'done'
60   Created wheel for MarkupSafe: filename=MarkupSafe-1.1.1-py3-none-any.whl size=12629 sha256=a03aedf2140733fd4e1c3d47f1ffa26335ddd2a349b60e108ceb84e48612998b
61   Stored in directory: /root/.cache/pip/wheels/0c/61/d6/4db4f4c28254856e82305fdb1f752ed7f8482e54c384d8cb0e
62   Building wheel for future (setup.py): started
63   Building wheel for future (setup.py): finished with status 'done'
64   Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491058 sha256=6d781ad44d7230964ab437b2036b2b382186d0083039960587eeb76d1eb3f93b
65   Stored in directory: /root/.cache/pip/wheels/8e/70/28/3d6ccd6e315f65f245da085482a2e1c7d14b90b30f239e2cf4
66   Building wheel for nltk (setup.py): started
67   Building wheel for nltk (setup.py): finished with status 'done'
68   Created wheel for nltk: filename=nltk-3.5-py3-none-any.whl size=1434676 sha256=1f4703ad06f227692f94fed6c4564f56f36b982f86a502fab5af0a2e290f57b6
69   Stored in directory: /root/.cache/pip/wheels/ff/d5/7b/f1fb4e1e1603b2f01c2424dd60fbcc50c12ef918bafc44b155
70   Building wheel for regex (setup.py): started
71   Building wheel for regex (setup.py): finished with status 'error'
72   ERROR: Command errored out with exit status 1:
73    command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7xy4ih14/regex/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7xy4ih14/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-crjf4cl9
74        cwd: /tmp/pip-install-7xy4ih14/regex/
75   Complete output (17 lines):
76   running bdist_wheel
77   running build
78   running build_py
79   creating build
80   creating build/lib.linux-x86_64-3.8
81   creating build/lib.linux-x86_64-3.8/regex
82   copying regex_3/__init__.py -> build/lib.linux-x86_64-3.8/regex
83   copying regex_3/regex.py -> build/lib.linux-x86_64-3.8/regex
84   copying regex_3/_regex_core.py -> build/lib.linux-x86_64-3.8/regex
85   copying regex_3/test_regex.py -> build/lib.linux-x86_64-3.8/regex
86   running build_ext
87   building 'regex._regex' extension
88   creating build/temp.linux-x86_64-3.8
89   creating build/temp.linux-x86_64-3.8/regex_3
90   gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/local/include/python3.8 -c regex_3/_regex.c -o build/temp.linux-x86_64-3.8/regex_3/_regex.o
91   unable to execute 'gcc': No such file or directory
92   error: command 'gcc' failed with exit status 1
93   ----------------------------------------
94   ERROR: Failed building wheel for regex
95   Running setup.py clean for regex
96 Successfully built PyYAML tornado MarkupSafe future nltk
97 Failed to build regex
98 Installing collected packages: click, MarkupSafe, Jinja2, Markdown, PyYAML, tornado, six, livereload, future, joblib, regex, tqdm, nltk, lunr, mkdocs
99     Running setup.py install for regex: started
100     Running setup.py install for regex: finished with status 'error'
101     ERROR: Command errored out with exit status 1:
102      command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7xy4ih14/regex/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7xy4ih14/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-begr9ft9/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/regex
103          cwd: /tmp/pip-install-7xy4ih14/regex/
104     Complete output (17 lines):
105     running install
106     running build
107     running build_py
108     creating build
109     creating build/lib.linux-x86_64-3.8
110     creating build/lib.linux-x86_64-3.8/regex
111     copying regex_3/__init__.py -> build/lib.linux-x86_64-3.8/regex
112     copying regex_3/regex.py -> build/lib.linux-x86_64-3.8/regex
113     copying regex_3/_regex_core.py -> build/lib.linux-x86_64-3.8/regex
114     copying regex_3/test_regex.py -> build/lib.linux-x86_64-3.8/regex
115     running build_ext
116     building 'regex._regex' extension
117     creating build/temp.linux-x86_64-3.8
118     creating build/temp.linux-x86_64-3.8/regex_3
119     gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/local/include/python3.8 -c regex_3/_regex.c -o build/temp.linux-x86_64-3.8/regex_3/_regex.o
120     unable to execute 'gcc': No such file or directory
121     error: command 'gcc' failed with exit status 1
122     ----------------------------------------
123 ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7xy4ih14/regex/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7xy4ih14/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-begr9ft9/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/regex Check the logs for full command output.
127 ERROR: Job failed: exit code 1

[waylan]

First of all, nltk is not a direct dependency. It is a dependency of one of MkDocs' dependencies: yeraydiazdiaz/lunr.py. I would suggest taking this issue upstream.

Second of all, looking at the attached log, it appears that the regex lib is the one which failed to install, not nltk. And regex is not a dependency of MkDocs at all. I don't even have it installed and MkDocs works fine.

[klaascuvelier]

Hi Waylan, thanks for your reply.

I understand that you point me to another dependency, but please follow my reasoning;
Without any changes to the source code of mkdocs, the build starts to fail. The reason appears to be a dependency (of dependency).
It doesn't matter that it might be another dependency using a broken dependency. The end result is, mkdocs is broken and unusable and there is a workaround.

[facelessuser]

All you need to do is pip install mkdocs-material it will require the mkdocs and other dependencies you need.

There seems to be some complaint here about dependencies, but Material uses CI to build docs and it is having no issues. Also, I use mkdocs in CI and am having no issues. This ran today, 8 hours ago, using the latest mkdocs and mkdocs-material: https://github.com/facelessuser/pymdown-extensions/runs/587998198?check_suite_focus=true. No issues.

[waylan]

@klaascuvelier nowhere do we declare nltk to be a dependency. The only place that package is listed as a dependency is in lunr.py's list of dependencies. Presumably lunr.py is broken due to to the upstream change. Therefore, the issue needs so the fixed in lunr.py. When that happens, we will adjust our dependency on lunr.py to require the newer fixed version.

However, as @facelessuser points out, no-one else seems to be experiencing the issue. And as I pointed out, your log seems to indicate the issue is with regex, not nltk. As regex is not a dependency of MkDocs at all, and we can't replicate the problem, this appears to not be something we can address at all.

[froschdesign]

@waylan

…no-one else seems to be experiencing the issue.

That's not true because we had the same problem. See:

• https://github.com/laminas/laminas-di/runs/585289640#step:2:149
• https://github.com/laminas/documentation-theme/blob/255dfe55f64ccdec939740c8dd5a0827e701a7c3/github-actions/docs/Dockerfile#L21-L22

[waylan]

Perhaps I should stated that one one else has reported having the issue. In any event, thanks @froschdesign for providing a second report. However, again the issue is with regex not nltk and I have no idea were the regex dependency is coming from. Did the more recent version of nltk add it as a dependency or something?

[froschdesign]

@waylan

Did the more recent version of nltk add it as a dependency or something?

Yes, with this commit nltk/nltk@6fbad04 for version 3.5:

install_requires=[
    "click",
    "joblib",
    "regex",
    "tqdm",
],

https://github.com/nltk/nltk/blob/6404712d0a64c3d6e3700032c23a59803615460c/setup.py#L103-L108

regex is used instead of re: nltk/nltk@615e72f

[waylan]

Ah, okay. I see to ways forward:

1. Report upstream to lunr.py and they can restrict their dependency.
2. Report upstream to nltk as they clearly have an issue with their dependency requirement.

Actually, it would probably be best to report to both projects. I know I would want to receive the report if I was working on either. When we get some feedback about how they will move forward, that will inform us on how to proceed (do we require an older version or a newer version?)

[codisart]

Hello @waylan,

I add the same problems with our mkdocs build.

It seems to me that it could be possible to have another solution waiting for upstream fixes.
I think that you could commit a "3.4.5" version freeze for nltk.

[klaascuvelier]

I do indeed think that would be a solution.

I understand that it's not directly your library that's breaking something. But by using external libraries (that can break) you can't just point to them and wait for them to fix possible issues.

[klaascuvelier]

☝️ created an MR to hardcode the dependency

[squidfunk]

I just ran into this when preparing the latest release for mkdocs-material. This is how I fixed it.

[facelessuser]

Ah, it's an environment build issue for regex as it is built with C on Linux. I thought there were wheels for it? Maybe not for Linux I guess.

[waylan]

Thank you @squidfunk for providing such a helpful report.

Looking at the regex package, there are wheels (prebuilt files) for the following platforms:

• manylinux1_i686
• manylinux1_x86_64
• manylinux2010_i686
• manylinux2010_x86_64
• win32
• win_amd64

For any other platform, the package will need to be compiled on installation. Turns out this is not a version issue at all. Sure, earlier versions of nltk did not depend in regex, but the fix is not to restrict nltk to an older version (and possibly miss out on bug fixes) but to properly configure your environment.

As win32 and win_amd64 wheels are provided, this is a non-issue on Windows.

For information on manylinux see pypa/manylinux. PEP 513 defines how pip determines if a platform is compatible with manylinux1 and PEP 571 defines the same for manylinux2010. It appears that it would be common for Docker images to need some extra configuration here to make sure pip gets this correct. And that matches the reports we are getting, which are all from within Docker images.

It appears that one solution is to ensure that the appropriate gcc is available (if I understand correctly, its not actually used to compile anything, just to verify which wheel to install from). @squidfunk showed us how to do that.

The other option is to install the manylinux package and ensure the appropriate attribute (_manylinux.manylinux1_compatible or _manylinux.manylinux2010_compatible) in the module is True .

Finally, for completeness, I should note that no MacOS wheels are provided. However, I just checked and it appears that regex compiles and installs without issue on MacOS. I don't believe that I have ever personally built C code in my machine, so this is using the default config.

$ pip install regex
Collecting regex
  Downloading regex-2020.4.4.tar.gz (695 kB)
     |████████████████████████████████| 695 kB 1.4 MB/s
Building wheels for collected packages: regex
  Building wheel for regex (setup.py) ... done
  Created wheel for regex: filename=regex-2020.4.4-cp37-cp37m-macosx_10_12_x86_64.whl size=280684 sha256=c69aa1092a179f61214e020c38ca6fda11b4b657c9ff1b0ff87ae91a093dfec9
  Stored in directory: /Users/waylan/Library/Caches/pip/wheels/52/98/5c/8d5d7a5321d2de463dfc6f7b32aab8b16e564cfce553b28daf
Successfully built regex
Installing collected packages: regex
Successfully installed regex-2020.4.4

[waylan]

Just looked at yeraydiazdiaz/lunr.py and version 0.5.8 was just released which maintains compatibility with lunr.js 2.3.8, but also restricts its dependency to nltk>=3.2.5,<3.5.

As our dependency on lunr.py is hard, we should simply update it to version 0.5.8. That will guarantee no issues with future releases of ntlk and we avoid needing to list ntlk in our dependencies.