-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Installing black 19.10b0 without gcc installed causes an error #1112
Comments
It seems black's regex dependency fails to build without gcc, here's reproducing in ubuntu in virtualenvs (this is where it broke in our systems):
output:
Including python3-dev headers and build-essential makes the build succeed:
output:
|
I wouldn't say it's a bug, but it's definitely an inconvenience which might have an actual impact on the global docker related traffic. |
This makes using black pretty hard on modern CI/CD which usually uses docker. Can we please find a better tool other than regex or MR something into that library, so that it builds binary linux wheels |
The |
Instead of looking for an alternative to |
Since I've been involved with some wheel building platforms I just wanted to chime in and say that it's a non-zero amount of effort to maintain such systems and it's totally okay for the author of regex to decline doing that. And IMO it's not super fair to reply to the legitimate concern about this new requirement of the C compiler that we should simply help build this non-trivial system either. I haven't seen efforts to look for alternatives (in the public issues/PRs at least) that would be vastly less time-intensive. @zsol Do you know of any work in that direction? |
I don't think anyone has spent time looking at regex alternatives while keeping binary wheel availability in mind. I certainly haven't. I'd like to clarify - I don't think it's unacceptable to replace Black's dependency on |
I ran into this issue in an image that uses
I didn't need to install all of |
This was recently solved for pipx's CI in pypa/pipx#269. pipx uses travis, and the fix was to add the path |
Builds are often containerized, and I doubt that many containers have /usr/bin/gcc by default. I don't know what |
@impredicative regex has better Unicode support. Using regex fixed a bug where Black couldn't deal with certain Unicode identifiers. |
If an example of the problematic Unicode pattern and input string is known, perhaps it could be shared here so someone could report it to https://bugs.python.org/ in the hopes that is could be improved in the re module. |
Black seems to have dependencies on musl-dev and gcc past 19.3b0 version. Hopefully this is just temporary. If we wanted to continue using the latest versions, we could instead add something like `apk add musl-dev gcc` but this increases the duration of the process significantly. Fixes following CI check failure[1]: $ make ci-py-checks -f Makefile.ci python3 -m venv /builds/openwrtorg/gitlab-evaluation/.venv/openwrt-ci ... /.venv/openwrt-ci/bin/pip3 install black flake8 ... Installing collected packages: regex, click, attrs, appdirs, toml, typed-ast, pathspec, black, pycodestyle, mccabe, entrypoints, pyflakes, flake8 Running setup.py install for regex: started Running setup.py install for regex: finished with status 'error' ... gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/builds/openwrtorg/gitlab-evaluation/.venv/openwrt-ci/include -I/usr/include/python3.7m -c regex_3/_regex.c -o build/temp.linux-x86_64-3.7/regex_3/_regex.o unable to execute 'gcc': No such file or directory error: command 'gcc' failed with exit status 1 1. https://gitlab.com/ynezz/openwrt-gitlab-evaluation/-/jobs/326228113 Ref: psf/black#1112 Signed-off-by: Petr Štetiar <ynezz@true.cz>
Recent versions of black require gcc to build a new regex dependency. This may go away, so we'll just pin to a specific version black (which is best practice anyway). Related to psf/black#1112.
I added tests for the functions in __main___. Also added pytest-mock as depedencies. I had to pin black to version 19.3b0 because the newer version comes with regex depedency that require a C compiler. Known issue with black: psf/black#1112 Signed-off-by: Even Wiik Thomassen <even.thomassen@cognite.com>
…ILE (#103) * Configure config file path with environment variable COGNITE_CONFIG_FILE Signed-off-by: Even Wiik Thomassen <even.thomassen@cognite.com> * Added tests and locked black on version 19.3b0. I added tests for the functions in __main___. Also added pytest-mock as depedencies. I had to pin black to version 19.3b0 because the newer version comes with regex depedency that require a C compiler. Known issue with black: psf/black#1112 Signed-off-by: Even Wiik Thomassen <even.thomassen@cognite.com>
I added a PR #1172 that makes It would be great if you could help me to test my changes. |
For people looking for a compact Alpine Docker image with Black 19.10b0, installing Black with GCC can be done in a separate build stage to reduce the final image size. Example (see original Dockerfile): FROM alpine:3.10.3 AS black
RUN apk add --no-cache \
gcc~=8.3.0 \
musl-dev~=1.1.22 \
python3-dev~=3.7.5 \
&& pip3 install --target /opt/black black==19.10b0
FROM alpine:3.10.3
RUN apk add --no-cache python3~=3.7.5
COPY --from=black /opt/black /opt/black
ENV PATH="${PATH}:/opt/black/bin" \
PYTHONPATH="${PYTHONPATH}:/opt/black" This one results in an image of about 64.8MB in size. I hope this helps. |
Why is there a reason for |
@impredicative I have a library that generates Python source code from |
There should be an alternative to a hard dependency of The author of https://bitbucket.org/mrabarnett/mrab-regex/issues/343/wheel-for-linux
Why not revert the fix and dependency to reconsider a better solution? |
I love the suggestion of give up on weird edge cases of Unicode, it is not even 1% of the folks using black if I had to guess that face this odd edge case. That being said, I think that you could also take the library of regex internally into black (Much like urllib3 is embedded in requests) and build black as wheels for all deploy targets. But whatever the decision something needs to happen or folks will start to abandon ship. |
Black version 19.10b0 has a new `regex` dependency which requires building from source on Linux. Until this is fixed pin to earlier version. See: psf/black#1112
Everywhere in code |
FWIW, I've just asked @ambv to revert the change again given the low priority nature of the bug and the impact it has on many people. |
There's a pull request merged that will soon produce linux wheels for regex. |
@zsol , I couldn't find the pull request, but looks like there are wheels for mrab-regex 🎉 |
According to psf/black#1112, black > 19.10b0 needs gcc installed. Installing build-essential and python3-dev solves the issue. TT26059
According to psf/black#1112, black > 19.10b0 needs gcc installed. Installing build-essential and python3-dev solves the issue. TT26059 Co-authored-by: Jairo Llopis <Yajo@users.noreply.github.com>
Installing black version
19.10b0
in an environment withoutgcc
causes an error.To Reproduce:
docker build .
) using this Dockerfile:Expected behavior:
We should be able to install black without needing to have gcc installed separately.
Environment (please complete the following information):
The text was updated successfully, but these errors were encountered: