-
-
Notifications
You must be signed in to change notification settings - Fork 13k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
itstool: 2.0.2 -> 2.0.6 #72342
itstool: 2.0.2 -> 2.0.6 #72342
Conversation
To get python3 support. NixOS#63174 flipped itstool to python3, but itstool doesn't support python3 until 2.0.3 (and perhaps does not support it well until 2.0.5). Pressing forward instead of rolling back at worldofpeace's suggestion, who mentions that other distros seem to be able to ship recent versions of itstool. Tensions in this space seem two-fold. One set centers around libxml2 being a low-level C library with sharp edges, manual memory management, and performance concerns; the python libxml2 wrapper being quite thin (the most dubious character in this drama); and python's sentiment that it ought to be quite hard to crash the interpreter casually. This comes to a head in https://gitlab.gnome.org/GNOME/libxml2/issues/12 , where a use-after-free problem in idiomatic-looking python code is declared working-as-designed. The other set is around python3 being more UTF-8-aware than libxml2's python wrapper, such as https://bugzilla.gnome.org/show_bug.cgi?id=789714 and https://src.fedoraproject.org/rpms/libxml2/blob/master/f/libxml2-2.9.8-python3-unicode-errors.patch itstool is caught in this crossfire merely for being a widely-used python program that uses XML.
If this goes forward, #72335 (revert to python2) should not. I don't know which is all-around best. My interest here is just to be able to run NixOS tests on staging. |
We will need to apply the opensuse patch/hack for this to work. |
Maybe. I was able to reproduce the problem building |
fd9cfc4 mentions that other packages were broken as well, but doesn't say which ones. #41616 , which that fix is attached to, is about the I cannot build the gnome3 NixOS test with this PR. I get this error while trying to build
I can build zenity with the itstool-back-to-python2 PR #72335 |
i have the impression Python 2 and 3 get mixed up in that trace. It's using @chkno Could you push a commit with the patch here as well? |
No.
This change in
(edit: c0cecd0 below proposes this patch without the |
The zenity build failure is not related to itstool. Zenity builds just fine when the itstool 2.0.6 bump is applied immediately after the 2.0.4 -> 2.0.2 revert. I think this command will find the commit that introduced the zenity build failure:
|
@GrahamcOfBorg build gnome3.gnome-devel-docs gnumeric |
ok, the zenity build failure is because it uses Python libraries and especially applications ideally should not propagate their dependencies, but we're not there yet. We should be able to work around this by adding |
Could we do the same thing meson does for now?
|
With gnumeric, I am still getting the segfault on your branch. |
… crash 1. Gnumeric has unbalanced XML tags in its doc translations. 2. itstool's XML error handler tries to print this error with context. 3. libxml2's context snipper treats the data as bytes, not UTF-8. 4. python3Packages.libxml2 casts the context to a UTF-8 Python string. 5. itstool dereferences a null pointer. This patch intervenes at NixOS#4. In https://bugzilla.gnome.org/show_bug.cgi?id=789714#c4 , upstream suggests that intervening at NixOS#3 would be better -- that each of the four copies of xmlParserPrintFileContextInternal() have four additional UTF-8 problems, one of which is that the caret indicator ought to count "unicode characters" not bytes. But to position a caret correctly, a character count is not sufficient -- this would need to use icu's BiDi logic (with fallback to doing something wrong when libxml2 is configured not to use icu) -- which makes a 'correct' fix a much larger project than this simple band-aid.
(The bisect finished: a2c62e2 introduced the zenity build break.)
I captured a
The I also created a gnumeric PR to fix the malformed XML that's causing these errors that itstool / python3Packages.libxml2 fails to handle gracefully. |
That breaks packages that rely on Python like itstool does. NixOS#72342 (comment)
a2c62e2 looks unlikely to cause that. I think it is caused by #63147 as well. Anyway, I confirmed that https://github.com/jtojnar/nixpkgs/tree/zenity-fix fixes that instance. I would say that if you cherry-pick that and add the GNumeric patch we can merge this. We can always deal with the remaining packages later. |
Or since you added the libxml hack, the GNumeric one is not necessary. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or let's just merge this and I will handle the zenity fix separately.
Thanks. |
That breaks packages that rely on Python like itstool does. #72342 (comment)
Motivation for this change
To get python3 support. #63174 flipped itstool to python3, but itstool doesn't support python3 until 2.0.3 (and perhaps does not support it well until 2.0.5).
Pressing forward instead of rolling back at worldofpeace's suggestion in #63174 and #72335, where it is mentioned that other distros seem to be able to ship recent versions of itstool.
Tensions in this space seem two-fold. One set centers around libxml2 being a low-level C library with sharp edges, manual memory management, and performance concerns; the python libxml2 wrapper being quite thin (the most dubious character in this drama); and python's sentiment that it ought to be quite hard to crash the interpreter casually. This comes to a head in https://gitlab.gnome.org/GNOME/libxml2/issues/12 , where a use-after-free problem in idiomatic-looking python code is declared working-as-designed.
The other set is around python3 being more UTF-8-aware than libxml2's python wrapper, such as https://bugzilla.gnome.org/show_bug.cgi?id=789714 and https://src.fedoraproject.org/rpms/libxml2/blob/master/f/libxml2-2.9.8-python3-unicode-errors.patch
itstool is caught in this crossfire merely for being a widely-used python program that uses XML.
Things done
sandbox
innix.conf
on non-NixOS linux)nix-shell -p nix-review --run "nix-review wip"
(This is 4,000 packages, if I'm counting correctly. It would take a long time for me to run this to completion on my personal hardware)./result/bin/
)nix path-info -S
before and after)Notify maintainers
cc @jtojnar @worldofpeace @FRidh