itstool 2.0.4 crashes when freeing xmlDocs

[shaunix]

In itstool 2.0.4, we started calling freeDoc on xmlDoc object when we're done with them. libxml2's python bindings require manual frees. Previously, we were leaking them, which could cause runaway memory consumption. This was expecially true with --keep-entities or --load-dtd, which increases the size of the xmlDoc objects in memory.

Unfortunately, we're now freeing objects that are still in use, causing crashes. What I've figured out so far is that this only happens when we run scan_node in get_translated. This method is responsible for finding placeholder elements and recursively merging translations. A likely culprit is that, somewhere in prepping the new node to insert, we're not doing a deep enough copy, resulting in some dangling pointers to the xmlDoc we free.

[pullmoll]

I don't know python too well. Is there perhaps a way to set self._doc to something like NULL or invalid after self._doc.freeDoc() is called? Of course only, if indeed self._doc is to be considered invalid after that call.

[shaunix]

I've just committed some changes that I think fix the memory issues. I'd appreciate if people could test before I roll another release.

[pullmoll]

Looks good to me! I tested the latest commit 9b84c00 and with the itstool resulting from it could successfully build gtk-doc in the Void Linux environment.

[maxice8]

can confirm it worked on rhythmbox, gnome-user-docs ( except when cross-compiling to musl based systems) and evolution

[DimStar77]

building of gnome-getting-started-docs still makes itstools segfault here
https://build.opensuse.org/package/live_build_log/GNOME:Next/gnome-getting-started-docs/openSUSE_Factory/x86_64

also gnome-user-docs:
https://build.opensuse.org/package/live_build_log/GNOME:Next/gnome-user-docs/openSUSE_Factory/x86_64

[shaunix]

I can also reproduce on gnome-getting-started-docs. Thanks for testing.

[shaunix]

This crasher is unrelated, and seems to be a bug in libxml2:

https://bugzilla.gnome.org/show_bug.cgi?id=789714

[besser82]

This crasher is unrelated, and seems to be a bug in libxml2:

https://bugzilla.gnome.org/show_bug.cgi?id=789714

Confirming.

The fixing patch for libxml2 can be found here: https://bugzilla.opensuse.org/show_bug.cgi?id=1065270

[chkno]

Fixing libxml2's python wrapper's utf-8 problems doesn't fix itstool's use-after-free problem. At most, it just stops exercising itstool's error path, leaving itstool use-after-free problems that are triggered only on the error path dormant.

The Python2 end-of-life is pressuring distros to move to more up-to-date itstool versions, which require them to apply distro-patches to resolve this issue and/or #22 and/or https://gitlab.gnome.org/GNOME/libxml2/issues/64 and/or https://bugzilla.gnome.org/show_bug.cgi?id=789714 . Nix's patch: NixOS/nixpkgs#72342