python tools not functional from install (prefix) location when building from source

[white-haired-uncle]

Game and System Information

• Version:

Battle for Wesnoth v1.19.0-dev (b3802f44ea1-Clean) <unknown>
Started on Sat Apr 27 00:25:16 2024

Automatically found a possible data directory at: /home/USER/wesnoth
Battle for Wesnoth 1.19.0-dev

Library versions:
Boost:             1.83
Lua:               5.4.6
OpenSSL/libcrypto: 3.0.0m-dev (runtime 3.0.0m-dev)
libcurl:           8.5.0 (runtime 8.5.0)
Cairo:             1.18.0 (runtime 1.18.0)
Pango:             1.52.1 (runtime 1.52.1)
SDL:               2.30.0 (runtime 2.30.0)
SDL_image:         2.8.2 (runtime 2.8.2)
SDL_mixer:         2.8.0 (runtime 2.8.0)


Optional features:
Lua console completion:       yes
D-Bus notifications back end: yes

• OS:
  Linux u2404 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Description of the bug

The python tools (wmlindent, etc) will not run from the install directory, at least not without customization that doesn't seem to be documented. They will run from $prefix/share/wesnoth/data/tools/. They also run fine if you remove the binary from $prefix/bin and replace it with a link to the working version.

u2404$ ls -li /usr/local/bin/wmlindent 
1240228 -rwxrwxr-x 1 root root 16594 Apr 26 17:38 /usr/local/bin/wmlindent

u2404$ ls -li /usr/local/share/wesnoth/data/tools/wmlindent 
1494357 -rwxrwxr-x 1 root root 16594 Apr 26 17:38 /usr/local/share/wesnoth/data/tools/wmlindent

u2404$ /usr/local/bin/wmlindent
Traceback (most recent call last):
  File "/usr/local/bin/wmlindent", line 65, in <module>
    from wesnoth import wmltools3 as wmltools
ModuleNotFoundError: No module named 'wesnoth'

u2404$ /usr/local/share/wesnoth/data/tools/wmlindent
/usr/local/share/wesnoth/data/tools/wesnoth/wmltools3.py:71: SyntaxWarning: invalid escape sequence '\*'
  match_mult = re.match("(.+)\*(\d+)", token) # repeat syntax, eg. [melee*3]

Steps to reproduce the behavior

I've seen this for some time, there's nothing special about the versions listed above. I did, however, install a clean version of Ubuntu 24.04 (desktop, graphical install, all options left at default) and confirm the issue.

After installing ksh, openssh-server, git, I installed the following before building wesnoth:

g++ libboost-all-dev libsdl2-dev libsdl2-image-dev libsdl2-mixer-dev libsdl2-ttf-dev libcairo2-dev libpango1.0-dev libcrypto++-dev libssl-dev libvorbis-dev libreadline-dev python3-tk scons libcurl4-openssl-dev

scons nls=no wesnoth
sudo scons install wesnoth 
sudo scons install-pytools

Expected behavior

I'm sure there's some path that python depends on in case it doesn't find the wesnoth module "under" the directory the binary is running from. However, I don't see this documented anywhere, and I think it's reasonable to assume that if I install in $prefix, I want the versions (or links) in $prefix/bin to work without any customization. I would think scons could be configured to install symlinks from the $prefix/bin versions to the working versions.

Additional context

Could be related to #1988. Seemed like it was going to be at first at least.

I haven't tested cmake, but I will.

[soliton-]

@loonycyborg ^

[ProditorMagnus]

SyntaxWarning: invalid escape sequence '\*' indicates issues in file.

[white-haired-uncle]

There is no file in this case, but at least wmlindent is running as compared to the first case when it aborts due to missing module.

[Elvish-Hunter]

Since the Python tools appear to run correctly in previous versions, it could be related to the new Python version (I haven't updated yet so I can't tell for sure).
The SyntaxWarning: invalid escape sequence '\*' line certainly is specific to Python 3.12, and this is something that we'll have to fix because it's planned that a future Python version will convert that warning into a much harsher SyntaxError.

[white-haired-uncle]

Since the Python tools appear to run correctly in previous versions, it could be related to the new Python version (I haven't updated yet so I can't tell for sure).

This isn't new. I'm not positive it goes back to 1.16, but I'm over 50/50. Or if you mean python version I didn't pay attention, but the issue goes back a couple years at least.

The SyntaxWarning: invalid escape sequence '\*' line certainly is specific to Python 3.12, and this is something that we'll have to fix because it's planned that a future Python version will convert that warning into a much harsher SyntaxError.

Oh, that's a different issue I guess. I thought that was just because I ran it without any input, but I just ran it on /usr/local/share/wesnoth/data/campaigns/Secrets_of_the_Ancients/scenarios/11_Battleground.cfg.out and it does error. I was only trying to show that /usr/local/bin/wml* and /usr/local/share/wesnoth/data/tools/wml* worked differently.

Perhaps I just stumbled on this since I was working on u24.04 (python=3.12.3)? I haven't seen an issue with python=3.10.12 on an earlier 1.19 build.

[white-haired-uncle]

I tried to test this with cmake, but I can't find any way to build the WML tools other than scons. Perhaps that's documented along with the WML tools build instructions for scons.

On a different note, I did try setting, for example export PYTHONPATH=/usr/local/lib/python/site-packages and I still got an error, but a different one. I'm wondering if no one notices /usr/local/bin/wmlindent not working (for them) because they have their environment set correctly, whereas I am working from a clean slate. If so, I think it's reasonable for the user to expect at least some directions on what to set up (though just doing something like installing links in $prefix/bin to $prefix/share/wesnoth/data/tools/ seems like a better approach since that just works without the user having to do anything -- unless perhaps there's a reason to respect their environment?).

[CelticMinstrel]

I've added the Linux label since this is clearly dependent on Linux's practice of distributing an installed program and its associated files across various places.

I'd say the easiest fix is to just install a shell script that changes to the appropriate directory and runs the tool from there. There might be a better fix though.

But I am wondering if this is specifically a Wesnoth issue at all. Unless you installed the Flatpak version, this is probably something to raise with the Ubuntu package managers rather than with Wesnoth.

EDIT: Ah, it could still be a bug in scons install-pytools, which would indeed be on us.

[white-haired-uncle]

This was compiled from git, no packages involved. I can't say for sure, but I seem to remember that the tools don't come with package versions.

[CelticMinstrel]

Still specific to Linux in any case.

[loonycyborg]

Running tools installed fails because it can't find the "wesnoth" module. The build script seems to be installing that module to site-packages, so this could be probably fixed by setting it to correct value for your environment. There's python_site_packages_dir setting in scons script for that. I'm not sure how python is looking for site packages now. I bet it changed a lot. You can still launch them from data/tools directly and it will find "wesnoth" module then because it's in same dir as the tool in question. Probably with python it's expected to use setuptools/poetry/etc to do proper system installs.