Updating to Lua 5.4.2 crashes AwesomeWM

[Nuc1eoN]

Today I've updated from Lua v5.3.5 to v5.4.2 which is in the repos.

Unfortunately it seems, that update crashes the Xserver, or more specifically AwesomeWM.

This is the X crash message:

awesome: symbol lookup error: awesome: undefined symbol: lua_newuserdata

[sage-etcher]

TLDR

After upgrading Lua you have re-build Awesome and re-install Lua modules (LuaRocks stuff). Currently the stable version of LuaRocks:lgi does not have 5.4 compatibility. However, the experimental git version (non-released) does have 5.4 support, so if you manually install LGI from git, Awesome should work. However, currently I see no way of installing the needed LGI version with Alien.

awesome: symbol lookup error: awesome: undefined symbol: lua_newuserdata

The error message is due to lua_newuserdata behing defined as a preprocessor macro in version 5.4, as mentioned in this StackOverflow thread.

Full Process (so far)

Section 1: Installing Lua54 Compatible Compile Recipes

Install/update programs (and deps) for Lua54 support using Compile:

• Lua >= 5.4.0, < 5.5.0
• LuaRocks >= 3.5.0
• CMake >= 3.18.0

Re-Source Bash's Environment

It is important that LUA_PATH_5_4 and LUA_CPATH_5_4 are set correctly later, otherwise startx will complain. Bash will not source the new versions of Lua or LuaRocks's Environment files without a log-out or reboot. Either of them should work, or we can source them manually as follows. The first command re-sources all the Environment files, while the second commands re-source only Lua and LuaRocks

for f in $goboEnvironment/*--*; source $f
or
source /Programs/Lua/Current/Resources/Environment
source /Programs/LuaRocks/Current/Resources/Environment

Section 2: Installing LuaRocks Modules

Gobo provides a handful of LuaRocks modules to make AwesomeWM look and feel as it does. By default these modules are installed for Lua53, but now that Lua54 is being used, they have to be re-installed.

These modules should be easily installed using the sudo Alien --install LuaRocks:PACKAGE script:

• gobo-awesome
• gobo-awesome-alttab
• gobo-awesome-battery
• gobo-awesome-bluetooth
• gobo-awesome-gobonet
• gobo-awesome-light
• gobo-awesome-sound
• gobo-awesome-screenlock
• luafilesystem

If you're lazy like me, you could use this instead, to install all of them

for pkg in luafilesystem gobo-awesome{,-{alttab,battery,bluetooth,gobonet,light,sound}}; do
   sudo Alien-LuaRocks --install $pkg
done

Section 3: Installing LuaRocks Git Modules

Awesome requires LGI to be installed, however LuaRocks:lgi 0.9.2-1 does not have Lua54 support yet. If we want to use Lua 5.4 we will have to install the GIT version instead.

git clone https://github.com/lgi-devs/lgi.git
cd lgi
sudo luarocks make

Section 4: Compiling Awesome

After all of this you should be able to compile Awesome as normal.

sudo Compile Awesome 4.3
startx

Notes

The versions listed in Section 1 are based on when each software added Lua54 support, I have not tested them individually; durring testing I have used the following software/versions:

• Lua 5.4.3
• LuaRocks 3.9.2
• CMake 3.22.2
• Awesome 4.3

[Nuc1eoN]

Sounds like quite the rabbit hole, good work so far 💪

Inside CMake, Lua cannot find the Alien modules; Awesome cannot find LGI, so the build fails. However outside of CMake Lua has no issues finding them with require()???

Maybe try running Compile --debug, IIRC it will list which directories get pulled into the build environment. Also do not use --pure. Sounds like for some reason it doesn't include Aliens? Maybe it has to be added to the Dependencies file? Eg.
LuaRocks:LGI (out of my head, actual phrase may vary).

Is LGI properly installed? Is it installed inside /Programs (and properly symlinked) or inside the Aliens directory?

Right after Compiling, Lua's package.path variable doesn't include the correct Alien paths, a reboot seems to fix this.

This is probably worth investigating at some point.

[sage-etcher]

This was done earlier, I'm not at the system atm, and am done for the night. It's putting up quite a hasle though lmao

Is LGI properly installed? Is it installed inside /Programs (and properly symlinked) or inside the Aliens directory?

luarocks make installs the package into the Aliens subtree. It installs same as Alien --install LuaRocks:PACKAGE would, just from source instead.

earlier testing was done with installing LGI in /Programs. this works almost flawlessly, asside from that Compile complains that LuaRocks:lgi doesn't exist so you'd gotta either skip the dependency or add it to the exclude list. My concern with it though, once the LuaRocks version is updated, we'll be back at having the current issues.

Sounds like for some reason it doesn't include Aliens? Maybe it has to be added to the Dependencies file?

LuaRocks:lgi is already added in the dependency file

I wasn't running with --pure

I tested with Compile --debug and none of the Alien tree got pulled in by Runner.

The build fails on lgi-check (a build tool from Awesome). I was playing around with it some, and lgi-check, passes when ran normally, I ran it under Runner using Awesome's Dependency file and it passed there too, suprisingly?? but then it fails under Compile. This was happening no matter the version of Lua i was running.

though Lua is quite nice in showing what directories it looks in, and it appeared that in Compile Lua is only checking the /System/Index tree. So I figured it was going based on the default CPATH and LPATHs specified in Lua's luaconf.h file, so i wrote a patch for that, tested it and still only checking in /usr/include directories.

I have no clue where it's pulling it's package.path/package.cpath from. The lua shared libraries and installed headers, all included the alien subtree as defaults, but then still it decided to be a pain.

I investicated the lgi-check.c file aswell, and it was REAL BASIC, nothing that should be causing an issue.

Tried defining LUA_PATH and LUA_CPATH variables in the Recipe environment and still nothing changed

I tested it with the --no-sandbox flag in the Recipe aswell, but the issue still persisted. I'm at a bit a loss atm

I plan on investiging under the hood what Compile is doing and playing it by ear from there, but that's for tommorow.

This is probably worth investigating at some point.

and yes investigating that is a plan for the future, I'd like to get things working first, but yes.

one note, if the luaconf.h file is patched to include the Alien subtree as a default, then the restart isn't required. However, it's not ideal, and after a restart, the Alien directories are listed twice in package.path. not good, but thought it was interesting.

[Nuc1eoN]

I tested with Compile --debug and none of the Alien tree got pulled in by Runner.

I'd probably look into this part. Compile should pull in the Aliens tree, if it doesn't then that is an issue.

earlier testing was done with installing LGI in /Programs. this works almost flawlessly, asside from that Compile complains that LuaRocks:lgi doesn't exist so you'd gotta either skip the dependency or add it to the exclude list.

Seems to make sense. If LGI is installed under /Programs rather than Aliens, then the correct Dependency declaration would be LGI, not LuaRocks:lgi.

[sage-etcher]

Part that's confusing me the most is that when Lua panics and cant find a package, it prints out every directory it looked in. Whether the directory exists or not, doesn't matter, it gets printed all the same, no file: PATH/EXPECTED_FILE. And in Compile it doesn't look like Lua is even trying to check the Aliens/LuaRocks tree. So even if eveything else is in place through Runner, if Lua isn't checking the right locations, well, it isn't gonna find jack.

I am going to look more into Runner, but it seems to me that Lua is a bigger issue here.

if i recall correctly

$ sudo Runner -d LUA543_DEPENDENCIES_FILE /Data/Compile/Sources/awesome-4.3/_build/lgi-check

succeeds, and has propper paths. So I dont think the issue is coming directly from Runner. I've only been able to replicate the issue inside Compile.

[Nuc1eoN]

Yeah I get you. Sometimes GoboLinux feels like the first steps on the Moon. But slow and steady approach will lead to the result. Anyways have a good rest.

[sage-etcher]

Okay, I think I found the cause, it was Runner related.

Case	Dependency	Result
A	only LuaRocks	PASSES
B	only Lua	FAILS
C	neither	PASSES
D	both	PASSES

Case A passes because LuaRocks lists Lua as a dependency for itself.
Case C is fairly confusing, but isn't an issue in this scenario.

When running Compile, Runner pulls in dependencies from both Compile and Awesome. Awesome list neither and Compile lists only Lua, placing us firmly in Case B.

Compile already lists Lua # used by Lua rocks, so it would seem adding LuaRocks to Compile's dep-list would make the most sense.

[sage-etcher]

Right after Compiling, Lua's package.path variable doesn't include the correct Alien paths, a reboot seems to fix this.

restarting the machine is no longer necessary, though the Environment files do need to be re-sourced by the user. Unfortunately it doesn't appear that this can be done automatically

The main comment was updated with instructions and reasoning for such (Section 1 > Re-Source Bash's Environment).

[Nuc1eoN]

Case C is fairly confusing, but isn't an issue in this scenario.

Case C is definitely fairly confusing. However it does sound familiar, although I cannot remember what the exact case was for me.

---

@hishamhm as @sage-etcher mentioned, gobo-awesome-screenlock seems to be missing from LuaRocks. Although we can install it from git, this is probably not ideal. Do you know why it has been removed? Would you kindly re-publish it to LuaRocks?

[hishamhm]

@Nuc1eoN I don't think gobo-awesome-screenlock was ever uploaded to luarocks.org! I've just cloned the repo and uploaded it there using my luarocks account.

[hishamhm]

(I've never used that package myself; I wrote the other gobo-awesome entries, but this one was made by Lucas.)

[Nuc1eoN]

I don't think gobo-awesome-screenlock was ever uploaded to luarocks.org! I've just cloned the repo and uploaded it there using my luarocks account.

Great, thank you!:)

@sage-etcher , one note regarding:

It is important that LUA_PATH_5_4 and LUA_CPATH_5_4 are set correctly later, otherwise startx will complain. Bash will not source the new versions of Lua or LuaRocks's Environment files without a log-out or reboot. Either of them should work, or we can source them manually as follows. The first command re-sources all the Environment files, while the second commands re-source only Lua and LuaRocks

for f in $goboEnvironment/*--*; source $f
or
source /Programs/Lua/Current/Resources/Environment
source /Programs/LuaRocks/Current/Resources/Environment

Maybe let's add this re-sourcing to a PostInstall section of the Lua/LuaRocks recipes? That would prevent potential future users from potential issues.

At the very least I would suggest something like post_install_message="\n\nIMPORTANT:\nYou must run 'source /Programs/Lua/Current/Resources/Environment; source /Programs/LuaRocks/Current/Resources/Environment', or reboot/re-login the system.\n"

@hishamhm Do we have some in-build/native hook in Compile that we could use to re-source the ENVIRONMENT files?

EDIT:
I have found RebuildLinks which probably does just that. So we could use that unless Hisham knows an even nicer way:)

[hishamhm]

since each process keeps its own environment, there's no real way to automatically re-source the environment in a complete way after a package upgrade. You'll always have things like terminal windows still open running the old environment, etc. For things like the window manager, which is the parent process of all graphical applications, then yeah, I think it's reasonable to require a re-login to get the whole subprocess tree refreshed.

If re-sourcing the environment is something that is needed in between building dependencies so that the whole compilation succeeds, then code could be added to Compile for it to do that for its own process (I don't recall us reloading Environment files from within Compile... maybe it does already?), but I don't think there's a way to have a more automagical solution that refreshes everything.