Project: LuaRocks new distribution model

LuaRocks new distribution model

Pain point to fix

• users often have a hard time to get LuaRocks up and running;
  • sometimes they end up with multiple installations that mess with each other
  • sometimes LuaRocks can't find its own modules
    • even experienced LuaRocks developers bump into those error messages every now and then when working on complex scenarios
    • Of course "if you do everything right, everything works", but the goal is to make the process less error-prone
  • goal: no more big stacktraces with cannot find module luarocks.foo.bar
  • managing the dependencies of LuaRocks itself is hard
    • how to make LuaRocks depend on other modules reliably (and not get in an infinite dependency loop)

Proposed approach

A radical one: building all of LuaRocks into one big luarocks/luarocks.exe executable (including main program, all modules and the Lua interpreter).

Eventually, we'll also be able to depend on LuaFileSystem by bundling it, and we'll be able to split the LuaRocks codebase in separete parts (luarocks.fs is a great candidate, and it could also be greatly simplified) and bundle them and maintain them separately from the LuaRocks core.

The single executable is not an "installer" that expands into several Lua files: it remains a single executable.

This is the approach @leafo takes with the MoonScript interpreter, if my memory serves me right.

Proof of Concept

Here is a PoC script that implements it:

https://gist.github.com/hishamhm/6f8723f0fced81b0ff2f88e5217ccc69

Initial impressions:

• @tieske: Packing everything in a single executable seems a big change indeed.
• @hisham_hm: It's a great feeling to be able to move around the luarocks binary to a random directory, run it with LUA_PATH=invalid LUA_CPATH=invalid ./luarocks and see it run properly!

Immediate TODO items in that script are:

• hardcoding the "site config" values, which means that the dreaded site_config.lua file ceases to exist
• Windows support (there's one "gcc" line in the end that needs to be changed)
• static linking — note that it links liblua dynamically
  • @hisham_hm: I also tested it linking liblua.a statically (and LuaJIT) and it works, but it needs of course paths taken care by the bootstrap process.
  • ideally, also link stuff like luafilesystem and friends statically to i so that luarocks.exe on Windows does not depend on a bunch of Unix-utils

Open Questions

• how to do the bootstrapping

  • @hisham_hm: to be honest, I was even thinking of the possibility of bootsrapping this using Autotools (it's ugly but I know how to do it, it adds no dependencies for end-users or distro maintainers, and it's solid — been using that on htop since forever)
  • bootstraping on Linux and on Windows are probably two entirely separate stories

• how to interact with tools like hererocks

• maintainability of install-time options

  • options to fiddle with the install, but then again, if this fixes the issues, the need for that is gone

• whether a LR install is specific to a Lua install or if it is serving for multiple Lua versions?

  • @hisham_hm: I've seen people attempting to run multiple Lua versions from a single LR install, but that is not currently supported out of the box.
    • I'd argue it would be easier to support multiple Lua versions with a single executable, due to less interaction with env variables

• how to add additional components to LR later? (e.g. new commands, etc.)

  • this could be done via a regular rock, because the .lua package loader still works. Once the LuaRocks core is up and running, things get reliable, the problem is finding the core.

• what about luarocks.loader? other programs need to find it

  • There are at least two possible solutions:
    1. same as above, and make luarocks.loader an "extra"
    2. a radical option: making the Lua interpreter bundled in the luarocks executable usable for running Lua programs (as in luarocks run foo.lua or even #!/usr/bin/env luarocks run; then luarocks.loader is in package.loaded)
       • this approach does not solve using luarocks.loader from third-party interpreters such as resty, so solution 1 would still be needed.

• if LuaRocks runs on its bundled interpreter, then how does it know for what Lua version it has to install a rock?

  • for Unix, currently a LR installation is fixed for a Lua version — if we make it use the same version it was bundled with it, it wouldn't be any different (that means three different possible binaries)
  • for Windows, current Windows package ships with Lua5.1 but can be configured to use any Lua; but even shipping three binaries in the Windows zip would be simpler than the current setup
  • We can only use the stock lua binaries shipped with LR for running LR itself
    • Once the rocks_trees are installed and the Lua paths for any third-party Lua interpreter is configured, that third-party Lua interpreter can still be used.