Skip to content

Project: LuaRocks new distribution model

Hisham Muhammad edited this page May 14, 2018 · 3 revisions

LuaRocks new distribution model

Pain point to fix

  • users often have a hard time to get LuaRocks up and running;
    • sometimes they end up with multiple installations that mess with each other
    • sometimes LuaRocks can't find its own modules
      • even experienced LuaRocks developers bump into those error messages every now and then when working on complex scenarios
      • Of course "if you do everything right, everything works", but the goal is to make the process less error-prone
    • goal: no more big stacktraces with cannot find module luarocks.foo.bar
    • managing the dependencies of LuaRocks itself is hard
      • how to make LuaRocks depend on other modules reliably (and not get in an infinite dependency loop)

Proposed approach

  • building all of LuaRocks into one big luarocks/luarocks.exe executable (including main program, all modules and the Lua interpreter).
    • Eventually, we'll also be able to depend on LuaFileSystem by bundling it, and we'll be able to split the LuaRocks codebase in separete parts (luarocks.fs is a great candidate, and it could also be greatly simplified) and bundle them and maintain them separately from the LuaRocks core.
    • The single executable is not an "installer" that expands into several Lua files: it remains a single executable.
    • This is the approach @leafo takes with the MoonScript interpreter, if my memory serves me right.
  • support adding add extra commands to LuaRocks
    • At startup, LuaRocks will scan each directory of the package.path for files named luarocks/cmd/*.lua
    • for better discoverability, some known-but-not-yet-installed extra commands could be listed by luarocks help by default and be automatically installed on first use

Status

Here is a PoC script that implements it:

https://gist.github.com/hishamhm/6f8723f0fced81b0ff2f88e5217ccc69

Initial impressions:

  • @tieske: Packing everything in a single executable seems a big change indeed.
  • @hisham_hm: It's a great feeling to be able to move around the luarocks binary to a random directory, run it with LUA_PATH=invalid LUA_CPATH=invalid ./luarocks and see it run properly!

Immediate TODO items in that script are:

  • hardcoding the "site config" values, which means that the dreaded site_config.lua file ceases to exist
  • Windows support (there's one "gcc" line in the end that needs to be changed)
  • static linking — note that it links liblua dynamically
    • @hisham_hm: I also tested it linking liblua.a statically (and LuaJIT) and it works, but it needs of course paths taken care by the bootstrap process.
    • ideally, also link stuff like luafilesystem and friends statically to i so that luarocks.exe on Windows does not depend on a bunch of Unix-utils

Open Questions

  • how to do the bootstrapping

    • @hisham_hm: to be honest, I was even thinking of the possibility of bootsrapping this using Autotools (it's ugly but I know how to do it, it adds no dependencies for end-users or distro maintainers, and it's solid — been using that on htop since forever)
    • bootstraping on Linux and on Windows are probably two entirely separate stories
  • how to interact with tools like hererocks

  • maintainability of install-time options

    • options to fiddle with the install, but then again, if this fixes the issues, the need for that is gone
  • whether a LR install is specific to a Lua install or if it is serving for multiple Lua versions?

    • @hisham_hm: I've seen people attempting to run multiple Lua versions from a single LR install, but that is not currently supported out of the box.
      • I'd argue it would be easier to support multiple Lua versions with a single executable, due to less interaction with env variables
  • how to add additional components to LR later? (e.g. new commands, etc.)

    • this could be done via a regular rock, because the .lua package loader still works. Once the LuaRocks core is up and running, things get reliable, the problem is finding the core.
  • what about luarocks.loader? other programs need to find it

    • There are at least two possible solutions:
      1. same as above, and make luarocks.loader an "extra"
      2. a radical option: making the Lua interpreter bundled in the luarocks executable usable for running Lua programs (as in luarocks run foo.lua or even #!/usr/bin/env luarocks run; then luarocks.loader is in package.loaded)
        • this approach does not solve using luarocks.loader from third-party interpreters such as resty, so solution 1 would still be needed.
  • if LuaRocks runs on its bundled interpreter, then how does it know for what Lua version it has to install a rock?

    • for Unix, currently a LR installation is fixed for a Lua version — if we make it use the same version it was bundled with it, it wouldn't be any different (that means three different possible binaries)
    • for Windows, current Windows package ships with Lua5.1 but can be configured to use any Lua; but even shipping three binaries in the Windows zip would be simpler than the current setup
    • We can only use the stock lua binaries shipped with LR for running LR itself
      • Once the rocks_trees are installed and the Lua paths for any third-party Lua interpreter is configured, that third-party Lua interpreter can still be used.
You can’t perform that action at this time.