Add Luarocks functionality to XMake #1043

RussellHaley · 2020-11-15T06:27:52Z

Is your feature request related to a problem? Please describe.

I would like to see XMake take on the LuaRocks functionality. I'd like to try and take it on myself, though it's low priority at the moment. I'm mostly asking for your input. Perhaps even if you could write item 1 below, that would allow me to look at the rest? I'm not suggesting you take it on unless you're really keen, but I would be willing to financially contribute.

Describe the solution you'd like

I see it happening in four steps:

Allow xmake to read and build rockspec files
Allow xmake to search the luarocks repository and download rocks
Allow xmake to read and manage luarocks config files
Allow xmake to manage (create, manage, update) rocktrees.

Describe alternatives you've considered

N/A

Additional context

I'm still new to xmake, so "bare with me". Luarocks may be a little different than xmake because it can be used to set up a global tree, user tree or project tree of binaries for use with scripts. But xmake has the tools for downloading, building and installing packages, but doesn't necessarily maintain a list of installed items (that's a guess)? A rockspec file is a subset of a xmake file. A rocktree file used to manage installed modules is really an install manifest; like an xmake.lua file is in reverse.
Luarocks is written to be portable between 5.1 and 5.x (I believe the default install is still 5.1), so there may be a possibility of just running luarocks as a Plugin (though I know little about plugins yet). I've gone through the LuaRocks code a few times.
Alternatively, the LuaRocks functionality is quite generic and perhaps could be easily massaged into xmake (as noted above)? I found the LuaRocks functionality of rock trees to be similar to npm in functionality.
Or maybe it's just a matter of a pre-built task in xmake that writes/installs to a known location and updates a rocktree file?
The "All In One" executable I use on Windows are built in linux and are binaries combined with the lua scripts. Luarocks uses a bunch of lua modules FROM the luarocks repository, so it may be a large part of the code is portable to xmake. Who knows?
One of the things that drives feature suggestion is a problem that I have with Luarocks when using llvm-mingw (e.g. clang/lld): Luarocks assumes mingw is gcc/mingw and links directly to the DLL, which doesn't work in llvm. I'm not saying Hisham at LuaRocks won't fix it, but it's been a problem for a while, which lead me to think of this solution. I will be re-posting about the issue at LuaRocks repo soon (not tonight, it's too late).

Thanks Ruki, I look forward to the conversation. It's low priority. Cheers!

waruqi · 2020-11-15T07:14:40Z

Do you mean to use xmake instead of luarocks' built-in build system to build all the binary libraries used by lua packages? Make all lua packages adapt to more toolchains, such as llvm-mingw?

If so, you can use the xrepo command provided by the dev/master version of xmake.

see https://github.com/xmake-io/xrepo/

This command can be used to find and install packages maintained by third-party package management tools, such as conan/vcpkg/homebrew/pacman.

Although xrepo does not support luarocks yet, but I will consider to add support for it. E.g

xrepo install -p mingw -a arm64 luarocks::xxx
xrepo fetch -p mingw -a arm64 luarocks::xxx

RussellHaley · 2020-11-15T08:21:03Z

Yes, that's almost exactly what I was thinking. I had probably seen that in your help files and it subliminally triggered the idea. :-)

Let me know what I can do to help.

Is there some way I can distribute my compiler/lua toolchain with xmake? Currently I'm using Windows msi installers. I have no desire to use windows store at the moment. I can use my MSI on chocolatey.

waruqi · 2020-11-15T08:32:25Z

Is there some way I can distribute my compiler/lua toolchain with xmake? Currently I'm using Windows msi installers. I have no desire to use windows store at the moment. I can use my MSI on chocolatey.

At present, xmake does not support generate installation packages, but you can customize on_package/xmake package to call other packaging tools, such as executing nsis to generate win installation packages, or use makeself to generate runfile self-installation scripts

Or customize on_install, and then execute xmake install in the source code to directly install the compiled binary and toolchain files.

waruqi · 2020-11-15T08:36:32Z

Yes, that's almost exactly what I was thinking. I had probably seen that in your help files and it subliminally triggered the idea. :-)

Let me know what I can do to help.

I can consider in the next version, let xmake package manager and xrepo command support it, but you may need to wait some time, the current version 2.3.9 I do not plan to add new new features, because it will be released soon.

RussellHaley · 2020-11-15T16:59:13Z

Is there some way I can distribute my compiler/lua toolchain with xmake? Currently I'm using Windows msi installers. I have no desire to use windows store at the moment. I can use my MSI on chocolatey.

At present, xmake does not support generate installation packages, but you can customize on_package/xmake package to call other packaging tools, such as executing nsis to generate win installation packages, or use makeself to generate runfile self-installation scripts

Or customize on_install, and then execute xmake install in the source code to directly install the compiled binary and toolchain files.

Thanks for the suggestions. I'm currently cross compiling on FreeBSD (as you know) and my WIX scripts are all entangled with Visual Studio. I will need to find a better solution to installers. Your xmake project is very exciting to me and I had (wild, fitful) dreams of using xmake as a binary package manager on Windows. :-)

waruqi · 2020-11-15T23:53:41Z

Thanks for the suggestions. I'm currently cross compiling on FreeBSD (as you know) and my WIX scripts are all entangled with Visual Studio. I will need to find a better solution to installers

You can try nsis. The windows installer of xmake is made by using nsi scripts. In addition, it provides makensis command line to make windows installation packages and it is open source. It seems that it can also be used on linux.

https://mohanjith.net/blog/2007/08/makensis-on-linux.html

Your xmake project is very exciting to me and I had (wild, fitful) dreams of using xmake as a binary package manager on Windows. :-)

At present, the focus of xmake is to build c/c++ and depend on package management. There are still many things to do, such as distributed compilation.

In the future, when I have free time, I may consider studying binary packaging and distribution. But currently this is not the focus of xmake.

RussellHaley · 2020-11-21T22:34:37Z

While I mentioned a binary package manager, that's more of a tangent. Here is some more thoughts on the Luarocks integration idea.

Luarocks does not handle pre-requisites and this falls over flat on Windows (I thank @siffiejoe for reminding me of this drawback). Ignoring the configuration warning, the following is an example where I am trying to use a pcre module:

C:\Windows\system32> luarocks install lrexlib-pcre
Warning: Lua 5.3 interpreter not found at C:\Program Files (x86)\WinLua\LuaRocks

Modules may not install with the correct configurations. You may want to configure the path prefix to your build of Lua 5.3 using

   luarocks config --local lua_dir <your-lua-prefix>

Installing https://luarocks.org/lrexlib-pcre-2.9.1-1.src.rock

Error: Could not find header file for PCRE
  No file pcre.h in C:/Program Files (x86)/WinLua/WLC/i686-w64-mingw32/include
  No file pcre.h in c:/mingw/include
  No file pcre.h in c:/windows/system32/include
You may have to install PCRE in your system and/or pass PCRE_DIR or PCRE_INCDIR to the luarocks command.
Example: luarocks install lrexlib-pcre PCRE_DIR=/usr/local

The pre-requisite problem originally made me thing of integrating xmake into luarocks (the opposite of this feature request). If one is using Lua and LuaRocks primarily to support a desktop scripting environment, calling xmake from luarocks makes "more sense". A custom call to an xmake script could be added to run a package command and install components in a directory known to luarocks, possibly even as an adendem to the rocktree.

HOWEVER, If one approaches the problem from an application perspective (e.g. an executable for distribution), then the problem is different. The developer would likely want the executable binaries to exist within the application directory structure with the scripts in a known location. Luarocks sort of supports this through project rock trees but again, it feels like the focus is desktop scripting (maybe I'm incorrect?). XMake is superior here because packages can be installed anywhere one wishes. XMake could be configured to work with a luarocks config script and place components where needed.

My final thought is some form of segue until then. I am wondering about creating an xmake script to build a bunch of packages and install them in a library directory.

Thoughts, comments? @siffiejoe?

Regards,

waruqi · 2021-01-14T13:50:51Z

@RussellHaley I have added one patch to support it. see luarocks/luarocks#1265

I tested it on macos/linux and macos/mingw, it works fine for me. but I cannot test it on windows. you can try it.

About using xmake instead of luarocks/builtin build type, this patch can also support it.

build = {
    type = "builtin",
    modules = {
        ["ltui"] = {
            sources = "src/core/lcurses/lcurses.c",
            libraries = {"curses"}
        }
    }

But currently I don't have a good way to completely hook and replace the luarocks/builtin build type, I can only modify it by hard-coding rockspec.build.type = "xmake" to test it in fetch.lua: fetch.load_local_rockspec()

# use xmake instead of `builtin` build type
# How to do it?
# export LUAROCKS_BUILDTYPE=xmake ?

$ luarocks install ltui  # builtin -> xmake
$ luarocks install LuaSocket #builtin -> xmake

waruqi · 2021-01-14T13:55:24Z

I think this issue has nothing to do with xmake v2.5.1, which does not require modification of xmake's code. So I removed it from v2.5.1 Milestone, but I will continue to pay attention to this issue.

RussellHaley · 2021-01-14T21:06:11Z

Hi Ruki,

Absolutely, please push this out. I have no ability to absorb this for a week or so. Thank you for the suggestions you have made. I will review them asap and will try and provide some feedback. Your the best Ruki!

waruqi · 2021-01-17T10:09:22Z

New solution: https://github.com/xmake-io/luarocks-build-xmake

RussellHaley · 2021-01-18T03:42:53Z

I have forked and installed it with a standard LuaRocks installation.

osboxes@osboxes ~/g/luarocks-build-xmake (master)> pwd
/home/osboxes/git/luarocks-build-xmake
osboxes@osboxes ~/g/luarocks-build-xmake (master)> sudo luarocks make rockspecs/luarocks-build-xmake-1.0-20.rockspec 

luarocks-build-xmake 1.0-20 depends on lua >= 5.1 (5.3-1 provided by VM)
luarocks-build-xmake 1.0-20 is now installed in /usr/local (license: Apache-2.0)

osboxes@osboxes ~/g/luarocks-build-xmake (master)>

I'll go do some testing with some simple C only scenarios in Windows.

waruqi · 2021-01-18T08:37:50Z

It should work now for all platforms on ci, but automatic installation of xmake is not yet supported. https://github.com/xmake-io/luarocks-build-xmake/actions

waruqi · 2021-01-20T02:25:13Z

The Lua source files specified in the module were not copied

I works for me on ci.

https://github.com/xmake-io/luarocks-build-xmake/blob/73a3455c309cae28d5e8fc31038bac451ad03316/rockspecs/example2-1.0-1.rockspec#L18-L22

https://github.com/xmake-io/luarocks-build-xmake/blob/73a3455c309cae28d5e8fc31038bac451ad03316/.github/workflows/linux.yml#L34

I tried with sudo to install in the system rocktree, but xmake is not in the root users path so sudo failed:

I have improved to find xmake to fix this problem.

waruqi · 2021-01-22T09:12:23Z

I have fully implemented luarocks-build-xmake, and support automatic download and installation of xmake if it is missing.

Example1 (with xmake.lua)

We can build c/c++ modules if the project contain xmake.lua

├── src
│   ├── test.c
│   └── test.h
└── xmake.lua

xmake.lua

We need to use add_rules("luarocks.module") to add build rules for luarocks modules.

add_rules("mode.debug", "mode.release")

target("example1.hello")
    add_rules("luarocks.module")
    add_files("src/test.c")

rockspec

package = "example1"
version = "1.0-1"
source = {
    url = "git://github.com/xmake-io/luarocks-build-xmake",
    tag = "example1"
}
dependencies = {
    "lua >= 5.1",
    "luarocks-build-xmake"
}
build = {
    type = "xmake",
    copy_directories = {}
}

Example2 (without xmake.lua)

We can use xmake as builtin build type to build c/c++ modules if the project does not contain xmake.lua

├── src
    ├── test.c
    └── test.h

rockspec

package = "example2"
version = "1.0-1"
source = {
    url = "git://github.com/xmake-io/luarocks-build-xmake",
    tag = "example2"
}
dependencies = {
    "lua >= 5.1",
    "luarocks-build-xmake"
}
build = {
    type = "xmake",
    modules = {
        ["example2.hello"] = {
            sources = "src/test.c"
        }
    },
    copy_directories = {}
}

Set special xmake version

dependencies = {
    "lua >= 5.1",
    "luarocks-build-xmake"
}
build = {
    type = "xmake",
    variables = {
        xmake = {
            version = "2.5.1"
        }
    },
    copy_directories = {}
}

Set xmake compilation configuration

dependencies = {
    "lua >= 5.1",
    "luarocks-build-xmake"
}
build = {
    type = "xmake",
    variables = {
        xmake = {
            plat = "mingw",
            arch = "x86_64",
            mode = "debug",
            cflags = "-DTEST1",
            cc = "gcc",
            ld = "gcc",
            ldflags = "...",
            mingw = "mingw sdk path",
            vs = "2019",
            vs_runtime = "MT",
            vs_toolset = "",
            vs_sdkver = "",
        }
    },
    copy_directories = {}
}

RussellHaley · 2021-01-23T07:38:16Z

Amazing, I look forward to testing. I postulate that the compilation configuration can be set through the luarocks config file in the "variables" block:

C:\Users\russh> cat 'C:\Program Files (x86)\WinLua\LuaRocks\config-5.3.lua'
rocks_trees = {
    {
        name = 'user',
        root = home .. [[\\luarocks]]
    },
    {
        name = 'system',
        root = 'C:\\Program Files (x86)\\WinLua\\Lua\\5.3'
    }
}
variables = {
        LUA_DIR = "C:\\Program Files (x86)\\WinLua\\Lua\\5.3\\bin",
        LUA_LIBDIR = "C:\\Program Files (x86)\\WinLua\\Lua\\5.3\\bin",
        LUA_INCDIR = "C:\\Program Files (x86)\\WinLua\\Lua\\5.3\\include",
        LUALIB = "lua53.lib",
        CC = "wlc32.exe",
        LD = "wlc32.exe"
}
external_deps_dirs = {
        "C:\\Program Files (x86)\\WinLua\\WLC\\i686-w64-mingw32/"
}
external_deps_patterns = {
        bin = {
                        "?.exe",
                        "?.bat"
                },
        lib = {
                        "?.lib",
                        "lib?.lib",
                        "lib?.a",
                        "lib?.dll.a",
                        "?.dll.a"
                },
        include = {
                        "?.h"
                }
}
external_lib_extension = 'lib'
local_by_default = true
verbose = false-- set to 'true' to enable verbose output

waruqi · 2021-01-23T16:32:40Z

variables = {
LUA_DIR = "C:\Program Files (x86)\WinLua\Lua\5.3\bin",
LUA_LIBDIR = "C:\Program Files (x86)\WinLua\Lua\5.3\bin",
LUA_INCDIR = "C:\Program Files (x86)\WinLua\Lua\5.3\include",
LUALIB = "lua53.lib",

It should work for lualib configs.

https://github.com/xmake-io/luarocks-build-xmake/blob/8b18f1f62ba26944f117ed549ddaeea210fbdd6e/src/xmake.lua#L550-L552

waruqi · 2021-02-18T15:27:50Z

I am going to close it, we can continue to discuss related topics in luarocks-build-xmake/issues.

waruqi added the feature request label Nov 15, 2020

waruqi added this to the v2.5.1 milestone Nov 15, 2020

RussellHaley mentioned this issue Nov 21, 2020

Windows 10: Compiling with llvm-mingw and lld: LuaRocks 3.0.3 works, 3.4.0 does not. luarocks/luarocks#1237

Closed

waruqi mentioned this issue Jan 14, 2021

Add xmake build type luarocks/luarocks#1265

Closed

waruqi removed this from the v2.5.1 milestone Jan 14, 2021

waruqi added the discussion label Jan 14, 2021

waruqi closed this as completed Feb 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Luarocks functionality to XMake #1043

Add Luarocks functionality to XMake #1043

RussellHaley commented Nov 15, 2020 •

edited

Loading

waruqi commented Nov 15, 2020

RussellHaley commented Nov 15, 2020

waruqi commented Nov 15, 2020

waruqi commented Nov 15, 2020

RussellHaley commented Nov 15, 2020 •

edited

Loading

waruqi commented Nov 15, 2020

RussellHaley commented Nov 21, 2020

waruqi commented Jan 14, 2021 •

edited

Loading

waruqi commented Jan 14, 2021

RussellHaley commented Jan 14, 2021

waruqi commented Jan 17, 2021

RussellHaley commented Jan 18, 2021

waruqi commented Jan 18, 2021

waruqi commented Jan 20, 2021

waruqi commented Jan 22, 2021

RussellHaley commented Jan 23, 2021

waruqi commented Jan 23, 2021

waruqi commented Feb 18, 2021

Add Luarocks functionality to XMake #1043

Add Luarocks functionality to XMake #1043

Comments

RussellHaley commented Nov 15, 2020 • edited Loading

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

waruqi commented Nov 15, 2020

RussellHaley commented Nov 15, 2020

waruqi commented Nov 15, 2020

waruqi commented Nov 15, 2020

RussellHaley commented Nov 15, 2020 • edited Loading

waruqi commented Nov 15, 2020

RussellHaley commented Nov 21, 2020

waruqi commented Jan 14, 2021 • edited Loading

waruqi commented Jan 14, 2021

RussellHaley commented Jan 14, 2021

waruqi commented Jan 17, 2021

RussellHaley commented Jan 18, 2021

waruqi commented Jan 18, 2021

waruqi commented Jan 20, 2021

waruqi commented Jan 22, 2021

Example1 (with xmake.lua)

xmake.lua

rockspec

Example2 (without xmake.lua)

rockspec

Set special xmake version

Set xmake compilation configuration

RussellHaley commented Jan 23, 2021

waruqi commented Jan 23, 2021

waruqi commented Feb 18, 2021

RussellHaley commented Nov 15, 2020 •

edited

Loading

RussellHaley commented Nov 15, 2020 •

edited

Loading

waruqi commented Jan 14, 2021 •

edited

Loading