Feedback report from trying to use this with niche tools

[IcedQuinn]

I spent several hours today looking in to adopting xmake but the documentation to add custom targets appears to be the most minimum it could be. I am coming in from systems like Plan 9 Mk, PyDoIt and Ninja where it is easy to set up arbitrary rules for obscure tools.

Some of these will probably need to be split in to their own separate issues.

• on_build examples looks like markdown but it just copies and renames a file, doesn't even show an example of os.exec and how one is intended to run say pandoc.
• it took a bit to hunt down os.exec is what you call to actually get files built
• pulling up the source code there are references to functions likes autogenfile which are never mentioned in the documentation
• there is almost no docs for on_build_files but then a huge example for lex on buildcmd, which also brings in concepts like tracking the extra dependencies generated by the rule, but the on_build sections never told you how to do this at all
• is on_build deprecated for on_buildcmd? it would appear to be since one is documented and the other is not, and the source code for things like protobuf, lex and capnproto use buildcmd
• how does one deal with jobs where many files are specified as inputs but only one or two of those actually are part of the command line?

two examples of order only dependencies here:

• in nim you compile similarly to pascal where you only specify the name of the application unit but you may still say its dependencies are required; so i say nim c foo but in mk or ninja i may say that this has order only dependencies to all of the other nim (or .pas) files.
• in imatix GSL you have to specify some number of script files and model files to run; i managed to reverse engineer how to use an 'object' target and a custom rule to do this and generate an output, but xmake doesn't associate the inputs and outputs together and always rebuilds it. i also tried not using separate targets but this just merged all the .gsl and .xml files together via add_file and didn't do what it should.

for example i was able to dig out a way to get a target to actually run what it should:

# xmake.lua
rule('gsl')

on_build_files(function (target, sourcebatch, opt)
    local script = target:values('gsl.script')
    for _, sourcefile in ipairs(sourcebatch.sourcefiles) do
        if path.extension(sourcefile) == '.xml' then
            os.exec('gsl -script:%s %s', script, sourcefile)
        end
    end
    os.exec('touch potato.script')
end)

target('potato')
set_kind('object')
set_values('gsl.output', 'potato.script')
set_values('gsl.script', 'potato.gsl')
set_filename('potato.script')
add_files('potato.gsl', 'potato.xml', {rule='gsl'})

but since i used the objects kind i lose access to being able to set a filename which is also aggravating. so i have to use custom values to set that and xmake will always try to rebuild these.

i could fiddle around with writing some documentation but i have no idea how this is supposed to work (short of just copying the protobuf code and changing some strings)

for example in plan 9 mk the job is just

potato.script: potato.gsl potato.xml
    gsl -script:potato.gsl potato.xml > $target

in please.build there's just a simple seven lines to make a new rule for something like this

def word_count(name:str, file:str) -> str:
    return genrule(
        name = name,
        srcs = [file],
        outs = [f"{name}.wc"],
        cmd = "wc $SRC > $OUT",
    )

i'm not yet sure if xmake is just not suitable for jobs other than bog standard cmake like builds, or if it's 👍 but just needs someone to write docs.

[waruqi]

some docs:

https://xmake.io/#/manual/custom_rule?id=ruleon_buildcmd_file
https://xmake.io/#/manual/custom_rule?id=ruleon_buildcmd_files
https://xmake.io/#/manual/project_target?id=targeton_build
https://xmake.io/#/manual/project_target?id=targeton_build_files

on_build examples looks like markdown but it just copies and renames a file, doesn't even show an example of os.exec and how one is intended to run say pandoc.

see https://xmake.io/#/manual/builtin_modules?id=osexecv

is on_build deprecated for on_buildcmd? it would appear to be since one is documented and the other is not, and the source code for things like protobuf, lex and capnproto use buildcmd

Both of these can be used and have not been discarded. But using on_buildcmd_file will be simpler, and it can handle dependency update issues more conveniently. If you use on_build_file, you can also use depend.on_changed to handle them, but it's a little more complicated.

xmake/xmake/rules/utils/symbols/export_all/export_all.lua

Line 41 in 0b09f8f

depend.on_changed(function ()

on_build_file is more flexible, but on_buildcmd_file is simpler. For novices, I recommend using on_buildcmd_file

related issue: #1246

for example i was able to dig out a way to get a target to actually run what it should:

rule('gsl')
    set_extensions(".xml")
    on_buildcmd_file(function (target, batchcmds, sourcefile, opt)
        local script = target:values('gsl.script')
        local outputfile = --? we need get output file path
        batchcmds:execv("gsl", {"-script:" .. script, sourcefile}, {stdout = outputfile})

        batchcmds:add_depfiles(sourcefile)
        batchcmds:set_depmtime(os.mtime(outputfile))
        batchcmds:set_depcache(target:dependfile(outputfile))
    end
end)

or

rule('gsl')
    set_extensions(".xml")
    on_build_file(function (target, sourcefile, opt)
        import("core.project.depend")
        local script = target:values('gsl.script')
        local outputfile = --? we need get output file path
        depend.on_changed(function ()
             os.execv("gsl", {"-script:" .. script, sourcefile}, {stdout = outputfile})
        end, {dependfile = target:dependfile(outputfile), files = sourcefile, lastmtime = os.mtime(outputfile)})
    end
end)

[waruqi]

more examples:

https://github.com/xmake-io/xmake/blob/master/xmake/rules/lex_yacc/lex/xmake.lua
https://github.com/PucklaMotzer09/VulkanKraft/blob/96a9dcfe9bc6d9ef3ba228c03488b7d040e9d3b4/xmake.lua#L6

[xq114]

I do agree that the documentation for rules is too simple. It lacks examples for many useful cases. For me I usually read the built-in rules under the folder https://github.com/xmake-io/xmake/tree/master/xmake/rules (but I can seldom find the exactly same case as what I need) and get help from github discussions. In my port of suitesparse https://github.com/xq114/SuiteSparse-xmake/blob/main/xmake.lua I felt that even with detailed help from @waruqi (much thanks!), writing rules is still not a easy thing to do, at least for now. I wish if there was a more detailed documentation about the target object, the option object, and more examples of rules explained in detail about the usage, design procedure and key points.

[waruqi]

The examples in the document really cannot meet the needs of all users. I suggest you can directly read https://github.com/xmake-io/xmake/tree/master/xmake/rules or https://github.com/xmake-io/xmake/tree/master/tests/projects

Of course, if I have time, I will consider adding more examples to the document.

It takes a lot of time and work to complete the documentation, and my current fragmentation time can only barely satisfy the resolution of issues and the development of some new features.

I will update part of the documentation before publishing new version, and I will also consider adding documentation for the option and target instance interface later.

[waruqi]

About rules, I have used batchcmd to simplify them before. I think it should not be too complicated to implement a rule by referring to the existing rules with batchcmd.

Of course, I haven't used batchcmd to simplify some of the built-in rules of xmake, so it seems to be more complicated, and I will continue to update them later.