rfe: extend run_target() to match run_command()

[stsp]

Currently it seems impossible to run
some command at build time, and
capture its output.
It would be good to add the capture
arg to run_target() (similar to the one
of run_command()), and add an stdout()
method to the class returned by
run_target().

[eli-schwartz]

Adding a .stdout() method to a run_target doesn't make much sense since a run_target would be executed by ninja and not by meson.

run_target doesn't produce output artifacts, it is used for creating "PHONY" targets such as ninja flash to copy a built firmware image to some connected hardware. If you want to capture its output, surely it should just be a custom_target, which does have a capture arg?

[stsp]

surely it should just be a custom_target, which does have a capture arg?

But its return type doesn't seem to
have stdout() method, so something
like this isn't currently possible:

t = custom_target(..., capture: true)
t2 = custom_target(..., depends: t, command: [some_cmd, '-e', t.stdout()])

[stsp]

I.e. custom_target() does have capture:,
but it still wants to create the file. It doesn't
give you the output as a variable that you
can use later.

[eli-schwartz]

surely it should just be a custom_target, which does have a capture arg?

But its return type doesn't seem to have stdout() method, so something like this isn't currently possible:

t = custom_target(..., capture: true)
t2 = custom_target(..., depends: t, command: [some_cmd, '-e', t.stdout()])

That is because you can capture the stdout in one of two ways:

• at build time (when ninja is run) by storing it in a file
• at setup time (when meson is run) by returning it in a meson variable

Both run_target and custom_target are compile rules, they are not run at all when meson is run and cannot return stdout in the meson.build file. run_command is different -- it is only run by meson, and it returns stdout to the meson.build file and cannot be re-evaluated by ninja.

[stsp]

Yes, I understand what you say.
What I don't understand, however,
is how the problem should be solved.
You seem to imply whatever runs by
ninja, cannot carry stdout between
targets. Something I easily do with
make.
So what should be a solution?
Extend ninja?

[stsp]

I mean, why ninja can't be instructed
to put stdout into a variable?
Why it can't be instructed to substitute
that variable somewhere later?

[eli-schwartz]

I am not sure what you mean by that. Make can't do what you're saying either.

Make does mix setup and compile in the same action though:

T = $(shell run_command ...)

custom_target1:
    some_cmd -e $(T)

Then, Make converts the command block, replacing variables computed during the setup pass, into a shell script, and executes it with the replacement T.

But make can't do that with two compile rules since each one is run inside its own new shell. This doesn't work:

custom_target1:
    T=$$(some_cmd1)

custom_target2: custom_target1
    some_cmd2 -e $$(T)

You can still do what make allows, with meson too. But the setup stage isn't run every time, so the value of T will be the same every time you run ninja.

If you want it to change without having to rerun meson each time, you could have some_cmd allow reading its data from a file instead.

[stsp]

You can still do what make allows, with meson too. But the setup stage isn't run every time, so the value of T
will be the same every time you run ninja.

No, the problem is, I really need to
run shell only after some other
custom_target() is already built.
So what I want, is:

t1 = custom_target()
t = run_shell(depends: t1, capture: true, check: true, command: some_cmd)
t2 = custom_target(..., depends: t1, command: [some_cmd2, '-e', t.stdout()])

This is possible with make simply because
the variable assigned with = is expanded
when target is being built, which is what I need.
If I use := for an assignment, then it is
expanded when parsed, which would be
an equivalent of run_command() in meson,
but its not what I need.
Of course when I write
t = run_shell(depends: t1, capture: true, check: true, command: some_cmd)
I realize that in make I can't precisely tell
that the assignment should be done after
t1 is being built. I have to rely on the fact
that t2 is built after t1 and the variable is
expanded when t2 is being built. Its fine
with me, although with meson I was hoping
to be able to do exactly this:

custom_target1:
    T=$$(some_cmd1)

custom_target2: custom_target1
    some_cmd2 -e $$(T)

which would be an advantage over make.

[eli-schwartz]

The assignment to T=$$(...) happens in a sh shell, not in make. It disappears as soon as the sh shell finishes building custom_target1.

You can have t2 = custom_target(.........) use a command that looks like this:

command: ['sh, '-c', 'T=$(cat custom_target1.out); some_cmd -e "$T";']

meson will produce a build rule that runs the entire sh command for you, including reading the "captured to a file" output of the first custom target so it can be passed as an argument to the current build rule.

This isn't performed at setup time.

[stsp]

Yes, something like this can work,
except that command for T is quite long,
and has sh -c and substitutions on its
own.
Googling a bit about ninja, it seems they
suggest to just write the wrapper scripts
for such cases.
Wrapper script can work, in which case I'll
end up with something like this:
command: ['sh', '-c', 'wrapper.sh', t1.full_path(), '@INPUT@', '@OUTPUT@'])
I'll do that, but I still wonder why meson
can't do that for me, allowing a simpler
syntax and no silly scripts.

[stsp]

So if there is nothing to improve on
meson side, please close.
I ended up with a clumsy script that
takes 7 parameters, and that works.

[stsp]

Why it takes 7 parameters, is because
I pass many find_program().full_path()
to it, which obviously could be much
better handled within meson.

[eli-schwartz]

Wrapper script can work, in which case I'll end up with something like this: command: ['sh', '-c', 'wrapper.sh', t1.full_path(), '@INPUT@', '@OUTPUT@'])

You can probably just use:

command: [find_program('wrapper.sh'), t1.full_path(), '@INPUT@', '@OUTPUT@']

I'll do that, but I still wonder why meson can't do that for me, allowing a simpler syntax and no silly scripts.

Someone would have to define what that syntax would look like for meson, and how it would perform that wrapping.

I worry that it is the kind of feature request easier to ask for than to implement...

[eli-schwartz]

Why it takes 7 parameters, is because I pass many find_program().full_path() to it, which obviously could be much better handled within meson.

You can just pass the find_program itself, no need for full_path(). Using full_path() is only really useful when the relative path that meson writes to the ninja file is "wrong" for the shell script because the shell script internally does a cd ... and the path breaks... but you could also use realpath in the shell script to get the absolute location of the arguments.

[stsp]

Ok, first let me point out that what
I want, is possible with make.
Proof:

all: bar

foo:
        $(eval P != echo $$@ | sed 's/oo/tst/')

bar: foo
        @echo $(P)

This example replaces oo in foo
with tst, so the output will be ftst.
Please note that I print the output in
a different recipe, which means I
"moved" the stdout of foo to bar.

As for not passing full_path() - thanks,
this indeed works when I pass the
individual arguments. If however I
concatenate strings for /bin/sh -c, I
still need to use full_path().

Someone would have to define what that syntax would look like for meson, and how it would perform that wrapping.

I think this can look like this:

t1 = custom_target()
c1 = run_shell(depends: t1, command: [find_program(p1), t1])
c2 = run_shell(depends: c1, command: [find_program(p2), c1.stduot()])
...
t2 = custom_target(depends: c5, command: [some_cmd_that_uses_c1...5.stdout()])

As was pointed out, make supports
this natively. I only showed an example
of variable passing, but the whole concept
can use the intermediate targets this way:

all: t2
.INTERMEDIATE: c1

t1:
        touch $@

c1: t1
        $(eval P != echo $$@tst)

t2: c1
        @echo $(P)
        touch $@

c1 here is exactly the intermediate target
that only runs a shell command. It needs to
be marked as .INTERMEDIATE:, in which
case make won't re-build anything based
on a missing file with that name.
So this fully matches the initial task.

If ninja doesn't support that natively, then
meson can use temporary files. It can translate
every run_shell() into a target that creates
a temporary file, and the stdout() method
will just print its content. Ordering is also
possible to keep with temporary files.