Checking correctness for multi-target rules

[elfring]

Multiple targets can be specified in a single make rule. While they are written only once at such a place, the make software usually expands the specification into several rules with the same recipe so far.
But this does not happen with pattern rules:

…
Pattern rules may have more than one target. Unlike normal rules, this does not act as many different rules with the same prerequisites and recipe. If a pattern rule has multiple targets, make knows that the rule’s recipe is responsible for making all of the targets. The recipe is executed only once to make all the targets.
…

• This technical detail has got important consequences on the handling of dependencies for (parallel) software builds because of multiple output files by special commands.
• How do you think about details from previous development discussions?

Can a variant of the command “remake --targets” help to identify questionable multi-target rules?

[rocky]

I'm having a hard time understanding the proposal, so let me try to paraphrase to see if I understand correctly.

multiple targets are considered bad because they don't work in parallel software builds. So you'd like an option that asks GNU Make like say --lint to look for this questionable practice and perhaps advises of other questionable practices?

Is this correct?

[elfring]

The tool “ninja” provides an other approach.

@liugang: How much do you care for the development challenges which I tried to point for the programming language “make” out?

[rocky]

@elfring I still don't understand what you want. Try to be expansive in answering.

Perhaps you could pretend such a feature exists and show examples of input, command and outputs. And why this is beneficial. Thanks.

[elfring]

multiple targets are considered bad

Not generally.

because they don't work in parallel software builds.

There are special dependencies to consider if you care for correct builds were multiple output files are involved by special commands like parser generators.

So you'd like an option that asks GNU Make like say --lint to look for this questionable practice and perhaps advises of other questionable practices?

Yes. - I imagine a few possibilities in such a direction.

[elfring]

And why this is beneficial.

Do you need any further clarification for the information in my initial description?

Did you improve the make software in such a way that you could convert special rules into pattern rules?

[rocky]

Ok, we've established that you'd like an option like --lint to look for questionable practices, for which there is one specific instance that is clear in your mind.

Understanding only this much, I think it a fine idea to add such an option to look for questionable practices. Why don't you fork the code, work on it and submit a pull request then?

I'll give the standard "remake" advertising: because this isn't as pervasively used as "GNU make" we are more free to try out new ideas (hopefully upward compatible). In contrast to GNU make, "remake" has dropped support for OSes that I doubt most people don't know about, let alone care. This includes VMS, OS/2, riscos, and amiga (however wonderful they were back in the day). And if support is lacking say for EMX, or DOS in "remake" that's okay too. We don't have as stringent release requirements for those OSes. So we can release sooner.

[rocky]

Do you need any further clarification for the information in my initial description?

I just understand that there's something you think is important to add to GNU Make. Right now I'm thinking you are going to do the coding since you understand what you want best.

So right now, I don't have to understand more. In the Pull Request the specifics will be clear. But as best as I can tell, what you want to do seems reasonable.

Did you improve the make software in such a way that you could convert special rules into pattern rules?

In general I did only the minimum to get the additional features "remake" provides.

The first time I forked, I was too aggressive about fixing the old code. Since GNU make also slowly improves it's old code, there were lots of merge conflicts in subsequent GNU make releases (4.1, 4.2.1) in how I improved the old code versus how GNU make did it. So as a result, I now don't try to fix things that aren't broke for my purposes.

[elfring]

… to look for questionable practices, for which there is one specific instance that is clear in your mind.

Correct.

Why don't you fork the code, work on it and submit a pull request then?

• I am not familiar enough with the internal structure for the make software.
• I tend to look more into other source files so far.

I just understand that there's something you think is important to add to GNU Make.

This is right.

You provide a few special extensions already. So I hope that it can be easier to try new ideas out here.

The ordinary GNU make software provides the data export “--print-data-base”. I am looking for the support of other data analysis and transformation possibilities.

[rocky]

I am not familiar enough with the internal structure for the make software.

No one is born knowing the internal structure for the make software. I can say for myself that there was a definitely was time when I wasn't. But if you think the idea important enough, then I imagine it should be motivation enough learn what you need to do to work on the code. I'll try in whatever way I can to address questions you have (which is more help than I got).

Or if you think it likely that:

• your description of the problem is clear, along with
• the concept of what needs to be done, and
• the need so overwhelming,

we can just let this issue hangout as an issue. Someday someone else might come across this and be motivated to add it.

For myself, I've not experienced this problem although I have no doubt it is there along with many other problems.

[elfring]

For myself, I've not experienced this problem although I have no doubt it is there along with many other problems.

How do you think about to compare the consequences for parallel software builds by the following make examples?

A)

msg.h msg.c: msg.spec
	@echo Generating msg.h and msg.c from an interface description.
	@sleep 2
	@touch msg.h msg.c

B)

%.h %.c: %.spec
	@echo Generating $*.h and $*.c from an interface description.
	@sleep 2
	@touch $*.h $*.c

[rocky]

Yes, I had read the example you gave in the link to Eric Melski's January 8, 2009 article.

As I said, I don't doubt it is a problem others encountered. Although I thank you for bringing to my attention a situation found in GNU make that causes problems, right now I'm not looking for problems to solve.

When I do, I sometimes look at the ranking of projects and open issues. And here I note that this project is not the highest-ranked projects of those I've written. In fact it looks like you haven't even upvoted it.

Clearly this is something that is important to you that you felt compelled to mention it .

There's nothing stopping you from working on it yourself. Or hire someone to work on it. (Contact me privately if you want me to work on it.) Or convince enough of your friends or interested parties to upvote both this issue and the remake project, so that the community has indicated that it is worth my time to do it free for the community.

[elfring]

Clearly this is something that is more important to you.

It seems so just because I came along affected make scripts where it would be nice that not only a serial build can be guaranteed to be correct.

[rocky]

For others who find elfring just has hard to understand as I do, see also: https://savannah.gnu.org/bugs/?51338

Apparently this idea, or something motivated from the same kind of problem has been aired 3 months ago as a GNU Make bug, although it was closed as "not a bug".

In that discussion, the train of thought was to provide additional built-in function(s) or extend or change the expressiveness of some functions to assist parallel make builds.

It appears that, then as now, although the problem is well observed there are several ways this could be addressed. But no concrete idea has been accepted, if in fact it is fully thought out.

In that bug report, the GNU make maintainer says that changing the makefile language is something that should be addressed in the gnu make mailing list, not as a bug or feature report. That I guess spawned the link cited at the beginning: "the handling of dependencies for (parallel) software builds".

Furthermore, this direction is not something the GNU make maintainer cares to explore.

On the other hand, adding additional functions GNU make is open to, provided it is well specified. But that so far hasn't been done either.

The idea of providing a lint check is I guess new in this discussion.

[rofl0r]

thanks for the info. i too had a hard time making sense out of his ultra-minimalistic snippets of information, including the referal to people that are not even participating in the discussion.

[elfring]

🔮 Would you like to achieve any collateral evolution according to functionality which is supported by the software combination “GNU make 4.3”?

[rocky]

Are you volunteering to make this happen?

[deliciouslytyped]

Edit: consider the content of this post moved to https://github.com/deliciouslytyped/make-groupedtargets
This post is going to be a bit off topic, but it seems as good a place as any to summarize some things;

this seems like one of those somewhat more obscure features of make that is(was?) bafflingly missing decent support.
Edit: (I mean in make itself, not remake.)
With the research I've been able to manage, things do seem to point to gmake's pattern rules being the cleanest, albeit still hacky solution to multiple-output targets. This may have been fixed by "grouped targets" in 4.3, see [3] and [4].

A link I haven't seen mentioned here yet is https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html , which goes quite in depth with the pertinent issues. (It is also this article which leads to my bafflement to the lack of a decent solution; you can't expect everyday developers to have to deal with this, and I myself don't understand the contents of that article yet). Given that it's in the automake documentation, maybe automake has support for dealing with this, but I haven't gotten to the point of researching that.

Perhaps the issues listed in that document are also a good test for remakes debugging capabilities.

To aggregate some of the above, the most useful things I've found on the topic so far are:
[1] https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html
[2] https://www.cmcrossroads.com/article/rules-multiple-outputs-gnu-make - as linked previously (I actually found this because I didn't believe that was pattern rules were really gmake's solution...)
[3] https://lists.gnu.org/archive/html/info-gnu/2020-01/msg00004.html linked above, has some content:

* New feature: Grouped explicit targets
  Pattern rules have always had the ability to generate multiple targets with
  a single invocation of the recipe.  It's now possible to declare that an
  explicit rule generates multiple targets with a single invocation.  To use
  this, replace the ":" token with "&:" in the rule.  To detect this feature
  search for 'grouped-target' in the .FEATURES special variable.
  Implementation contributed by Kaz Kylheku <address@hidden>

This looks interesting, and I should look into it. If not for this thread, I don't think I would have ever found this.
Grouped targets appear to be documented at
[4] https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html

There is a little bit of exposition at https://stackoverflow.com/a/63935393 and a common pitfall seems to be trying to use it with older versions: https://stackoverflow.com/a/63162448 , https://stackoverflow.com/questions/61345698/gnu-make-grouped-targets-are-not-grouped

Here is a proposed solution for the lack of an automatic variable expanding to all grouped targets; https://stackoverflow.com/questions/60402075/is-there-an-automatic-variable-for-all-grouped-targets-in-makefile/60441626#60441626

https://lwn.net/Articles/810247/ reports to have a portable implementation of grouped targets at https://github.com/david-a-wheeler/make-booster

Also I should probably correct my terminology, people seem to say multi-target, though I don't think that's obviously the same thing as multi-output. Maybe the term is now grouped targets.

• If indeed grouped targets are what's wanted for multiple-outputs, the automake documentation should be updated to reflect this.

Now if only The GNU Make Book or Managing Projects With GNU Make was updated. :)

If anyone has any additional resources to add to this list, that would be cool.