Support for multi-layered fanout/fanin

[brettviren]

TBB limits the multiplicity of fanout/fanin to something like 10. We need a way to produce subgraphs with larger fans.

This should be added to pgraph.jsonnet at top level.

I see in the very specific pgrapher/experiment/dune-vd/funcs.jsonnet that it has grown to include a multifanpipe function. Perhaps this is partly what is needed. However, some problems:

• We need just fanin/fanout separate from "fanpipe" `pattern
• These functions need to be shared at top level
• The funcs.jsonnet files should be removed as pgraph.jsonnet is the correct location
• In any case, top level files should not be copied into experiment areas as that hides changes in the top-level files and encourages import-chaos

[brettviren]

Some comments/suggestions on how to do this:

It would be best if the multifan* functions can be called identically to the existing fan* functions and produce the same results as the current fan* functions in the case of low multiplicity. If that can be done, then they should replace the current fan* functions and not add any new multi* names.

If the high-multiplicity case requires additional function arguments, they can be added to these new functions as long as they have default values that are relevant to the low-multiplicity case. I think this means that the fin* and fout* arguments for the current multi* functions should default to empty lists.

[brettviren]

#238

[brettviren]

One good thing about ignoring bugs is sometimes they get fixed by others.

I just happen to check in on this issue and as of oneTBB 2021 the hand‑enumerated specializations are now fully variadic. In principle, there is no limit to the multiplicity.

However, there are some practical scaling issues to consider before embracing a fan that spans an entire DUNE FD module. Here are some scaling numbers with oneTBB 2023.1 / g++ 13.3.

ports	policy	port types	compile	text	.o
2	queueing	same	~5.0 s*	52 KB	186 KB
150	queueing	same (int)	6.5 s	110 KB	672 KB
200	queueing	same	8.4 s	134 KB	921 KB
150	reserving	same	6.2 s	143 KB	786 KB
100	queueing	distinct	30.3 s	791 KB	4.5 MB
100	queueing	same	4.5 s	99 KB	516 KB

*The N=2 time is mostly the fixed cost of parsing flow_graph.h; subtract ~3 s of that from every row to see the marginal node cost.

The other issue is single N-way fanin is still a big sync point making the graph as slow as the slowest upstream pipeline while holding progressively more data in memory. The hierarchical fans we end up using to get around the pre 2021 limit may still be best.

[brettviren]

I think the direct issue is solved with the Jsonnet move so I close.

[HaiwangYu]

@brettviren Thanks for the information. I think we need some extra effort to support more fanout with in WCT?
For that I opened a new issue to trace that #495