Description of abstract methods

[mihaibudiu]

No description provided.

[jafingerhut]

@cc10512 @ChrisDodd I have a half-memory that the primary interest in abstract functions was for ones that could read any variable/packet-header-field in its lexical scope, not only parameters and the top-level scope?

Perhaps I am remembering that about virtual functions, and I am not really clear right now on whether those are the same as, or different than, this definition of abstract functions.

[mihaibudiu]

We discussed that we will introduce these features one by one.

[ChrisDodd]

Yes, all the use cases we have for them involve being able to access any variable in scope at the point of the definition of the implementation method.

[jafingerhut]

Mihai made a comment at the most recent LDWG about letting the desired use cases drive the addition of features to the P4 language. I think that is a good pragmatic approach, and support that goal.

I wanted to add a comment on this PR similar to something I said during that LDWG meeting, around the time that we were noting that the current proposal in this PR enables only pure functions to be used as abstract methods, i.e. no reading nor writing of variables in the lexical environment outside of the abstract method body, outside of those that are parameters:

This pure-function-only proposal seems to me to address 0% of the desired use cases of abstract methods (see Chris Dodd's comment from 2019-Jul-10 elsewhere on this PR). If I am missing any that are handled by this proposal, please correct me.

That seems to be at odds with the goal of letting the desired use cases drive the feature additions. If this addition gets into the spec as it is today (2019-Nov-12), without the additions that enable some significant fraction of the desired use cases, then we've added something to the language that no one wants to use, and will have to deal with questions of maintaining backwards compatibility in future language spec releases, for a feature that no one uses.

That is why I asked about soon addressing the use cases that motivated the addition of the feature, preferably starting long before the next language spec release, so we have time to think about the actual use cases and how to enable them.

I encourage anyone who has even one concrete use case that they would be willing to write up a sample code snippet, and the desired behavior/meaning of the code, to link to them from this issue. I suspect there are already some in other comments linked from this topic on the LDWG wiki.

[mihaibudiu]

I don't think this has 0% use cases; many of the applications described in the "Packet transactions" paper can be built using this tool.

[jafingerhut]

Is this the Packet Transactions paper you are referring to?

"Packet Transactions: High-Level Programming for Line-Rate Switches", SIGCOMM 2016, Anirudh Sivaraman et al, https://cs.nyu.edu/~anirudh/domino-sigcomm.pdf

If so, I'd be happy to re-read that paper (it has been a couple of years since I went through it), and try to write P4_16 code for v1model/PSA architectures, and see if I can figure out a way to do it without abstract methods, and if so, also if I can figure out a better way to do it with abstract methods. If you have any particular example from that paper in mind, I could focus on that one first.

[jafingerhut]

I haven't written translations into P4_16 of all of the examples in the Packet Transactions paper, but I have done so for about 5 of them, and looked at the Domino code for the rest of the examples.

Here is a link into my fork of the repo containing the Domino code, published by the authors of the paper, in particular a link to the 'flowlet' example given in the paper: https://github.com/jafingerhut/domino-examples/blob/master/domino_programs/flowlets.c

And here is a link to my demonstration of one way to translate that flowlet code into P4_16, using v1model's Register extern. It starts with a big comment block explaining my steps for doing this translation, that works for every example that I translated. It is very straightforward: https://github.com/jafingerhut/domino-examples/blob/master/domino_programs/flowlets.p4

As mentioned in comments there, I do not see how abstract methods enable a better implementation of these packet transactions in P4_16. I could be lacking imagination on that point, of course, and am happy to read any proposed examples showing how.

In particular, all of these actions require access to one or more registers, and values of fields from the input packet (or at least values calculated earlier before invoking the action), and several of them calculate values that are required by later code after calling the action. Those map naturally to in/out parameters of actions, but I don't see how one would write code using pure-function-restricted abstract methods.

For these examples, I don't even see how abstract methods would enable better P4_16 code, even if the abstract methods allowed reading and writing variables in the lexical context. So even if abstract methods were generalized in that way, can someone show code demonstrating how they make implementing these behaviors any better?

[jafingerhut]

This directory contains several other .p4 files that I wrote, that are hand-translated from the similarly named Domino source file with a .c suffix: https://github.com/jafingerhut/domino-examples/tree/master/domino_programs

[mihaibudiu]

You are right that you can describe these computations in pure P4, but I suspect that for a long time no compiler will be able to synthesize these into workable programs using the constraints of the available hardware.

Externs with abstract methods describe architecture-specific accelerators that can be partially programmed/configured. By instantiating such externs and programming them (by implementing the abstract methods) you are exploiting effectively hardware features.

One other way to think about such constructs is as being part of a "target language" rather than a "source language". Ideally, a mature compiler would consume a program like the one you wrote and generate code using such externs, which could then be efficiently mapped by the back-end. But I suspect that won't happen for a while.

P4 is all about offering a high-level language while exposing features of the target for people who need them. That's why P4 has such a funny execution model, with all these holes and unspecified events: because in real high-speed architectures not everything is programmable. Such accelerators exist in some architectures, and these constructs allow them to be exposed. Indeed, if you use such accelerators you cannot expect your programs to be portable (unless PSA would choose to standardize such accelerators).

[jafingerhut]

So, no example of using abstract methods to implement any of these example Domino program, then?

[mihaibudiu]

I could try to craft something, but perhaps a Barefoot contribution would be more precise than what I can summon.

[jafingerhut]

I broadened my imagination a bit, and came up with an example program that defines a new extern using a pure-function-only abstract method that works something like a register, and can be used to implement all of the Domino example programs (or at least most of them). It is a bit clunky, I think, but given an implementation of the new extern described, it shoe-horns in a way to get something like the effect of accessing variables in the lexical scope, but without doing that.

I wouldn't call it pretty, though. More like an exercise in working around constraints of the language: https://github.com/jafingerhut/domino-examples/blob/master/domino_programs/flowlets-abstract-method-pure.p4

[mihaibudiu]

I think this looks very good.
You can also push the hash2 method inside the extern, so you can feed sport and dport as inputs.

[jafingerhut]

You could do that, but the more you push inside of it, the less general purpose it becomes.

[jafingerhut]

Mihai wrote in an earlier comment, about my P4_16 program that used normal actions with register read, followed by assignment statements, then register writes:

"You are right that you can describe these computations in pure P4, but I suspect that for a long time no compiler will be able to synthesize these into workable programs using the constraints of the available hardware."

Why do you say this? That is exactly what the Packet Transactions paper shows how to do: compile programs like that to any target machine, if you are given descriptions of the stateful operations that machine can do at line rate.

[mihaibudiu]

1. There is still a long distance from a research paper to an industrial artifact. I know that Anirudh @anirudhSK is working to make this "more real".
2. Even if you assume that the compiler can generate such code from a very high level description, the externs would serve as a high-level "target language" that the compiler can generate code for.
3. There may be other classes of accelerators that may not be as amenable to the techniques described in "packet transactions", but which may still be described well using such externs. I imagine that FPGA-based systems come with libraries of accelerators that people can invoke, which could benefit from such a construct. @gbrebner can perhaps weigh in.

[jafingerhut]

So, regardless of whether the current proposal, restricted to abstract methods that are pure functions of their in and inout parameters, addresses 0% of the use cases, or some other number, it seems clear from reviewing the issues linked from this issue: #766

that most of the time that the topic of virtual functions or abstract methods has been raised, people have repeatedly and consistently said that their primary use cases would only be possible if the function/method could at least read variables from the lexical scope. If we do not address that until after the next spec release, it seems to me that feedback is being ignored, at the risk of hampering the LDWG's ability to make future changes that meet those needs, and/or adding much-less-useful, or possibly useless, language features.

It seems to me that those issues should be addressed head on, not postponed.

[jafingerhut]

@ChrisDodd @hanw @cc10512 Also, in reviewing the issues I mention in my previous comment on this, while there are often requests for reading variables in the lexical scope, I couldn't quickly find any examples of desired usage. Having 3 or so of those, preferably with different places that the abstract method was intended to be called, e.g. maybe one as an action selector, one as a register update of some kind, and one that is something else, would be a good starting point for discussion.

Without such examples, it seems like the discussion isn't very grounded in reality, but leads to imagination of possible ways to describe restrictions on when abstract methods are executed that we may or may not ever want.

[jafingerhut]

I have done a fresh review of this PR on 2021-May-03, and it looks good to me.

[mihaibudiu]

As agreed in the LDWG working groups from May 2021, we merge this in the spec.