[PSA] Default Port Id type

[samar-abdi]

The current default port ID type is a 10 bit bitstring. Is there a strong reason for this? It would be preferable to have a default 32 bit port ID type for couple of reasons:

1. SDN apps that need to be ported from the Openflow version to an equivalent P4 version already use a uint32 to define port numbers.
2. P4Runtime defines uint32 port ids for packet IO.

Of course, one could override the Port ID type, but that would be needed across all P4 programs and seems like an unnecessary cognitive load/ potential for errors.

[jafingerhut]

Do you mean that you would prefer it if the psa.p4 file that one #include'd in P4 programs defined a port ID type to be the same as bit<32>, regardless of the target device?

If so, I believe the motivation for allowing it to be significantly smaller in P4 programs is that bits of storage space will be at a premium in many such implementations. If one had a 32-bit port ID as an action parameter for a table with a million entries, for example, and all of those port ID values were in the range 0 to 63 for that device, then you are wasting 58 Mbits ~= 7 Mbytes of relatively small on-chip memory in an ASIC or FPGA that achieves the highest packet rates. Don't compare that 7 Mbytes to general purpose CPU DRAM sizes (typically tens or hundreds of gigabytes) -- compare it it on-chip on CPUs with typically several tens of Mbytes, maybe low hundreds of Mbytes total. That 7Mbytes would be filled with all 0s all of the time.

Are you concerned that the control plane will need to trim 32-bit port ids down to platform-specific bit widths in order to create P4Runtime API calls?

[samar-abdi]

While this is a valid concern, I am not aware of direct tables with actions that have a port ID parameter. Port IDs are typically used in L3 routing actions, and they are best modeled as indirect tables, so we will end up with only few tens of action entries even if there are a million flows.

By the same token, if an action takes src and dst mac as parameters, a direct table with a million entries will require ~96MB for storing actions alone!

The problem is in control plane portability as you say in your last comment. Since P4Runtime uses uint32 (in network byte order) for port ID, a P4Runtime server on the switch will have to interpret and map that uint32 value to a 10-bit value. And different switches may do so differently.

[jafingerhut]

PortID is only one of several fields proposed in the PSA right now to be a type bit where the width in bits W can differ across PSA implementations.

I am wondering if there is a straightforward way in P4Runtime to define these as uint32, or whatever width you think is convenient in P4Runtime, and yet in the P4 source code and the data plane implementation, the field is exactly the width specified in the P4 source code? e.g. control plane writes would check that the value fits within the bit size of the implementation, with the most significant bits of the uint32 that don't fit being discarded, or even better checked to ensure they are 0, and if not returning an error to the entity that created the out-of-range update request? Reads by the control plane would simply 0-fill the data plane value up to a uint32.

If this could be done, it would enable data planes to have different widths for these things, but the same control plane API would work for all of them. Presumably there would be a way for P4Runtime to query what range of port numbers is supported by a particular device, so that the control plane would know which port numbers not to try using at all.

[samar-abdi]

Thanks Andy. I agree. Probably the best option is to use a 'bytes' type for port ID (and other unspecified width fields) in P4Runtime. We already do that for match fields and action parameters. I'll start a PR for the change in P4Runtime and we'll see how that goes.

On a separate note, do we really need to specify a default bitwidth? A user may inadvertently specify a 32 bit Port ID for one domain and not specify anything for another domain (ending up with 10 bit port ID). Probably better to catch this at compile time?

[jafingerhut]

@samar-abdi On your last question, this reminded me that your first sentence of your original issue confuses me: "The current default port ID type is a 10 bit bitstring."

Do you have a link or other reference where you find this default port ID type of 10 bits? I am not familiar with it.

[samar-abdi]

It is line 30 of psa.p4
https://github.com/p4lang/p4-spec/blob/master/p4-16/psa/psa.p4
My request is to remove lines 30-36 from psa.p4 that sets default types for standard metadata.

[jafingerhut]

Right now those lines do serve one purpose -- it lets us implement 'make check' for that psa.p4 file and verify that several example P4_16 programs that #include it compile correctly.

Maybe we could add a comment there indicating why those lines are present? I do not think of them as default values -- merely examples.

[samar-abdi]

OK, thanks. I was wondering if it would be ok to separate psa.p4 into psa_core_types.p4, psa_externs.p4 and psa_pipeline.p4 for the sake of modularity. It would also make it easier if one wants to extend the psa externs or define their own programmable pipeline/package.

[jafingerhut]

Making a separation like you suggest would make it more convenient if someone wanted to use psa_core_types.p4 and psa_externs.p4 in another architecture, but they wanted to use something other than psa_pipeline.p4. Seems like a good idea to me.

[vgurevich]

@samar-abdi ,

Sorry to be a little late to the discussion...

port ID is the physical ID that is actually used inside the hardware. Specifically it is used to communicate between fixed-function and P4-programmable components of the device and unless this device has specialized remapping hardware, they cannot be reliably remapped just in P4.

Therefore, the right thing to do is to perform remapping in the software. For example, we can have the following declaration in psa.h:

@api_type("my_port_id_t") typedef bit<HW-specific-width> port_id_t;

This will tell the API generator that every time it needs to generate an API which involves something of the type port_id_t in P4 program, the corresponding API structure member or function parameter should

1. Have the type my_port_id_t (and it can be a 32bit number)
2. Should be translated using some unspecified mechanism on both input and output. This would allow you to use front panel port numbers for almost all intents and purposes
3. If you generate CLIs it will also allow you to generate code that will display port name in a system-specific way and accept inputs in special form, such as "xe10" or "1/20"

Most importantly, you will be able to use the same type when you work with the APIs for the fixed-function devices, e.g. MACs, PRE, etc. They typically do not have remapping in them and thus you will have to have such a layer anyways.

In my view this is much easier, efficient and more reliable than trying to force PSA to carry software port numbers around.

Thanks,
Vladimir

[jafingerhut]

@vgurevich Are you saying that there must always be some layer of remapping of Port ID values between the control plane and the P4 code?

Or are you saying that you expect at least some platforms will want to have such a remapping, because P4 program Port ID values will be cryptic hardware-specific numbers that aren't front panel port numbers, but the P4Runtime API should never have to deal with those?

[vgurevich]

@jafingerhut ,

Not necessarily. What I am saying is that if control plane wants to be portable at PD level (and the level of the corresponding APIs for fixed-function devices such as PRE), then it needs to perform port remapping.

There are other solutions for control plane portability. For example, one can define higher-level APIs on top of P4Runtime and fixed-function APIs and perform remapping in these higher-level APIs. This is actually a fairly common practice.

It is also possible to have remapping done even higher up.

So, it's a choice, but if @samar-abdi wants to have portability at P4Runtime/fixed API layer (and I want to re-emphasize that just P4Runtime will not be enough in the real-life situations unless the APIs for the fixed-function components are folded into that framework), then port remapping must be done there.

[jafingerhut]

So maybe no one minds this, but if there is some kind of port ID numerical value remapping done between one level of the control plane software, and the P4 program, then the following kinds of things become tricky at best, and perhaps should be explicitly disallowed:

• P4 port ID values in search keys should probably be restricted to 'exact' match_kind. If they were 'range', 'ternary', or 'lpm', then depending upon the numerical remapping going on, those might be nonsensical (e.g. if the port ID numerical remapping going on was 'reverse the bits in the uint32 port ID in the control plane to construct an 8-bit port ID in the P4 program', then 'lpm' and 'range' can't easily be translated from the control plane to the P4 program behavior). I am not saying that example is a common thing someone wants to do with port ID values in a P4 program, but probably best to warn people not to use such match_kind's on port ID's in PSA portable programs, if it is so tricky to support them.

• Casting Port ID values to bit types, or from bit types to Port ID values, would be a suspicious operation, easily messed up, if it meant that the control plane API didn't know the P4 program was taking a bit type, and later casting it to port ID.

If a port ID is used as part of an exact match key in a P4 table, or as an action parameter of a table's action, should the P4Runtime API have a special "PortID" type in it, so software knows to remap those values, and not other bit values that happen to have the same width, but do not represent Port ID values?

The same questions would apply to any other types of values, other than port ID, where you expect there to be remapping of numerical values between the control plane API and the P4 program numerical values. Are there any others besides Port ID where you expect this to happen?

[vgurevich]

@jafingerhut ,

Your understanding is totally correct in terms of effects, but I think all these are not result of the remapping, but simply the result of the fact that HW port numbering often has its own logic, which is quite different from the SW requirements. The remapping (or the need for it) is the result of the same problem. And the fact is: it is quite real.

You are totally correct enumerating the potential consequences. Yes, in order to be truly portable, the data plane program should not make assumptions about specific values of port ID and everything you said are classic examples of what not to do. Another important example is that the value for the CPU port should be parameterized, i.e. one cannot rely on it being always 0, 0xFFFFFFFF or anything else. Instead, PSA should probably state that an architecture can define a const CPU_PORT_ID (or an extern if it needs to be configured at runtime) and that's what should be used.

In terms of casting, I'd simply state "be careful and understand the consequences", but I would not try to mandate anything beyond that. Casting itself serves as a nice "yellow flag", IMHO.

Just in case you might wonder about why I sometimes have stronger opinions about what to mandate and sometimes less so, the answer is: I am trying to mandate more when I see that something might not be easily implementable (and not only on Barefoot, but on other devices) and I am trying to be more hands off when things are easy to implement, but simply allow the programmers to screw themselves if they choose not to follow common sense practices.

[samar-abdi]

@vgurevich Thanks for the clarification. This is what I see potentially happening:

1. Keep PSA as is.
2. port id type is network order bytes in P4Runtime packet IO (and potentially other) API.
3. An org may define port id type (say bit<32>) in the P4 program.
4. The PD API defines a uint32 type for port ID (since the P4 program uses bit<32>).
5. The control plane apps think it is OK to use the 32 bit space for port ids, since that is what the PD API suggests.

If vendor A internally uses 8-bit port ID and vendor B internally uses 16-bit port ID, then should we have two versions of the P4 spec? And is it the responsibility of PD API to recognize this and do the appropriate mapping from the universe of 32-bit port IDs used by the control plane apps to the vendor-specific 8-bit or 16-bit port ID? I don't know how PD API can do that without knowing the vendor -specific port ID space and keeping state, both of which we strongly want to avoid.

IMHO, there will need to be some standardization of port ID space and special ports such as CPU_PORT across switches for control plane software to be portable. And this space will need to be clearly defined in a P4 spec that is used across the org. A vendor may use which ever port implementation in hardware as long as their P4 compiler can map the port ID space to their internal hardware implementation.

Whether this standardization is done in PSA or outside is debatable. My vote will be to do this standardization in PSA. I may be missing something, but I do not see the upside of making the port ID types programmable.

[vgurevich]

@samar-abdi ,

Your understanding is correct: the definition of the type for the port ID in PD API (or any other "front-end" API) does not have to match the definition of port_id_t in PSA "bit-for-bit". This statement, by the way, is not limited just to port_id_t. Other types might also warrant some mapping in PD. For example, there are two schools of thought with regards to IPv4 addresses: some people prefer them to be in network byte order, while others might like them in host byte order. Again, in such cases the data plane itself should stay the same -- it's the API that can easily perform remapping, based on certain conventions. Obviously, some kinds of remapping (like the port number remapping) might be more involved and as a result one needs to be careful with various operations (see @jafingerhut comments above), but the nice thing that this is all can be done on top of the same data plane (and the same architecture), so each customer can decide for themselves what they want.

A couple of other thoughts, based on your statements above:

If vendor A internally uses 8-bit port ID and vendor B internally uses 16-bit port ID, then should we have two versions of the P4 spec?

First of all, just as a reminder, port ID is not a part of the P4 language spec, period. As for the PSA, the thinking is that we will require every vendor to define a type, port_id_t that will have the width, appropriate for a given target. This is very similar to how types are often defined in C. For example, int is always present, but can be 16 or 32 bits wide, "long" can be 32 or 64 bits, etc. Similarly, you do not know what pid_tor size_t are mapped to and portable programs are supposed to work regardless of what it is.

is it the responsibility of PD API to recognize this and do the appropriate mapping from the universe of 32-bit port IDs used by the control plane apps to the vendor-specific 8-bit or 16-bit port ID?

Yes, it is the responsibility of a control plane API. Note, that ultimately the users think in terms of front-panel port numbers, whereas the chip "thinks" in device-specific port numbers and their mapping from one to another is quite complicated and board-specific and, sometimes, dynamic, so remapping is unavoidable. Then, why not do it on one place only?

I don't know how PD API can do that without knowing the vendor-specific port ID space and keeping state, both of which we strongly want to avoid.

Such a state has nothing to do with the device running state (e.g. there is no need to recover it during HA events). It is under the control of the control plane anyway. Therefore while I understand the general desire "to avoid keeping state" I do not see how it is applicable in this situation. Yes, the API implementation will need to know the vendor-specific port ID space and then somewhere it needs to be aware of the board-specific mapping. All these are very easy to define and this needs to be done anyways.

IMHO, there will need to be some standardization of port ID space and special ports such as CPU_PORT across switches for control plane software to be portable. ... Whether this standardization is done in PSA or outside is debatable. My vote will be to do this standardization in PSA.

I understand your desire, but it requires a lot of HW support. And it is not cheap -- every time you do remapping, you lose valuable processing cycles. And, again, even if you do it, it is only a half of the equation, the second half being board layout. We can put anything we want into PSA, but if hardware can't do it (or you get the hardware than is expensive, slow or both), what's the point?

[samar-abdi]

@vgurevich
Thanks for the detailed reply. By P4 spec, I meant the P4 specification model, i.e. the P4 program that we are using to define the switch behavior (sorry for the confusion). I am not advocating doing remapping in the switching ASIC, but in the P4Runtime service that runs on the local CPU on the board.

For instance, consider the classical set_nhop action in an L3 routing table. The output port is a parameter of this action. When the control plane programs a flow with set_nhop action, it needs to know the output port parameter type (which should ideally be derived from the P4 program). The implementation of the flow programming operation (in P4Runtime service) should remap the logical P4 port to the HW specific port. All of this is done in software on the local CPU, so there is no hardware cost.

Indeed, there will be some effort to port the P4Runtime service from one switch vendor to another, but the control plane can become switch agnostic by relying on only two entities: the PSA-targeted P4 program and the P4Runtime API.

[vgurevich]

@samar-abdi

Excellent. In this case we are in agreement. P4Runtime is a possible place to do it or it can be done a little higher up the stack as I outlined before. This is a choice that control plane designers should be allowed to make.

If you want this to be supported in P4Runtime or one of its frontends, you need to engage the P4 API working group.

[jafingerhut]

@samar-abdi @vgurevich Is it reasonable to say that the P4Runtime API can always use the same numerical value to specify the port id of the "to/from control CPU" port, and that any translation between that numerical value and what is needed in a particular P4 data plane implementation should be done by a "driver software" layer between the P4Runtime API and the P4 data plane?

That would enable the P4Runetime API to avoid the need of keeping track of different numerical values for each device, and allow the P4 data planes of different implementations to use whatever internal numbering scheme is convenient for it.

[vgurevich]

@jafingerhut ,

Port remapping can be done in many different layers and the choice depends on the particulars of a given system design. I would not recommend mandating remapping in P4Runtime, but I am totally fine suggesting this as an option.

Sent from my Samsung SM-G930T using FastHub

[jafingerhut]

I am not proposing remapping inside the P4Runtime implementation, but beneath it, i.e. somewhere between P4Runtime and the P4 program.

I'm sure you have heard an earful of feedback on this from 17 different sources, and I have no big horse to bet on in this race, but saying that you can do it beneath P4Runtime, or above, seems like a recipe for confusion.

[vgurevich]

@jafingerhut ,

I think that the easiest way out might be to leave it to P4Runtime. In this particular case, PSA is totally independent from it (as long as we agree that port numbers do not have any special meaning and are platform/target dependent (including their width)). We do not need to know anything about P4Runtime in order to define port-related intrinsic metadata.

Please, note that P4Runtime is supposed to be a generic framework, that should be applicable to any architecture, not only to PSA. All these remappings are not only architecture-specific; to a certain degree they are program-specific and more often than not they are also system-specific. At least for me, it is easier to know that the bottom layers are transparent and easy to understand -- then I can easily build whatever I need on top of them instead of fighting with them.

[cc10512]

It is clear that we need some sort of mapping mechanism and that the PSA is not the right layer for this. I think the PSA should stick closer to the physical layer, and as @vgurevich rightly points out, keep it so that the port numbers have no special meaning. As long as we provide externs for all entities that need meaning, it seems that it should be straightforward to not overload ports. Happy to let this one to P4Runtime to solve :)

[samar-abdi]

Agree with remapping left outside PSA. However, can we consider adding methods to the PRE and BQE externs that send packets to CPU? IMHO dropping packets, multicasting and sending/cloning to CPU are integral to expressing switch behavior.
In short don't mandate how we identify a CPU port, but have externs that must be implemented to send/clone packets to CPU. In fact, this way, we might be able to leave port ID remapping outside P4Runtime. My apologies if this has already been considered.

[hanw]

@samar-abdi There is a proposal to use the clone extern to support sending packet to CPU. It uses a metadata clone_spec to configure PRE to set the destination of the cloned packet to CPU. PSA does not specify what value is used to identify CPU.

[samar-abdi]

@jafingerhut @vgurevich @cc10512
We had a long discussion about standardizing port IDs, and P4Runtime does not seem like the right place to do it either. A PSA-targeted P4 program may define a port id type, and P4Runtime would need to define a set of standard ports for that type. Since P4Runtime is P4 program independent (also for portability), we cannot define values on unspecified type.

I like @jafingerhut 's suggestion of implementing remapping in SW below P4Runtime. In that context, it would be useful to have a logical notion of ports in PSA as opposed to a physical notion.

Defining a standard set of SDN ports and port type across switches is essential to control plane portability. If the SDN ports and port type are not standardized in either PSA or P4Runtime, the controller will have to release a "standard" P4 header file with port id type and standard port definitions + some config that does a mapping of standard SDN ports to a vendor's ports. Definitely not a desirable option.

IMHO, we should try to resolve this issue within PSA/P4Runtime and not punt it to the controller :)

[vgurevich]

@samar-abdi ,

I think that P4Runtime is not only program-independent (obviously), but also should be architecture-independent. Basically, if I am not mistaken, the idea is that it becomes the common framework (protocol/API) to control any P4 program, running on any P4 target that implements any architecture.

If you agree with the statement above, you would probably agree that standardizing port numbers will be fairly "heavy-handed", would it not? Because if we accept the existence of this layer, then it is equivalent to saying: if you access what P4Runtime thinks is a port with the number 0xFFFFFFFF (or, should it be 0 or something else?) , then it has the following semantics across all programs/architectures/targets (e.g. the packet will go to the control plane). And also it will have all the other consequences @jafingerhut described before: these entities cannot participate in anything but exact match, the semantics for the values that are outside of port range will be difficult to define, etc, etc.

Also, a lot will depend on whether or not P4Runtime will be able to cover non-P4 components (PRE, TM, etc.) otherwise, how would you ensure that the port mapping is the same?

Again, this all makes sense in the context of a particular development project, but I would not recommend imposing all this on each and every user of P4/PSA/P4Runtime.

[samar-abdi]

@vgurevich
P4Runtime is architecture independent, but has special API for counters, meters, action profile members and groups, all of which are outside the P4 language but PSA entities. So, it is not entirely PSA agnostic. It is just that a switch without support for action profile members will reject any attempt to manipulate action profile members via P4Runtime.

Also, I am not convinced that standardizing port numbers is heavy handed. It is practical. It enables portability at the expense of flexibility in logical port numbering. In a world where every vendor is free to define their own logical port bitwidth and numbering, we are inviting trouble for the controller.

Imagine a SDN controller talking to two switches from two different vendors who have chosen to define their own port types and numbering. Clearly there will be two different P4 programs for the same forwarding logic that the controller wants to see in the fabric. Which means that when pushing a forwarding pipeline config to all switches, the controller will need to check the make and model of the switch. Things get even worse if there is a match on a port id in one of the tables. Now, the controller will have to keep track of which match key to build for which switch. And then we have lost all notion of being vendor agnostic, which is by far the greatest attraction of P4.

Perhaps there is a difference between how we see PSA. IMHO, it should attempt to define a logical programmable forwarding architecture that enables plugging-in switches from different vendors in a fabric without having to make major changes to controller software. The controller should be able to push the same PSA-targeted P4 program to all PSA-compliant P4-programmable switches.

[vgurevich]

@samar-abdi ,

You make a number of important points here that I'll try to address in some order.

What's the goal of PSA?

I agree that the current PSA specification does not explicitly articulate the goals of PSA, which is unfortunate. This was discussed in the meetings before, and if I understand it correctly, full control plane portability and silicon independence was deemed to be out of scope for the project, not because it is not a worthy goal (it is), but because it involves a lot of other components that PSA is not in control of.

In my view, the goal of PSA is more limited: it is to ensure that there is a common architecture, implemented on more than one target, that a program, written to that architecture and following some simple rules and limitations can execute the same way on these targets. Even that is actually a difficult goal, so I'd recommend getting there first.

I recommend you bring it up in the meeting and perhaps we can codify it in the document to make sure everyone is on the same page.

The greatest attraction of P4 is being vendor-agnostic

I think P4 has a lot of very nice "properties" and which one is the most important is probably different depending on who you ask. Portability usually comes at a price, such as reduced functionality or performance (something PSA spec already mentions) and we need to be careful, to make sure that price is paid only by the people who need it and it doesn't affect people who believe, for example, that the biggest attraction of P4 is the ability to define a custom data plane algorithm.

Is P4Runtime PSA-specific or architecture-agnostic?

@antoninbas can correct me, but the fact that the current incarnation of P4Runtime is relatively tightly coupled with PSA is probably more of a reflection of how it is being developed (on PSA), rather than the end goal. It is true, that we do not currently have a mechanism that would allow an arbitrary architecture specify the control plane interface for its externs and therefore it is currently manually written in P4Runtime just for the specific externs it is aware of (which happen to be PSA ones), but it doesn't have to be that way always.

What should an SDN controller do?

I agree that in a multi-vendor environment, it would be nice to be able to simplify SDN controller and allow it, for example, to use the same port numbers everywhere. However, I think that this kind of standardization is more customer and application-specific. Not every SDN controller is going to "think" in terms of P4Runtime and not every agent is going to necessarily use "raw" P4Runtime as the control interface. For those that will, I am fairly certain that most vendors will be providing port remapping infrastructure and you can mandate that it is there. You can also require their P4Runtime implementations to perform remapping and so on. The question is whether the same requirements should be imposed on everyone else throguh standardization, and I feel that they should not.

[antoninbas]

P4Runtime already exposes a mechanism to extend the API and support arbitrary externs. However, we decided that P4Runtime would provide "native" support of all PSA externs.
I support the proposal by @samar-abdi at 95%. In a nutshell, my vision is that if I should be able to take the exact same P4 program and the exact same P4Runtime messages and use same across vendors. More precisely:

• I believe the port type should be standardized in PSA, with a fixed bitwidth. This port type should be the logical port type.
• The mapping from logical to physical port should be defined for each new system. This mapping will be made available to the compiler and to the switch runtime stack, which must enforce it.
• I do not believe however that there is a need for PSA to define standard port values, e.g. for the CPU port. This is not needed for portability IMO.
  Since silicon-independence is actually one of the primary goals of the P4 API WG, support for logical ports is something of major importance in the development of P4Runtime.

[vgurevich]

@antoninbas ,

As we discussed in the meeting, everyone agrees on the vision. It should be possible to take the same P4 program, compile it on two different PSA-compliant devices and send exact same P4Runtime messages. Thing is P4Runtime is the best solution for this goal.

I do not believe this vision requires ports to have fixed bitwidth in PSA. All that is required is for P4Runtime implementations from diferent vendors to accept fields of a certain bitwidth (standardized by P4Runtime) as logical port numbers and to perform the translation on a back end before accessing the device (or after reading it).

For example, P4Runtime can universally mandate that there is a concept of a port ID (logical port ID) and it is a logical 32-bit number. Then, each P4Runtime implementation will translate it into 9-bit number on one device and into 8- or 10-bit number on another. Moreover, in the process it can perform board- or system-specific translation as well. All that it is required from the P4 program and the control plane is to treat port numbers as enums (i.e. only do exact match on them, don't do ternary or math, etc.). All that is required from the compiler is to clearly mark which fields represent ports, which is easily achieved either by designating a type or using an annotation. This will eliminate the need for the compiler to be aware of any specific mapping. Moreover, these logical port numbers can perfectly co-exist with the real, "physical" port numbers if one wishes to have that and without any additional expense on the data plane resources, which are really scarce (do you really want to do 32-bit match and keep 32-bit action data for ports when 9 bit is all that is required)?

As discussed, similar approach will be required for all enum values, such as parser errors, meter colors and any other user-defined enums.

[samar-abdi]

A common use case is to prepend a header that includes the ingress port number, and punt to CPU. With the current approach, the ingress port number will be the physical port number for that specific device and the controller would need to keep a map from physical to logical ports for all switch types, and it will also need a map from switch id to switch type to figure out which port map to use.

We cannot achieve this with P4Runtime alone, because it would require the API to modify the packet!

The alternative is to add a table that matches on ingress port number and prepends the logical port number, effectively doing a translation in hardware. Again, to generate flows for this table, the controller will need to know device-specific port numbering and mapping information.

I do not think there is a clean way around this issue without some standardization of port numbers, and an understanding that port numbers being referred to in a P4 program are logical, not physical.

I am very wary of leaving port definition to the switch vendor, then having to look up the datasheet to know which port number means what, then encoding a physical-to-logical map and having a set of hard-coded maps in the controller code-base.

Also, while the savings of 9-bit vs 32 bit match is desirable, the match keys typically get much wider, considering ipv6 addresses and mac addresses.

[ChrisDodd]

This seems to indicate that the P4Runtime should have some visibility on packets received from the dataplane via the CPU port, so it CAN (effectively) modify the packet -- more likely it should have an interface to extract target specific data from the packet and convert it into a target-neutral form.

[samar-abdi]

In that case, P4Runtime would need to know about all variables of Port Id type in the P4 program, effectively making it program dependent. Which means every time we change the P4 program, the P4Runtime API will change and we will need to re-image the switch.

@vgurevich @jafingerhut if we define a 32-bit port ID in a P4 program, how does a switch populate the 32 bits of ingress port in parser input metadata? Would such a P4 program compile only on targets with hardware port id <= 32 bits long; And does it 0-fill the unused MSBs in ingress_port?

If 32 bits being wasteful is the biggest concern, we can talk about reducing the width. Leaving it unspecified IMHO is the really problematic part. With a fixed bitwidth, P4Runtime can define a canonical port enumeration, and the controller and switch stack can adapt accordingly.

[jafingerhut]

@samar-abdi @vgurevich Do you know if any of this is yet documented explicitly in the P4 Runtime API? I don't think the PSA draft mentions any of this yet. If it isn't mentioned in either, I think it would be good to have a few sentences about it in the PSA at some point.

[jafingerhut]

@samar-abdi @vgurevich @antoninbas Have you thought of what should be done in the following scenario?

Suppose you want to write a P4 program that can send a value of type PortId_t (e.g. the ingress port on which a packet arrived) in the header of packets copied from the data plane to the control plane.

You want to write one P4 source program that should be portable across multiple PSA implementations.

Several of those implementations restrict headers to be a multiple of 8 bits long. You want to put the PortId_t value in a header, but you don't know how many bits each PSA implementations use for type PortId_t.

How do you write the definition of the header that includes the PortId_t field, such that it always ends up being a multiple of 8 bits long?

A similar scenario exists if the control plane wants to send a packet with a header containing a port number to a data plane, e.g. because the control plane expects the P4 program running on the device to send the packet out of that physical port of the device. How do you define the header in the P4 program? (same answer should likely work for this case as the previous one)

One possible answer: Pick a bit width (e.g. 32 bits) for such port numbers, and require that in such places the P4 program must use a value of that type, maybe called something like "PortId32_t". Casting a PortId_t value to type PortId32_t, or vice versa, is always a correct thing to do in a P4 program written for PSA. Every PSA implementation is required to define its PortId_t type to contain at most 32 bits.

EDIT: Another minor variation on that idea: Create a type "PortIdBytes_t" that is guaranteed to be at least as many bits in size as PortId_t, and always a multiple of 8 bits long. It can be used in headers. e.g. if PortId_t is 13 bits on a PSA implementation, its type PortIdBytes_t would be 16 bits wide. If PortId_t is 22 bits wide on a PSA implementation, its type PortIdBytes_t would be 24 bits wide. Both can include them in headers, as long as the remaining fields of the header add up to a multiple of 8 bits (but those fields presumably have widths known to the P4 programmer).

The same idea could be used for other types in PSA that have bit widths that can vary across PSA implementations.

[mihaibudiu]

Using 32 bits is probably safe. We had this choice in several other places, e.g., packet length.

[samar-abdi]

@jafingerhut
Yes, we have thought of this. This is a common packet-io case. The P4 program will define headers for packets to/from controller. The port fields in the headers will still use PortId_t to allow usual matching etc. There will be no translation in the dataplane. However, the port fields will be annotated with @port_translation(“controller_port_bitwidth : 32”). This will allow the P4Runtime service to intercept such packets and translate the port values back and forth. Of course it could be any bitwidth: doesn't have to be 32. The translation tables will be part of the switch config.

[jafingerhut]

@samar-abdi Thanks for the info.

You say "There will be no translation in the data plane". I understand the reasons for that.

However, will there be "padding" in the data plane? For example, if a PSA implementation has type PortId_t as an alias for bit<13>, is the expectation that I can have a field of that type in a P4_16 header definition, annotate it with @port_translation("controller_port_bitwidth : 32"), and the P4 compiler should pad that field up to 32 bits wide in the data plane code?

If so, makes sense.

If not, then I don't see how to write portable code that makes the header type a multiple of 8 bits long, and also includes a field with type PortId_t.

[samar-abdi]

Actually, the controller_port_bitwidth annotation tells the P4Runtime service to accept 32-bit port numbers from the controller, do a translation to HW port numbers and prepare the CPU header for the dataplane.

You are right that this may result in different sized CPU headers with odd bitwidths on different targets, which is not desirable. To avoid that you may define 16-bit Ports in the CPU header definition and still annotate them with @port_translation so that P4Runtime service does the translation for you. Of course, now the expectation is that only the least significant N-bits of the port field in CPU header should be populated, where N = bitwidth of PortID_t of the target AND N <=16 for all targets.

IMHO, the whole idea of unspecified bitwidths in PSA standard metadata is an impractical hardline approach, and it would make life simpler all across if we fixed these to some reasonable upper bounds. A smart P4 compiler and P4Runtime service can do all sorts of magic to compress the match key widths for hardware optimization, which appears to be the major concern here.

[jafingerhut]

I think one approach is as I mentioned in an earlier comment: a minimal-to-the-bit PortId_t type for minimal storage requirements for the data plane, and a padded-up-to-multiple-of-8-bits type, e.g. PortIdBytes_t, for use in headers. It could also be a reasonable upper bound size that we think is the maximum across all PSA implementations.

That said, we could certainly talk to P4 compiler implementers and ask about the idea of making types that are different widths for different uses in the same P4 program, e.g. 13 bits when used as a search key in the hardware, 32 bits when used in a header (only for a few specially marked types, not everything).

@mbudiu-vmw @ChrisDodd Any quick impressions about such an approach?

[mihaibudiu]

This is a long discussion, and I have not followed it entirely, so perhaps I am repeating arguments already made.

The dataplane should not do anything to your values that you don't explicitly indicate in your program. If you want a value to be widened you should use a cast between two different types in your P4 program.

The question is about moving values between the control plane and the data plane - either by control-plane API calls or by digest packets sent to the control-plane. But I think ports are nothing special in this case, the question applies to any odd-sized value, e.g., an action argument that is 13 bits wide.

[samar-abdi]

@jafingerhut I agree with your approach: PortIdBytes_t can be defined in the P4 program (outside PSA). If there is a need to populate this header field with a PortId_t standard metadata, say egress_port, we can use casting as Mihai suggested.

Perhaps I am missing something, but I don't entirely follow why we need CPU header to be byte aligned?

[jafingerhut]

It is not an absolute requirement in the P4_16 spec that headers must be a multiple of 8 bits long, but it is explicitly allowed that implementations can restrict headers to multiple of 8 bits long, and I believe that many implementations do, even simple_switch in p4lang/behavioral-model.

See P4_16 v1.0.0 language spec, section 7.2.2. "Header types", for this sentence:

"There is no padding or alignment of the header fields. Architectures may impose additional constraints on header types—e.g., restricting headers to sizes that are an integer number of bytes."

This restriction is not specific to CPU headers -- it is all headers in a P4 program.

Such a restriction can reduce the cost of implementing high performance parsers and deparsers, for example. I am not aware of any networking protocol that defines "complete" headers that are not multiples of 8 bits long. Most define headers that are multiples of 32 bits long.

[jafingerhut]

@samar-abdi - If we create a type like PortIdBytes_t, why not include it in psa.p4? I would expect a significant fraction of P4 programs written for PSA devices will want fields like this in to-CPU and/or from-CPU headers. Why make each program writer discover this issue for themselves and perhaps devise different solutions?

[jafingerhut]

Additional comment to my previous one. If we create PortIdBytes_t for use in to/from CPU headers, those values will need implementation-specific translation by a local P4Runtime API agent, just as much, and for the same reasons, as PortId_t values in table keys or action parameters.

[samar-abdi]

No harm including PortIdBytes_t it in psa.p4 and really not a concern on the controller side given the annotation.
I assumed it would just be easier for the P4Runtime service implementation to deal with consistent length packetio headers across heterogeneous switches (if there is common code that an org wants to use across multiple P4Runtime service implementations). Then again, in reality, it will just be 16 bits regardless.

[jafingerhut]

OK, I've got a proposed PR for the PSA spec that interested people can think about, here: #562

[jafingerhut]

Proposed PR has been merged in, so closing this issue.