Define and document flow control

[shefty]

There needs to be a mechanism for applications to enable/disable flow control, along with events defined when flow control or buffer overruns occur.

[shefty]

Flow control is an endpoint attribute or capability. The reporting of a flow control issue would likely go to the control EQ. The behavior of having versus not having flow control needs to be defined based on the endpoint type. The DGRAM EP would likely not use flow control. The MSG EP may or may not. If flow control is not enabled, the behavior needs to be defined -- possibly resulting in the EP failing. The behavior of an RDM EP is more difficult to define.

[shefty]

There may be different levels of flow control. E.g. IB only supports flow control based on receive buffers, not CQ space. With RDM endpoints, flow control may need to take CQ space into consideration. An app may need to be aware of what level of flow control is supported and when flow control overruns a the local CQ, remote CQ, remote receive buffers, etc.