core: Add endpoint options for max message size and inject size

Today, `ep_attr->max_msg_size` and `tx_attr->inject_size` define the
transport's size limitation for a single data transfer operation. It lacks
the flexibility to allow providers having different limits on different
categories of operations (msg/tagged/rma/atomic). At the same time,
applications can't express usage models that have very different size
limits (e.g, only need small send/recv, but want large RMA).

Here a set of options are added to the fi_getopt()/fi_setopt() interface.
These options allow the maximum message size and inject size be retrieved
or set independently for msg/tagged/rma/atomci operations. The existing
definition of `ep_attr->max_msg_size` and `tx_attr->inject_size` remain
unchanged -- they are the upper bound of the limits for all the data
transfer functions.

A typical application will call fi_getopt() to retrieve the default values
first, and call fi_setopt() if lower values are desired. If -FI_ENOPROTOOPT
is returned, `ep_attr->max_msg_size` and `tx_attr->inject_size` should be
used.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>

include/rdma/fi_endpoint.h

@@ -70,6 +70,14 @@ enum {
 	FI_OPT_XPU_TRIGGER,		/* reserved for compatibility */
 	FI_OPT_CUDA_API_PERMITTED,	/* bool */
 	FI_OPT_SHARED_MEMORY_PERMITTED, /* bool */
+	FI_OPT_MAX_MSG_SIZE,		/* size_t */
+	FI_OPT_MAX_TAGGED_SIZE,		/* size_t */
+	FI_OPT_MAX_RMA_SIZE,		/* size_t */
+	FI_OPT_MAX_ATOMIC_SIZE,		/* size_t */
+	FI_OPT_INJECT_MSG_SIZE,		/* size_t */
+	FI_OPT_INJECT_TAGGED_SIZE,	/* size_t */
+	FI_OPT_INJECT_RMA_SIZE,		/* size_t */
+	FI_OPT_INJECT_ATOMIC_SIZE,	/* size_t */
 };
 
 /*

man/fi_endpoint.3.md

@@ -568,6 +568,70 @@ The following option levels and option names and parameters are defined.
   shared memory will not be used and the implementation of intra-node communication
   is provider dependent.
 
+- *FI_OPT_MAX_MSG_SIZE - size_t *
+: Define the maximum message size that can be transferred by the endpoint
+  in a single untagged message. The size is limited by the endpoint's configuration
+  and the provider's capabilities, and must be less than or equal to
+  `ep_attr->max_msg_size`.
+  Providers that don't support this option will return -FI_ENOPROTOOPT. In that
+  case, `ep_attr->max_msg_size` should be used.
+
+- *FI_OPT_MAX_TAGGED_SIZE - size_t *
+: Define the maximum message size that can be trnasferred by the endpoint
+  in a single tagged message. The size is limited by the endpoint's configuration
+  and the provider's capabilities, and must be less than or equal to
+  `ep_attr->max_msg_size`.
+  Providers that don't support this option will return -FI_ENOPROTOOPT. In that
+  case, `ep_attr->max_msg_size` should be used.
+
+- *FI_OPT_MAX_RMA_SIZE - size_t *
+: Define the maximum message size that can be transferred by the endpoint
+  via a single RMA operation. The size is limited by the endpoint's configuration
+  and the provider's capabilities, and must be less than or equal to
+  `ep_attr->max_msg_size`.
+  Providers that don't support this option will return -FI_ENOPROTOOPT. In that
+  case, `ep_attr->max_msg_size` should be used.
+
+- *FI_OPT_MAX_ATOMIC_SIZE - size_t *
+: Define the maximum data size that can be transferred by the endpoint
+  via a single atomic operation. The size is limited by the endpoint's configuration
+  and the provider's capabilities, and must be less than or equal to
+  `ep_attr->max_msg_size`.
+  Providers that don't support this option will return -FI_ENOPROTOOPT. In that
+  case, `ep_attr->max_msg_size` should be used.
+
+- *FI_OPT_INJECT_SIZE - size_t *
+: Define the maximum message size that can be injected by the endpoint
+  in a single untagged message. The size is limited by the endpoint's configuration
+  and the provider's capabilities, and must be less than or equal to
+  `tx_attr->inject_size`.
+  Providers that don't support this option will return -FI_ENOPROTOOPT. In that
+  case, `tx_attr->inject_size` should be used.
+
+- *FI_OPT_TAGGED_INJECT_SIZE - size_t *
+: Define the maximum message size that can be injected by the endpoint
+  in a single tagged message. The size is limited by the endpoint's configuration
+  and the provider's capabilities, and must be less than or equal to
+  `tx_attr->inject_size`.
+  Providers that don't support this option will return -FI_ENOPROTOOPT. In that
+  case, `tx_attr->inject_size` should be used.
+
+- *FI_OPT_RMA_INJECT_SIZE - size_t *
+: Define the maximum data size that can be injected by the endpoint
+  in a single RMA operation. The size is limited by the endpoint's configuration
+  and the provider's capabilities, and must be less than or equal to
+  `tx_attr->inject_size`.
+  Providers that don't support this option will return -FI_ENOPROTOOPT. In that
+  case, `tx_attr->inject_size` should be used.
+
+- *FI_OPT_ATOMIC_INJECT_SIZE - size_t *
+: Define the maximum data size that can be injected by the endpoint
+  in a single atomic operation. The size is limited by the endpoint's configuration
+  and the provider's capabilities, and must be less than or equal to
+  `tx_attr->inject_size`.
+  Providers that don't support this option will return -FI_ENOPROTOOPT. In that
+  case, `tx_attr->inject_size` should be used.
+
 ## fi_tc_dscp_set
 
 This call converts a DSCP defined value into a libfabric traffic class value.