Clarifying FI_MR_ENDPOINT behavior

[iziemba]

It seems FI_MR_ENDPOINT is lacking some documentation which is causing confusion on how exactly it should function. The following is the current documentation for FI_MR_ENDPOINT.

This mode bit indicates that the provider associates memory regions with endpoints rather than domains. Memory regions that are registered with the provider are created in a disabled state and must be bound to an endpoint prior to being enabled. To bind the MR with an endpoint, the application must use fi_mr_bind(). To enable the memory region, the application must call fi_mr_enable().

The following are some questions which were raised.

• Since the MR is associated with the endpoint, where does libfabric user define RKEY enforcement take place with FI_MR_ENDPOINT (i.e. does fi_mr_reg() return -FI_EKEYREJECTED or does fi_mr_bind()/fi_mr_enable() return -FI_EKEYREJECTED)?
• Can a single MR be bound to multiple EPs?
• Where does MR binding occur with SEPs (i.e on the SEP or the child TX/RX contexts)?

Through email discussion, the following was the suggested answers to the above questions.

• User requested MR keys are enforced by the domain object.
• MRs can be bound to SEPs. All child TX and RX contexts will inherit the MR. If there are zero RX contexts created, the MR will not be remotely targetable.
• The same MR can be bound to multiple EPs and SEPs before the MR is enabled. Once the MR is enabled, it can no longer be bound to addition EPs and SEPs.

Note: If there is a use case, it may be possible to allow MRs to be bound to specific TX and RX contexts instead of all TX and RX contexts inheriting the MR when bound to an SEP.

Thoughts on this?

[iziemba]

I just got a couple use cases requiring the following behaviors for FI_MR_ENDPOINT.

• When the MR is bound to the SEP, the number of TX/RX contexts are not yet known. The TX/RX contexts are dynamic. Thus, even after the MR is enabled, the request is to allow for TX/RX contexts to inherit the MR.
• Allow MRs to be bound to specific TX and RX contexts.

@shefty : Thoughts on this?

[shefty]

The goal is to minimize the pain being inflicted on the users, because that ultimately will produce the most efficient/maintainable code.

Based on a straightforward reading of the API, a context is not considered an endpoint. Endpoints are allocated by fi_endpoint, fi_passive_ep, and fi_scalable_ep calls. Contexts reuse the same SW objects out of convenience, but are usually referred to as queues. MR should be considered as accessed by endpoints, not contexts. I think this distinction becomes clearer if you think in terms of endpoints with shared contexts.

A MR has a single rkey, obtainable through fi_mr_key(). Portable applications will maintain unique rkeys per domain. Technically a provider isn't strictly bound by this if they require FI_MR_ENDPOINT. Two MRs could have the same rkey as long as they do not share any endpoints. Cheers to the developers who wants to pursue that path. From the viewpoint of a user, an rkey identifies a unique remote memory region.

The number of Rx/Tx contexts, including whether a shared context is used, are associated with an SEP at creation time. Context information is given as input into the fi_scalable_ep() call. It's anticipated, but not required, that all HW contexts needed by an SEP will be reserved at creation. The allocation of the HW contexts may be separate from the user allocating the SW objects used to access them.

The ability to restrict a MR to a specific Tx or Rx context is not supported by the API. Such a feature should be treated independently from whether a provider requires FI_MR_ENDPOINT. To date, there are no application requirements for this level of control, likely because most apps would rather not deal with memory registration at all, let alone defining a feature to make it more difficult to use.

Related to all of this, if a provider requires FI_MR_ENDPOINT and a MR will only be used with a single EP, then the EP could be specified as part of the fi_mr_regattr() call. That would allow combining registration, binding the MR to the EP, and enabling the MR under the same call.

[iziemba]

Sorry for the delayed responses (been OOO).

I would like to walk through a couple of examples just to confirm how this would behavior to an application.

Example 1: Threaded application using RMA and a standard EP per thread. The underlying provider only requires FI_MR_ENDPOINT. The following is how the application would register its MRs during init.

/* Assume domain is allocated per process, an EP per thread, and each EP has its own unique CQ. */
struct fid_domain *dom;
struct fi_ep *eps[];

uint64_t global_mr_rkey = 0x1234;
struct fid_mr *mr;
int ret;
void *buf;
size_t buf_size;

ret = fi_mr_reg(dom, buf, buf_size, FI_REMOTE_READ | FI_REMOTE_WRITE | FI_READ | FI_WRITE, 0, global_mr_rkey, 0, *mr, NULL);
assert(ret == FI_SUCCESS);

for (i = 0; i < NUM_THREADS; i++) {
    ret = fi_mr_bind(mr, eps[i], 0);
    assert(ret == FI_SUCCESS);
}

ret = fi_mr_enable(mr);
assert(ret == FI_SUCCESS);

The end result would be an MR with user-defined rkey 0x1234 being accessible from all endpoints correct?

Example 2: Threaded application using RMA and an SEP per process with a unique TX and RX context per thread. The underlying provider only requires FI_MR_ENDPOINT. The following is how the application would register its MRs during init.

/* Assume domain is allocated per process, an SEP per process, a TX/RX context per thread, and each TX/RX context pair has its own unique CQ. */
struct fid_domain *dom;
struct fi_ep *seps;
struct fi_ep *txcs[];
struct fid_ep *rxcs[];

uint64_t global_mr_rkey = 0x1234;
struct fid_mr *mr;
int ret;
void *buf;
size_t buf_size;

ret = fi_mr_reg(dom, buf, buf_size, FI_REMOTE_READ | FI_REMOTE_WRITE | FI_READ | FI_WRITE, 0, global_mr_rkey, 0, *mr, NULL);
assert(ret == FI_SUCCESS);

ret = fi_mr_bind(mr, seps, 0);
assert(ret == FI_SUCCESS);

ret = fi_mr_enable(mr);
assert(ret == FI_SUCCESS);

The end result is since the MR was bound to the SEP, all child TX/RX contexts will inherit this MR. Meaning, I should be able to do an fi_writedata() operation to any RXC and the completion event should appear on the CQ associated with the target RXC?

Example 3: Threaded application using RMA and an SEP per process but TX and RX contexts are allocated on demand. The underlying provider only requires FI_MR_ENDPOINT. The following is how the application would register its MRs during init.

/* Assume domain is allocated per process, an SEP per process. */
struct fid_domain *dom;
struct fi_ep *seps;
struct fi_ep *txcs[];
struct fid_ep *rxcs[];
int context_index = 0;

uint64_t global_mr_rkey = 0x1234;
struct fid_mr *mr;
int ret;
void *buf;
size_t buf_size;

ret = fi_mr_reg(dom, buf, buf_size, FI_REMOTE_READ | FI_REMOTE_WRITE | FI_READ | FI_WRITE, 0, global_mr_rkey, 0, *mr, NULL);
assert(ret == FI_SUCCESS);

ret = fi_mr_bind(mr, seps, 0);
assert(ret == FI_SUCCESS);

ret = fi_mr_enable(mr);
assert(ret == FI_SUCCESS);

Then during runtime, TX/RX contexts are allocated.

ret = fi_tx_context(sep, context_index, NULL, &txcs[context_index], NULL);
assert(ret == FI_SUCCESS);

ret = fi_rx_context(sep, context_index, NULL, &rxcs[context_index], NULL);
assert(ret == FI_SUCCESS);

context_index++;

Since the MR was bound to the SEP, all the TX/RX contexts will inherit the MR. This may result in the provider having to allocate addition hardware resources during TX/RX context creation to support the MR. Thus, this can result in fi_tx_context and/or fi_rx_context failing due to an MR resource allocation failure, correct?

[shefty]

If I'm following the example, the answer is yes, depending on how the provide implements everything. A provider may allocate all underlying Tx/Rx resources when an SEP is created, or it could allocate resources on-demand later. On-demand allocations could always fail later, independent of MRs.