please help me on where jNVMf specify below parameters like SPDK

[gwnet]

hello buddy,

SPDK support couples of NVMe request type. how about jNVMf, can you point to me the code?
kernel target server will have different process for different cmd parameter.

req->cmd.psdt = SPDK_NVME_PSDT_SGL_MPTR_CONTIG;
req->cmd.dptr.sgl1.keyed.type = SPDK_NVME_SGL_TYPE_KEYED_DATA_BLOCK;
req->cmd.dptr.sgl1.keyed.subtype = SPDK_NVME_SGL_SUBTYPE_ADDRESS;
req->cmd.dptr.sgl1.keyed.length = req->payload_size;
req->cmd.dptr.sgl1.address = (uint64_t)payload;

static int
nvme_rdma_build_contig_request(struct nvme_rdma_qpair *rqpair,
struct spdk_nvme_rdma_req *rdma_req)
{
struct nvme_request *req = rdma_req->req;
void *payload = req->payload.contig_or_cb_arg + req->payload_offset;
struct ibv_mr *mr;
uint64_t requested_size;

assert(req->payload_size != 0);
assert(nvme_payload_type(&req->payload) == NVME_PAYLOAD_TYPE_CONTIG);

requested_size = req->payload_size;
if (!g_nvme_hooks.get_rkey) {

	mr = (struct ibv_mr *)spdk_mem_map_translate(rqpair->mr_map->map, (uint64_t)payload,
			&requested_size);
	if (mr == NULL) {
		return -1;
	}
	req->cmd.dptr.sgl1.keyed.key = mr->rkey;
} else {
	req->cmd.dptr.sgl1.keyed.key = spdk_mem_map_translate(rqpair->mr_map->map,
				       (uint64_t)payload,
				       &requested_size);
}

if (requested_size < req->payload_size) {
	SPDK_ERRLOG("Data buffer split over multiple RDMA Memory Regions\n");
	return -1;
}

/* The first element of this SGL is pointing at an
 * spdk_nvmf_cmd object. For this particular command,
 * we only need the first 64 bytes corresponding to
 * the NVMe command. */
rdma_req->send_sgl[0].length = sizeof(struct spdk_nvme_cmd);

/* The RDMA SGL needs one element describing the NVMe command. */
rdma_req->send_wr.num_sge = 1;

req->cmd.psdt = SPDK_NVME_PSDT_SGL_MPTR_CONTIG;
req->cmd.dptr.sgl1.keyed.type = SPDK_NVME_SGL_TYPE_KEYED_DATA_BLOCK;
req->cmd.dptr.sgl1.keyed.subtype = SPDK_NVME_SGL_SUBTYPE_ADDRESS;
req->cmd.dptr.sgl1.keyed.length = req->payload_size;
req->cmd.dptr.sgl1.address = (uint64_t)payload;

return 0;

}

[gwnet]

and also from nvme spec PSDT need physical continuous, SPDK use huge page then each 2MB is continous, but jNVMf, have to use 4K continuous?
any insight to share me?

PRP or SGL for Data Transfer (PSDT): This field specifies whether PRPs or SGLs are used for any datatransfer associated with the command. PRPs shall be used for all Admin commands for NVMe over PCIe.SGLs shall be used for all Admin and I/O commands for NVMe over Fabrics. This field shall be set to 01b forNVMe over Fabrics 1.0 implementations. The definition is described in the table below.Value Definition00b PRPs are used for this transfer.01bSGLs are used for this transfer. If used, Metadata Pointer(MPTR) contains an address of a single contiguous physicalbuffer that is byte aligned.10bSGLs are used for this transfer. If used, Metadata Pointer(MPTR) contains an address of an SGL segment containingexactly one SGL Descriptor that is Qword aligned.11b ReservedIf there is metadata that is not interleaved with the logical block data, as specified in the Format NVMcommand, then the Metadata Pointer (MPTR) field is used to point to the metadata. The definition of theMetadata Pointer field is dependent on the setting in this field. Refer to Figure 11.	Value	Definition	00b	PRPs are used for this transfer.	01b	SGLs are used for this transfer. If used, Metadata Pointer(MPTR) contains an address of a single contiguous physicalbuffer that is byte aligned.	10b	SGLs are used for this transfer. If used, Metadata Pointer(MPTR) contains an address of an SGL segment containingexactly one SGL Descriptor that is Qword aligned.	11b	Reserved
Value	Definition
00b	PRPs are used for this transfer.
01b	SGLs are used for this transfer. If used, Metadata Pointer(MPTR) contains an address of a single contiguous physicalbuffer that is byte aligned.
10b	SGLs are used for this transfer. If used, Metadata Pointer(MPTR) contains an address of an SGL segment containingexactly one SGL Descriptor that is Qword aligned.
11b	Reserved

[PepperJo]

Please take a look at the NVMe over Fabrics specification. Not all types are supported by NVMf. PRPs are NOT supported via NVMf.

[gwnet]

nvmeof spec does not have it. this is defined in nvme spec. from the spec, it mention it need below payload is physical continuous. how about your jNVMf? SPDK use huge page, each page has 2MB. but jNVMf make sure underlying is physical continous?
req->cmd.dptr.sgl1.keyed.length = req->payload_size;
req->cmd.dptr.sgl1.address = (uint64_t)payload;

[PepperJo]

I can only urge you to read the NVMf specification. It is clearly layed out there how memory is handled. Certainly there is no need to make the underlying memory physical continuous since this is handled by the RDMA NIC.