README sample program compile error

[gwnet]

hello buddy,
===this part cannot compile well even after I put done into my class member. I am just using one simple example. I will paste you all the code.

volatile boolean done = false;

sqe.setNumberOfLogicalBlocks(1);
sqe.setNamespaceIdentifier(new NamespaceIdentifier(1));
volatile boolean done = false;
/* set callback */
command.setCallback(new OperationCallback() {
@OverRide
public void onStart() {
}

  @Override
  public void onComplete() {
      System.out.println("command completed");
      /* check response */
      NvmCompletionQueueEntry cqe = response.getResponseCapsule().getCompletionQueueEntry();
      StatusCode.Value statusCode = cqe.getStatusCode();
      if (!statusCode.equals(GenericStatusCode.getInstance().SUCCESS)) {
          System.out.println("Command not successful");
      }
      done = true;
  }

  @Override
  public void onFailure(RdmaException e) {
      System.out.println("There was an RDMA error when executing the command");
      done = true;
  }

});
command.execute(response);
/* wait until command is done */
while (!done);

[gwnet]

package com.ibm.jnvmf.utils;

import com.ibm.jnvmf.*;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeoutException;

public class NvmfClientHelloworld {

private volatile boolean done = false;
public static void main(String[] args)
        throws InterruptedException, ExecutionException, IOException, TimeoutException, RdmaException {

    try{
        String name = System.console().readLine("Name:");
        System.out.println(name);
    }catch(Exception ex)
    {
        ex.printStackTrace();
    }

    Nvme nvme = new Nvme();
    /* target ip and port we want to connect to */
    InetSocketAddress socketAddress = new InetSocketAddress("192.168.219.4", 4420);
    /* controller NVMe qualified name */
    NvmfTransportId tid = new NvmfTransportId(socketAddress,
            new NvmeQualifiedName("nqn.2019-12.com.wayne:test"));
    /* connect to controller */
    Controller controller = nvme.connect(tid);
    /* enable controller */
    controller.getControllerConfiguration().setEnable(true);
    controller.syncConfiguration();
    controller.waitUntilReady();
    /* create a io queue pair with submission queue size 64 */
    IoQueuePair queuePair = controller.createIoQueuePair(64);
    /* allocate and register buffer to be used for transfer */
    ByteBuffer buffer = ByteBuffer.allocateDirect(4096);
    KeyedNativeBuffer registeredBuffer = queuePair.registerMemory(buffer);
    /* create a new read command */
    NvmReadCommand command = new NvmReadCommand(queuePair);
    /* create a response */
    Response<NvmResponseCapsule> response = command.newResponse();
    /* set buffer */
    NvmIoCommandCapsule commandCapsule = command.getCommandCapsule();
    commandCapsule.setSglDescriptor(registeredBuffer);
    /* set length, LBA and namespace identifier */
    NvmIoCommandSqe sqe = commandCapsule.getSubmissionQueueEntry();
    sqe.setStartingLba(0);
    /* read one block*/
    sqe.setNumberOfLogicalBlocks(1);
    sqe.setNamespaceIdentifier(new NamespaceIdentifier(1));

    
    
    /* set callback */
    command.setCallback(new OperationCallback() {
        @Override
        public void onStart() {
        }

        @Override
        public void onComplete() {
            System.out.println("command completed");
            /* check response */
            NvmCompletionQueueEntry cqe = response.getResponseCapsule().getCompletionQueueEntry();
            StatusCode.Value statusCode = cqe.getStatusCode();
            if (!statusCode.equals(GenericStatusCode.getInstance().SUCCESS)) {
                System.out.println("Command not successful");
            }
            done = true;
        }

        @Override
        public void onFailure(RdmaException e) {
            System.out.println("There was an RDMA error when executing the command");
            done = true;
        }
    });
    command.execute(response);
    /* wait until command is done */
    while (!done);

}

}

[PepperJo]

Please let me know what the compilation error is, there might have been some small changes to the API that are not reflected in the example.

[gwnet]

man, I have fixed the compile error. now I suffer the null pointer error.
admin@ogden-dirt:/tmp/wayne/test1> java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8089 -cp target/jnvmf-1.7-jar-with-dependencies.jar:target/jnvmf-1.7-tests.jar -Djnvmf.legacy=true com.ibm.jnvmf.utils.NvmfClientHelloworld
Listening for transport dt_socket at address: 8089
Name:1
1
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.NullPointerException
at com.ibm.jnvmf.QueuePair.post(QueuePair.java:244)
at com.ibm.jnvmf.Command.execute(Command.java:79)
at com.ibm.jnvmf.utils.NvmfClientHelloworld.main(NvmfClientHelloworld.java:82)
admin@ogden-dirt:/tmp/wayne/test1>

[gwnet]

looks like we did not specify response callback, then it is null

final void post(Command command, SVCPostSend postSend, Response response) throws IOException {
command.getCallback().onStart();
response.getCallback().onStart(); ------ here crash
short commandId = nextCommandIdentifier();
command.setCommandId(commandId);
responseMap[commandId] = response;
commandMap[commandId] = command;
postSend.getWrMod(0).setWr_id(commandId);
postSend.execute();
}

[gwnet]

I work around the null pointer exception like this, the null pointer gone, but I never see the response is get back. the done flag is never set. Could you please help? I will paste my test code below.

final void post(Command command, SVCPostSend postSend, Response response) throws IOException {
command.getCallback().onStart();
OperationCallback callback = response.getCallback();
if (callback != null) {
response.getCallback().onStart();
}

short commandId = nextCommandIdentifier();
command.setCommandId(commandId);
responseMap[commandId] = response;
commandMap[commandId] = command;
postSend.getWrMod(0).setWr_id(commandId);
postSend.execute();

}

[gwnet]

my test code:
package com.ibm.jnvmf.utils;

import com.ibm.jnvmf.*;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeoutException;

public class NvmfClientHelloworld {

public static void main(String[] args)
        throws InterruptedException, ExecutionException, IOException, TimeoutException, RdmaException {

    try{
        String name = System.console().readLine("Name:");
        System.out.println(name);
    }catch(Exception ex)
    {
        ex.printStackTrace();
    }

    Nvme nvme = new Nvme();
    /* target ip and port we want to connect to */
    InetSocketAddress socketAddress = new InetSocketAddress("192.168.219.4", 4420);
    /* controller NVMe qualified name */
    NvmfTransportId tid = new NvmfTransportId(socketAddress,
            new NvmeQualifiedName("nqn.2019-12.com.wayne:test"));
    /* connect to controller */
    Controller controller = nvme.connect(tid);
    /* enable controller */
    controller.getControllerConfiguration().setEnable(true);
    controller.syncConfiguration();
    controller.waitUntilReady();
    /* create a io queue pair with submission queue size 64 */
    IoQueuePair queuePair = controller.createIoQueuePair(64);
    /* allocate and register buffer to be used for transfer */
    ByteBuffer buffer = ByteBuffer.allocateDirect(4096);
    KeyedNativeBuffer registeredBuffer = queuePair.registerMemory(buffer);
    /* create a new read command */
    NvmReadCommand command = new NvmReadCommand(queuePair);
    /* create a response */
    Response<NvmResponseCapsule> response = command.newResponse();
    /* set buffer */
    NvmIoCommandCapsule commandCapsule = command.getCommandCapsule();
    commandCapsule.setSglDescriptor(registeredBuffer);
    /* set length, LBA and namespace identifier */
    NvmIoCommandSqe sqe = commandCapsule.getSubmissionQueueEntry();
    sqe.setStartingLba(0);
    /* read one block*/
    sqe.setNumberOfLogicalBlocks(1);
    sqe.setNamespaceIdentifier(new NamespaceIdentifier(1));

    final boolean[] done = {false};

    /* set callback */
    command.setCallback(new OperationCallback() {
        @Override
        public void onStart() {
        }

        @Override
        public void onComplete() {
            System.out.println("command completed");
            /* check response */
            NvmCompletionQueueEntry cqe = response.getResponseCapsule().getCompletionQueueEntry();
            StatusCode.Value statusCode = cqe.getStatusCode();
            if (!statusCode.equals(GenericStatusCode.getInstance().SUCCESS)) {
                System.out.println("Command not successful");
            }
            done[0] = true;
        }

        @Override
        public void onFailure(RdmaException e) {
            System.out.println("There was an RDMA error when executing the command");
            done[0] = true;
        }
    });



    command.execute(response);
    /* wait until command is done */
    while (!done[0])
    {
        //System.out.println("not done yet");
    }
    response.getResponseCapsule();

    System.out.println("done ");

}

}

[PepperJo]

You need to also set the callback for the response. It intentionally does not check if it is set for performance reasons. (We are counting cycles here)

[PepperJo]

Just for your information. The code you pasted (resp. the example in the README) does not work as intended anymore. We now differentiate between command and response completion, i.e. when the command completes it just means the command was successfully send to the other side. For the whole operation to complete your response has to complete. So please move the callback code from the command to the response for it to work again.

[gwnet]

ok， I will try. but I found the command complete callback is not invoked at all. can you also try work out the simple sample and try from your side and share me too? I will try at my morning

[gwnet]

I double check benchmark sample, it also create callback for command.
and do we must call below poll to get command's response done?

476 do {
477 queuePair.poll();
478 } while ((commands.isEmpty() || responses.isEmpty()) && !rdmaException.isEmpty());
479 if (!rdmaException.isEmpty()) {
480 throw rdmaException.remove();
481 }

[gwnet]

I follow your suggestion. move the callback from request to the response. but the application still hanging there, no callback is invoked.

==below is application hang there===
admin@ogden-dirt:/tmp/wayne/test1> java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8089 -cp target/jnvmf-1.7-jar-with-dependencies.jar:target/jnvmf-1.7-tests.jar -Djnvmf.legacy=true com.ibm.jnvmf.utils.NvmfClientHelloworld
Listening for transport dt_socket at address: 8089
Name:1
1
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Listening for transport dt_socket at address: 8089

===below is code===
package com.ibm.jnvmf.utils;

import com.ibm.jnvmf.*;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeoutException;

public class NvmfClientHelloworld {

public static void main(String[] args)
        throws InterruptedException, ExecutionException, IOException, TimeoutException, RdmaException {

    try{
        String name = System.console().readLine("Name:");
        System.out.println(name);
    }catch(Exception ex)
    {
        ex.printStackTrace();
    }

    Nvme nvme = new Nvme();
    /* target ip and port we want to connect to */
    InetSocketAddress socketAddress = new InetSocketAddress("192.168.219.4", 4420);
    /* controller NVMe qualified name */
    NvmfTransportId tid = new NvmfTransportId(socketAddress,
            new NvmeQualifiedName("nqn.2019-12.com.wayne:test"));
    /* connect to controller */
    Controller controller = nvme.connect(tid);
    /* enable controller */
    controller.getControllerConfiguration().setEnable(true);
    controller.syncConfiguration();
    controller.waitUntilReady();
    /* create a io queue pair with submission queue size 64 */
    IoQueuePair queuePair = controller.createIoQueuePair(64);
    /* allocate and register buffer to be used for transfer */
    ByteBuffer buffer = ByteBuffer.allocateDirect(4096);
    KeyedNativeBuffer registeredBuffer = queuePair.registerMemory(buffer);
    /* create a new read command */
    NvmReadCommand command = new NvmReadCommand(queuePair);
    /* create a response */
    Response<NvmResponseCapsule> response = command.newResponse();
    /* set buffer */
    NvmIoCommandCapsule commandCapsule = command.getCommandCapsule();
    commandCapsule.setSglDescriptor(registeredBuffer);
    /* set length, LBA and namespace identifier */
    NvmIoCommandSqe sqe = commandCapsule.getSubmissionQueueEntry();
    sqe.setStartingLba(0);
    /* read one block*/
    sqe.setNumberOfLogicalBlocks(1);
    sqe.setNamespaceIdentifier(new NamespaceIdentifier(1));

    final boolean[] done = {false};

    /* set callback */
    response.setCallback(new OperationCallback() {
        @Override
        public void onStart() {
        }

        @Override
        public void onComplete() {
            System.out.println("command completed");
            /* check response */
            NvmCompletionQueueEntry cqe = response.getResponseCapsule().getCompletionQueueEntry();
            StatusCode.Value statusCode = cqe.getStatusCode();
            if (!statusCode.equals(GenericStatusCode.getInstance().SUCCESS)) {
                System.out.println("Command not successful");
            }
            done[0] = true;
        }

        @Override
        public void onFailure(RdmaException e) {
            System.out.println("There was an RDMA error when executing the command");
            done[0] = true;
        }
    });



    command.execute(response);
    /* wait until command is done */
    while (!done[0])
    {
        //System.out.println("not done yet");
    }
    response.getResponseCapsule();

    System.out.println("done ");

}

}

[gwnet]

hello man, I create another sample application based on your benchmark code. it works on the flow. I am not hanging any more. but I like to confirm the NVMe read do really read things back. but when I try to dump things in the bytebuffer after read response is done. I find the string are still all 0.

could you please help? maybe you can show me read and write works from your side. and share me your checking code.

package com.ibm.jnvmf.utils;

import com.ibm.jnvmf.*;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.util.ArrayDeque;
import java.util.Queue;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeoutException;

public class NvmfClientTest {
public static void main(String[] args)
throws InterruptedException, ExecutionException, IOException, TimeoutException, RdmaException {

    try {
        String name = System.console().readLine("Name:");
        System.out.println(name);
    } catch (Exception ex) {
        ex.printStackTrace();
    }


    Nvme nvme = new Nvme();
    /* target ip and port we want to connect to */
    InetSocketAddress socketAddress = new InetSocketAddress("192.168.219.4", 4420);
    /* controller NVMe qualified name */
    NvmfTransportId tid = new NvmfTransportId(socketAddress,
            new NvmeQualifiedName("nqn.2019-12.com.wayne:test"));
    /* connect to controller */
    Controller controller = nvme.connect(tid);
    /* enable controller */
    controller.getControllerConfiguration().setEnable(true);
    controller.syncConfiguration();
    controller.waitUntilReady();

    IdentifyControllerData identifyControllerData = controller.getIdentifyControllerData();
    System.out.println("Identify Controller Data: \n" +
            "PCI Vendor ID: " + identifyControllerData.getPciVendorId() + "\n" +
            "PCI Subsystem Vendor ID: " + identifyControllerData.getPciSubsystemVendorId() + "\n" +
            "Serial Number: " + identifyControllerData.getSerialNumber() + "\n" +
            "Model Number: " + identifyControllerData.getModelNumber() + "\n" +
            "Controller ID: " + identifyControllerData.getControllerId().toShort());
    System.out.println("--------------------------------------------------------");
    /* create a io queue pair with submission queue size 64 */
    IoQueuePair queuePair = controller.createIoQueuePair(64);

    //
    //prepare buffer
    //
    ByteBuffer buffer = ByteBuffer.allocateDirect(4096);
    KeyedNativeBuffer registeredBuffer = queuePair.registerMemory(buffer);
    byte bytes[] = new byte[registeredBuffer.capacity()];

    registeredBuffer.put(bytes);
    registeredBuffer.clear();

    CharBuffer bufferChar = buffer.asCharBuffer();


    StringBuilder sb = new StringBuilder();
    for(int w=0; w< bufferChar.length(); w++)
    {

        sb.append(bufferChar.get(w));
    }

    System.out.print(sb.toString());

    //
    //prepare cmd
    //
    final Queue<RdmaException> rdmaException = new ArrayDeque<>(1);
    final Queue<Command> commands = new ArrayDeque<>(1);
    final Queue<Response> responses = new ArrayDeque<>(1);

    NvmIoCommand<? extends NvmIoCommandCapsule> command;

    command = new NvmReadCommand(queuePair);

    //command.setSendInline(inline);
    NvmIoCommandCapsule commandCapsule = command.getCommandCapsule();

    commandCapsule.setSglDescriptor(registeredBuffer);

    command.setCallback(new OperationCallback() {
        @Override
        public void onStart() {
        }

        @Override
        public void onComplete() {
            commands.add(command);
        }

        @Override
        public void onFailure(RdmaException e) {
            rdmaException.add(e);
        }
    });


    commands.add(command);
    Response<NvmResponseCapsule> response = command.newResponse();
    response.setCallback(new OperationCallback() {
        long startTime;

        @Override
        public void onStart() {

        }

        @Override
        public void onComplete() {

            responses.add(response);//wgao, why add response twice ??
        }

        @Override
        public void onFailure(RdmaException e) {
            rdmaException.add(e);
        }
    });
    //responses.add(response);

    NvmIoCommand<?> command1 = (NvmIoCommand) commands.remove();
    NvmIoCommandCapsule commandCapsule1 = command1.getCommandCapsule();
    NvmIoCommandSqe sqe = commandCapsule1.getSubmissionQueueEntry();
    sqe.setStartingLba(0);
    sqe.setNumberOfLogicalBlocks(8);
    sqe.setNamespaceIdentifier(new NamespaceIdentifier(1));
    command1.execute(response);


    while(responses.isEmpty())
    {
        queuePair.poll();
    }
    //do {
    //    queuePair.poll();




   // } while ((commands.isEmpty() || responses.isEmpty()) && !rdmaException.isEmpty());
    //if (!rdmaException.isEmpty()) {
    //    throw rdmaException.remove();
    //}

    Response<NvmResponseCapsule> response1 = responses.remove();
    NvmCompletionQueueEntry cqe = response1.getResponseCapsule().getCompletionQueueEntry();
    StatusCode.Value statusCode = cqe.getStatusCode();
    if (statusCode != null) {
        if (!statusCode.equals(GenericStatusCode.getInstance().SUCCESS)) {
            throw new UnsuccessfulComandException(cqe);
        } else {

            response1.getResponseCapsule();
            System.out.println("good done ");

            CharBuffer bufferChar1 = buffer.asCharBuffer();

            StringBuilder sb1 = new StringBuilder();
            for (int w = 0; w < bufferChar1.length(); w++) {

                sb1.append(bufferChar.get(w));
            }

            System.out.print(sb1.toString());
        }
    }


}

}

[PepperJo]

I recommend writing and then reading something. If the device has been e.g. trimmed then all the data will be zero so you would not see anything. Alternatively write some random data to the buffer before doing the read such that you know it would have been overwritten with zeros.

[gwnet]

I am pretty sure there is no trim on my disk. I will try write random data to buffer before doing read.
my disk is currently tested on other purpose. cannot write. I am reading the 0 LBA, it should be partition table, should not be all 0. I can use dd to verify that. could you please provide me one sample of read and dump?