about Memory growth test of Tritonserver #1141

jackyh · 2022-01-28T04:08:31Z

We try to test the memory growth by gather the stats of memory usage when doing inference.
each time when we do an inference, we will get the statistics of memory it allocated. we found that:
("The max allocation of Memory when doing a single inference" - "The average allocation of Memory when doing a single inference") / ("The max allocation of Memory when doing a single inference") = 0.46, which means the variation is too big, why? it varies from about 700MB to 1500MB.

attached is the simple.java file and the test.sh script, probably to reproduce this, one need to modify the dir in the test.sh accordingly.

@saudet

L0_memory_growth.zip

saudet · 2022-01-28T05:17:47Z

There's a couple of things that could be happening, but the first thing you should check is for dangling Pointer objects. Try to run a command like this:

mvn clean compile exec:java -Dorg.bytedeco.javacpp.logger.debug -DargLine=-Xmx1000m 2>&1 | grep Collecting | grep -v 'ownerAddress=0x0'

If you see any output from that, you should find where those Pointer objects are not getting deallocated and call close() on them, or you could use PointerScope where appropriate: http://bytedeco.org/news/2018/07/17/bytedeco-as-distribution/

jackyh · 2022-01-28T14:30:54Z

looks like there's quite a few there. Since there's lots of "new BytePointer" in Simple.java, which ones need me to do a close() on it?
client_10.log

saudet · 2022-01-28T14:44:05Z

Just try to use PointerScope...

jackyh · 2022-01-28T14:52:44Z

it will help to release/close by self?

saudet · 2022-01-28T14:58:29Z

Kind of, it's like a scope in C++, see the example here: http://bytedeco.org/news/2018/07/17/bytedeco-as-distribution/

jackyh · 2022-01-29T10:13:19Z

Kind of, it's like a scope in C++, see the example here: http://bytedeco.org/news/2018/07/17/bytedeco-as-distribution/

I did some test today with "PointerScope" as attached. But memory usage is still dangled, vibrated:
("The max allocation of Memory when doing a single inference" - "The average allocation of Memory when doing a single inference") / ("The max allocation of Memory when doing a single inference") = 0.52, which means the variation is still too big. it varies from about 50MB to 500MB. Do we have more ways to debug?
20220129_PointerScope.zip

saudet · 2022-01-29T11:10:28Z

Please check the debug log like I asked you to do above #1141 (comment)

jackyh · 2022-01-29T11:48:38Z

Please check the debug log like I asked you to do above #1141 (comment)

looks you mean even if "PointerScope" added, there still will be more leaks of pointers, if "PointerScope" does not include all the pointers?

jackyh · 2022-01-29T15:08:16Z

I added "try (PointerScope scope = new PointerScope()) {" just at the beginning of the Main function, why there's still lots of "Debug: Collecting org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x0,deallocatorAddress=0x0]" in the output log file?

saudet · 2022-01-30T00:36:13Z

Those are fine, their ownerAddress is 0. If you see any that have an address other than 0, then you should find what those are. If all that you see do not have an address, then you're probably dealing with GC issues of the Java heap. Try a different one:
https://developers.redhat.com/articles/2021/11/02/how-choose-best-java-garbage-collector

saudet · 2022-01-30T04:06:09Z

BTW, how did you make sure this is happening only with Java, and not with C++? Maybe it's a problem with Triton...

jackyh · 2022-01-30T13:04:10Z

BTW, how did you make sure this is happening only with Java, and not with C++? Maybe it's a problem with Triton...

good point.

jackyh · 2022-01-30T13:12:39Z

searched the log file:
Debug: Releasing org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x7f29bf518190,deallocatorAddress=0x7f29c7ec4090]
Debug: Collecting org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x0,deallocatorAddress=0x0]
all the log of Collecting are ownerAddress=0x0

jackyh · 2022-02-03T13:21:23Z

Samuel:
We designed our test case like this:

we will start a thread to monitor the memory usage:
a, this thread will monitor the usage of memory every two seconds
b, each time, we will use this to do statistics:
DoubleSummaryStatistics stats = new DoubleSummaryStatistics();
double memory = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
stats.accept(memory);
c, each time, we will use this to calculate the delta:
double memory_allocation_delta = stats.getMax() - stats.getAverage();
double memory_allocation_delta_mb = memory_allocation_delta / 1E6;
double memory_allocation_delta_percent = memory_allocation_delta / stats.getMax();
here, if the "memory_allocation_delta_percent" is larger than 10 percent, then the test will fail.
for the main process, we will do this:
for(int i = 0; i < 1000000; i++){
RunInference(server, model_name, is_int, is_torch_model);
}

We assume, every two seconds, there will be some member of function "RunInference" be processed, some memory will be allocated, some memory will be freed during each call of "RunInference", so the variation of "memory_allocation_delta_percent" should not be larger than 10%, how do you think? This is the right way to test memory growth of Java process?

@saudet

saudet · 2022-02-03T21:30:03Z

Well, that's a question about Triton more than anything else, I think. All buffers should be preallocated as much as possible, so variations like that don't occur.

jackyh · 2022-02-04T13:15:00Z

Well, that's a question about Triton more than anything else, I think. All buffers should be preallocated as much as possible, so variations like that don't occur.

since, each time when to do the reference, in function of "RunInference", we will allocator lots of buffers, do you mean this needs to be replaced by some static/preallocated memory? If we re-allocate these buffers each time when doing inference, this variation is common to Java process?

saudet · 2022-02-04T13:39:44Z

That has nothing to do with Java! You're allocating these buffers for Triton, not Java. This is something that needs to be fixed for Triton.

jackyh · 2022-02-04T13:56:57Z

buffers for Triton, not Java. This is something that needs to be fixed for Triton.

Yes, we allocate these buffers for Triton to do inference or compare some result. So, let's say, if I allocate these buffers as some static/preallocated ones, then the variation issue is fixed, does that mean the GC is not working well enough?

saudet · 2022-02-04T22:24:01Z

Preallocating and reusing objects that use memory on the Java heap helps the GC, but it's possible to tune the GC to be able to cope better with larger amounts of garbage too, yes.

jackyh · 2022-02-05T13:03:51Z

Preallocating and reusing objects that use memory on the Java heap helps the GC, but it's possible to tune the GC to be able to cope better with larger amounts of garbage too, yes.

so you mean the ways listed https://developers.redhat.com/articles/2021/11/02/how-choose-best-java-garbage-collector#parallel_collector here to tune the GC for larger amounts of garbage?

saudet · 2022-02-05T14:08:10Z

That kind thing, yes, but if the requests that you get don't require allocating different kinds of buffers all the time, it's more efficient to just reuse those buffers. That's probably what your users are asking about.

jackyh · 2022-02-06T10:09:43Z

here's the default JVM parameters:

root@4a42d065cf6e:/workspace/javacpp_presets_upstream/javacpp-presets/tritonserver# java -XX:+PrintCommandLineFlags -version
-XX:G1ConcRefinementThreads=10 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=524877248 -XX:MaxHeapSize=8398035968 -XX:+PrintCommandLineFlags -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment (build 11.0.13+8-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.13+8-Ubuntu-0ubuntu1.20.04, mixed mode, sharing)

Which parameters you think that probably need to tune?

jackyh · 2022-02-06T10:35:11Z

That kind thing, yes, but if the requests that you get don't require allocating different kinds of buffers all the time, it's more efficient to just reuse those buffers. That's probably what your users are asking about.

so, for this, since I want to make the largest allocated memory as static ones, how can I know which buffers/object is the largest one?

saudet · 2022-02-06T11:41:30Z

Not just the largest one, all of them, if possible. I'm guessing that ideally your users want this to be "garbage free" to get the lowest latency possible, for real time applications, but I'm just guessing. You should try to find out what the needs of your users are, and then we can figure out how to meet those needs.

jackyh · 2022-02-06T13:13:26Z

Not just the largest one, all of them, if possible. I'm guessing that ideally your users want this to be "garbage free" to get the lowest latency possible, for real time applications, but I'm just guessing. You should try to find out what the needs of your users are, and then we can figure out how to meet those needs.

temporary, this test is just internally, probably users will have such sort of requirements? I'm not sure what JAVA users will most care about

saudet · 2022-02-07T00:05:33Z

temporary, this test is just internally, probably users will have such sort of requirements? I'm not sure what JAVA users will most care about

Well, if what you care most about is money, HFT is where it's at for low-latency Java applications:
https://www.efinancialcareers.com/news/2020/11/low-latency-java-trading-systems
https://medium.com/@jadsarmo/why-we-chose-java-for-our-high-frequency-trading-application-600f7c04da94
https://www.azul.com/use-cases/trading-risk/
https://github.com/OpenHFT

But personally I prefer working on embedded systems such as the ones from aicas:
https://www.aicas.com/wp/use-cases/
@jjh-aicas Do you have use cases where machine learning and GPUs could be of help?

jackyh · 2022-02-07T13:57:37Z

Samuel:

Today I did more tests on GC and Heap:

Command line arg is: -DargLine=-Xmx1000m
While the test is running, Memory allocated (heap) will grow gradually from about 60M to 4000M (Here: Memory allocated is calculated by: Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(). Detail is attached as client.log
info of gc is collected by cmd: jstat -gc 10524 500. Looks like "OU" grows fast! Detail is attached as gc.log

Why "OU" grows fast here? @saudet
client.log
gc.log

saudet · 2022-02-07T14:51:31Z

That's the "old space" apparently: https://docs.oracle.com/javase/7/docs/technotes/tools/share/jstat.html
Here's some doc about that: https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/garbage_collect.html
So it just looks like there are buffers that can't be freed because they are still referenced somewhere.

jackyh · 2022-04-07T13:52:15Z

Actually, you made me think about something. It is possible to store a reference to any Java object in a Pointer using JNI. I've added a couple of helper methods for that in commit bytedeco/javacpp@f11d67d. With that, instead of allocating and deallocating new native objects, something like this should work (not tested):

diff --git a/tritonserver/samples/Simple.java b/tritonserver/samples/Simple.java
index 8fa4d4f339..fd5271f8b7 100644
--- a/tritonserver/samples/Simple.java
+++ b/tritonserver/samples/Simple.java
@@ -197,7 +197,7 @@ public class Simple {
             // releasing the buffer.
             if (!allocated_ptr.isNull()) {
               buffer.put(0, allocated_ptr);
-              buffer_userp.put(0, new BytePointer(tensor_name));
+              buffer_userp.put(0, Loader.newGlobalRef(tensor_name));
               System.out.println("allocated " + byte_size + " bytes in "
                                + TRITONSERVER_MemoryTypeString(actual_memory_type.get())
                                + " for result tensor " + tensor_name);
@@ -213,16 +213,16 @@ public class Simple {
             TRITONSERVER_ResponseAllocator allocator, Pointer buffer, Pointer buffer_userp,
             long byte_size, int memory_type, long memory_type_id)
         {
-          BytePointer name = null;
+          String name = null;
           if (buffer_userp != null) {
-            name = new BytePointer(buffer_userp);
+            name = (String)Loader.accessGlobalRef(buffer_userp);
           } else {
-            name = new BytePointer("<unknown>");
+            name = "<unknown>";
           }
 
           System.out.println("Releasing buffer " + buffer + " of size " + byte_size
                            + " in " + TRITONSERVER_MemoryTypeString(memory_type)
-                           + " for result '" + name.getString() + "'");
+                           + " for result '" + name + "'");
           switch (memory_type) {
             case TRITONSERVER_MEMORY_CPU:
               Pointer.free(buffer);
@@ -254,7 +254,7 @@ public class Simple {
               break;
           }
 
-          name.deallocate();
+          Loader.deleteGlobalRef(buffer_userp);
 
           return null;  // Success
         }

I will try this now! Today I did something like:

if (!allocated_ptr.isNull()) {
buffer.put(0, allocated_ptr);
//jack added 0407
tensor_name_ptr = Pointer.malloc(tensor_name.length());
buffer_userp.put(0, tensor_name_ptr);
//System.out.println("string size of tensor_name" + ": " + tensor_name.length() + " is newed.");
//Pointer.free(allocated_ptr);
Pointer.free(tensor_name_ptr);

JMC says "no memory leak", but still can only run 5600000 iterations, then "unknown exception"...

jackyh · 2022-04-07T14:35:09Z

Actually, you made me think about something. It is possible to store a reference to any Java object in a Pointer using JNI. I've added a couple of helper methods for that in commit bytedeco/javacpp@f11d67d. With that, instead of allocating and deallocating new native objects, something like this should work (not tested):

diff --git a/tritonserver/samples/Simple.java b/tritonserver/samples/Simple.java
index 8fa4d4f339..fd5271f8b7 100644
--- a/tritonserver/samples/Simple.java
+++ b/tritonserver/samples/Simple.java
@@ -197,7 +197,7 @@ public class Simple {
             // releasing the buffer.
             if (!allocated_ptr.isNull()) {
               buffer.put(0, allocated_ptr);
-              buffer_userp.put(0, new BytePointer(tensor_name));
+              buffer_userp.put(0, Loader.newGlobalRef(tensor_name));
               System.out.println("allocated " + byte_size + " bytes in "
                                + TRITONSERVER_MemoryTypeString(actual_memory_type.get())
                                + " for result tensor " + tensor_name);
@@ -213,16 +213,16 @@ public class Simple {
             TRITONSERVER_ResponseAllocator allocator, Pointer buffer, Pointer buffer_userp,
             long byte_size, int memory_type, long memory_type_id)
         {
-          BytePointer name = null;
+          String name = null;
           if (buffer_userp != null) {
-            name = new BytePointer(buffer_userp);
+            name = (String)Loader.accessGlobalRef(buffer_userp);
           } else {
-            name = new BytePointer("<unknown>");
+            name = "<unknown>";
           }
 
           System.out.println("Releasing buffer " + buffer + " of size " + byte_size
                            + " in " + TRITONSERVER_MemoryTypeString(memory_type)
-                           + " for result '" + name.getString() + "'");
+                           + " for result '" + name + "'");
           switch (memory_type) {
             case TRITONSERVER_MEMORY_CPU:
               Pointer.free(buffer);
@@ -254,7 +254,7 @@ public class Simple {
               break;
           }
 
-          name.deallocate();
+          Loader.deleteGlobalRef(buffer_userp);
 
           return null;  // Success
         }

I will try this now! Today I did something like:

if (!allocated_ptr.isNull()) { buffer.put(0, allocated_ptr); //jack added 0407 tensor_name_ptr = Pointer.malloc(tensor_name.length()); buffer_userp.put(0, tensor_name_ptr); //System.out.println("string size of tensor_name" + ": " + tensor_name.length() + " is newed."); //Pointer.free(allocated_ptr); Pointer.free(tensor_name_ptr);

JMC says "no memory leak", but still can only run 5600000 iterations, then "unknown exception"...

There's no 1.5.8-snapshots including the latest JavaCPP? how to make pom.xml to include the latest one?

org.bytedeco javacpp-presets 1.5.8-SNAPSHOT

org.bytedeco
tritonserver
2.18-${project.parent.version}
JavaCPP Presets for Triton Inference Server

org.bytedeco javacpp

the above doesn't work, not including the latest code of javapp.

saudet · 2022-04-07T14:37:49Z

You can add an entry like this to override the version from the dependencies:

  <dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>javacpp</artifactId>
    <version>1.5.8-SNAPSHOT</version>
  </dependency>

jackyh · 2022-04-07T14:52:53Z

You can add an entry like this to override the version from the dependencies:
  <dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>javacpp</artifactId>
    <version>1.5.8-SNAPSHOT</version>
  </dependency>

thanks, do it now.

You can add an entry like this to override the version from the dependencies:
  <dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>javacpp</artifactId>
    <version>1.5.8-SNAPSHOT</version>
  </dependency>

still not works:

[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /workspace/forMemoryLeakIssue/jackyh0407/java_cpp_presets_0407/javacpp-presets/tritonserver/samples/Simple
.java:[201,41] cannot find symbol
symbol: method newGlobalRef(java.lang.String)
location: class org.bytedeco.javacpp.Loader
[ERROR] /workspace/forMemoryLeakIssue/jackyh0407/java_cpp_presets_0407/javacpp-presets/tritonserver/samples/Simple.java:[220,34] cannot find symbol
symbol: method accessGlobalRef(org.bytedeco.javacpp.Pointer)
location: class org.bytedeco.javacpp.Loader
[ERROR] /workspace/forMemoryLeakIssue/jackyh0407/java_cpp_presets_0407/javacpp-presets/tritonserver/samples/Simple.java:[258,17] cannot find symbol
symbol: method deleteGlobalRef(org.bytedeco.javacpp.Pointer)
location: class org.bytedeco.javacpp.Loader

even when I delete this:
rm -rf /root/.m2/repository/org/bytedeco/javacpp/1.5.8-SNAPSHOT/

jackyh · 2022-04-07T14:54:47Z

You can add an entry like this to override the version from the dependencies:
  <dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>javacpp</artifactId>
    <version>1.5.8-SNAPSHOT</version>
  </dependency>
thanks, do it now.
You can add an entry like this to override the version from the dependencies:
  <dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>javacpp</artifactId>
    <version>1.5.8-SNAPSHOT</version>
  </dependency>
still not works:

[ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /workspace/forMemoryLeakIssue/jackyh0407/java_cpp_presets_0407/javacpp-presets/tritonserver/samples/Simple .java:[201,41] cannot find symbol symbol: method newGlobalRef(java.lang.String) location: class org.bytedeco.javacpp.Loader [ERROR] /workspace/forMemoryLeakIssue/jackyh0407/java_cpp_presets_0407/javacpp-presets/tritonserver/samples/Simple.java:[220,34] cannot find symbol symbol: method accessGlobalRef(org.bytedeco.javacpp.Pointer) location: class org.bytedeco.javacpp.Loader [ERROR] /workspace/forMemoryLeakIssue/jackyh0407/java_cpp_presets_0407/javacpp-presets/tritonserver/samples/Simple.java:[258,17] cannot find symbol symbol: method deleteGlobalRef(org.bytedeco.javacpp.Pointer) location: class org.bytedeco.javacpp.Loader

even when I delete this: rm -rf /root/.m2/repository/org/bytedeco/javacpp/1.5.8-SNAPSHOT/

I modified javacpp-presets/tritonserver/pom.xml as you wrote above. not works.

saudet · 2022-04-07T15:21:39Z

Maybe something went wrong with the latest builds. I've restarted the builds, please try again with mvn -U ...

jackyh · 2022-04-08T00:04:26Z

Maybe something went wrong with the latest builds. I've restarted the builds, please try again with mvn -U ...

OK, will do just after breakfast. I need to try my best.

jackyh · 2022-04-08T00:20:52Z

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project simple: Compilation failure: Compilation failure:
[ERROR] /workspace/forMemoryLeakIssue/jackyh0407/java_cpp_presets_0407/javacpp-presets/tritonserver/samples/Simple.java:[201,41] cannot find symbol
[ERROR] symbol: method newGlobalRef(java.lang.String)
[ERROR] location: class org.bytedeco.javacpp.Loader
[ERROR] /workspace/forMemoryLeakIssue/jackyh0407/java_cpp_presets_0407/javacpp-presets/tritonserver/samples/Simple.java:[220,34] cannot find symbol
[ERROR] symbol: method accessGlobalRef(org.bytedeco.javacpp.Pointer)
[ERROR] location: class org.bytedeco.javacpp.Loader
[ERROR] /workspace/forMemoryLeakIssue/jackyh0407/java_cpp_presets_0407/javacpp-presets/tritonserver/samples/Simple.java:[258,17] cannot find symbol
[ERROR] symbol: method deleteGlobalRef(org.bytedeco.javacpp.Pointer)
[ERROR] location: class org.bytedeco.javacpp.Loader
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

not works. I have to try a new clean container.

…llback (issue bytedeco/javacpp-presets#1141)

saudet · 2022-04-08T01:55:16Z

Ok, I finally spent a little of time to debug this myself. I think I found the cause of the leak. I've pushed a fix in commit bytedeco/javacpp@0efc632. DeleteLocalRef() wasn't getting called on the temporary byte[] allocated to create a String for the callback. With this fix, I do not get any memory leaks.

If you're still not able to get the latest binaries from the snapshots for some reason, you can simply clone https://github.com/bytedeco/javacpp and mvn install that locally in your container.

You'll also need to rebuild the presets for Triton itself after that to make this effective:
https://github.com/bytedeco/javacpp-presets/tree/master/tritonserver#steps-to-run-this-sample-inside-an-ngc-container

jackyh · 2022-04-08T01:58:24Z

Ok, I finally spent a little of time to debug this myself. I think I found the cause of the leak. I've pushed a fix in commit bytedeco/javacpp@0efc632. DeleteLocalRef() wasn't getting called on the temporary byte[] allocated to create a String for the callback. With this fix, I do not get any memory leaks.

If you're still not able to get the latest binaries from the snapshots for some reason, you can simply clone https://github.com/bytedeco/javacpp and mvn install that locally in your container.

You'll also need to rebuild the presets for Triton itself after that to make this effective: https://github.com/bytedeco/javacpp-presets/tree/master/tritonserver#steps-to-run-this-sample-inside-an-ngc-container

great thanks! try it this morning.

jackyh · 2022-04-08T06:06:53Z

Ok, I finally spent a little of time to debug this myself. I think I found the cause of the leak. I've pushed a fix in commit bytedeco/javacpp@0efc632. DeleteLocalRef() wasn't getting called on the temporary byte[] allocated to create a String for the callback. With this fix, I do not get any memory leaks.

If you're still not able to get the latest binaries from the snapshots for some reason, you can simply clone https://github.com/bytedeco/javacpp and mvn install that locally in your container.

You'll also need to rebuild the presets for Triton itself after that to make this effective: https://github.com/bytedeco/javacpp-presets/tree/master/tritonserver#steps-to-run-this-sample-inside-an-ngc-container

my god! I did all the things above, but always failed to find the symbols!!

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project simple: Compilation failure: Compilation failure:
[ERROR] /workspace/forMemoryLeakIssue/jackh0408/javacpp-presets/tritonserver/samples/Simple.java:[201,41] cannot find symbol
[ERROR] symbol: method newGlobalRef(java.lang.String)
[ERROR] location: class org.bytedeco.javacpp.Loader
[ERROR] /workspace/forMemoryLeakIssue/jackh0408/javacpp-presets/tritonserver/samples/Simple.java:[220,34] cannot find symbol
[ERROR] symbol: method accessGlobalRef(org.bytedeco.javacpp.Pointer)
[ERROR] location: class org.bytedeco.javacpp.Loader
[ERROR] /workspace/forMemoryLeakIssue/jackh0408/javacpp-presets/tritonserver/samples/Simple.java:[258,17] cannot find symbol
[ERROR] symbol: method deleteGlobalRef(org.bytedeco.javacpp.Pointer)
[ERROR] location: class org.bytedeco.javacpp.Loader
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

this make me mad...

jackyh · 2022-04-08T06:08:17Z

there's no "nm" in java?

how can I know if there's "newGlobalRef" in /root/.m2/repository/org/bytedeco/javacpp/1.5.8-SNAPSHOT/javacpp-1.5.8-SNAPSHOT.jar??

saudet · 2022-04-08T06:11:14Z

If you're using the "shaded" library of the presets for Triton, you'll need to update that one as well.

JAR files are just zip files, use unzip to extract the classes, and you can use the decompiler javap on those classes.

jackyh · 2022-04-08T06:23:42Z

newGlobalRef

looks like there's newGlobalRef in Loader.class:

root@55d883c095bc:~/.m2/repository/org/bytedeco/javacpp/1.5.8-SNAPSHOT/tmp/org/bytedeco/javacpp# javap Loader.class |grep newGlobalRef
public static native org.bytedeco.javacpp.Pointer newGlobalRef(java.lang.Object);

what's shaded? I just use these to compile and install tritonserver:

$ mvn clean install --projects .,tritonserver
$ mvn clean install -f platform --projects ../tritonserver/platform -Djavacpp.platform=linux-x86_64

jackyh · 2022-04-08T06:25:28Z

in Simple.java, it includes:

import org.bytedeco.javacpp.*;

what's error there? driving me mad...

saudet · 2022-04-08T06:30:34Z

For example, this file: https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/tritonserver-platform/2.18-1.5.8-SNAPSHOT/tritonserver-platform-2.18-1.5.8-20220318.033114-6-shaded.jar
If your build uses this file, it's going to use the Loader.class from there, not from the JAR of JavaCPP.

jackyh · 2022-04-08T06:41:56Z

I found this in javacpp-presets/tritonserver/samples/pom.xml:

org.bytedeco tritonserver-platform 2.18-1.5.8-SNAPSHOT shaded

I just remove "shaded" line:

org.bytedeco tritonserver-platform 2.18-1.5.8-SNAPSHOT

then do clean install, but, still get the same error...why??

jackyh · 2022-04-08T06:51:20Z

[INFO] Changes detected - recompiling the module!
[DEBUG] Classpath:
[DEBUG] /workspace/forMemoryLeakIssue/jackh0408/javacpp-presets/tritonserver/samples/target/classes
[DEBUG] /root/.m2/repository/org/bytedeco/cuda-platform/11.6-8.3-1.5.7/cuda-platform-11.6-8.3-1.5.7.jar
[DEBUG] /root/.m2/repository/org/bytedeco/javacpp-platform/1.5.7/javacpp-platform-1.5.7.jar
[DEBUG] /root/.m2/repository/org/bytedeco/javacpp/1.5.7/javacpp-1.5.7.jar
[DEBUG] /root/.m2/repository/org/bytedeco/javacpp/1.5.7/javacpp-1.5.7-linux-x86_64.jar

it still uses javacpp1.5.7?? why not 1.5.8?

saudet · 2022-04-08T07:05:42Z

It's possible, yes, unless we force it to a particular version, as above #1141 (comment).

saudet · 2022-04-08T07:11:04Z

Anyway, I've updated the shaded JAR file, so we can use it without Maven, for example, this way:

wget https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/tritonserver-platform/2.18-1.5.8-SNAPSHOT/tritonserver-platform-2.18-1.5.8-20220408.064151-7-shaded.jar
java -cp tritonserver-platform-2.18-1.5.8-20220408.064151-7-shaded.jar SimpleCPUOnly.java -r /workspace/models ...

jackyh · 2022-04-08T07:39:12Z

compiling is ok now.

I am now testing with:

java -Xms128m -Xmx384m -Dorg.bytedeco.javacpp.maxPhysicalBytes=800m -Dorg.bytedeco.javacpp.nopointergc=true -jar simple-1.5.8-SNAPSHOT.jar -r "/workspace/server-2.18.0/docs/examples/model_repository/models" -i 10000000

jackyh · 2022-04-08T08:35:39Z

compiling is ok now.

I am now testing with:

java -Xms128m -Xmx384m -Dorg.bytedeco.javacpp.maxPhysicalBytes=800m -Dorg.bytedeco.javacpp.nopointergc=true -jar simple-1.5.8-SNAPSHOT.jar -r "/workspace/server-2.18.0/docs/examples/model_repository/models" -i 10000000

10 millions passed! bravo!

jackyh · 2022-04-11T13:25:30Z

Ok, I finally spent a little of time to debug this myself. I think I found the cause of the leak. I've pushed a fix in commit bytedeco/javacpp@0efc632. DeleteLocalRef() wasn't getting called on the temporary byte[] allocated to create a String for the callback. With this fix, I do not get any memory leaks.

If you're still not able to get the latest binaries from the snapshots for some reason, you can simply clone https://github.com/bytedeco/javacpp and mvn install that locally in your container.

You'll also need to rebuild the presets for Triton itself after that to make this effective: https://github.com/bytedeco/javacpp-presets/tree/master/tritonserver#steps-to-run-this-sample-inside-an-ngc-container

Yes, I'm able to figure out from "oldSampleObject" that "BytePointer(tensor_name)" is not collected from heap. But since I'm not expert of JavaCPP, I'm not able to fix it...

saudet · 2022-04-11T23:16:01Z

Yes, I'm able to figure out from "oldSampleObject" that "BytePointer(tensor_name)" is not collected from heap. But since I'm not expert of JavaCPP, I'm not able to fix it...

Don't worry about that one, just apply the patch from above #1141 (comment).

saudet · 2022-11-03T04:30:22Z

The fix has now been released with JavaCPP 1.5.8! Thanks for reporting and for testing

saudet added help wanted question labels Feb 4, 2022

saudet added a commit to bytedeco/javacpp that referenced this issue Apr 8, 2022

* Fix Generator causing memory leaks for String parameters on ca…

0efc632

…llback (issue bytedeco/javacpp-presets#1141)

saudet added bug and removed help wanted question labels Apr 8, 2022

saudet closed this as completed Nov 3, 2022

about Memory growth test of Tritonserver #1141

about Memory growth test of Tritonserver #1141

Comments

jackyh commented Jan 28, 2022 • edited Loading

saudet commented Jan 28, 2022

jackyh commented Jan 28, 2022

saudet commented Jan 28, 2022 via email

jackyh commented Jan 28, 2022

saudet commented Jan 28, 2022

jackyh commented Jan 29, 2022

saudet commented Jan 29, 2022

jackyh commented Jan 29, 2022

jackyh commented Jan 29, 2022

saudet commented Jan 30, 2022

saudet commented Jan 30, 2022

jackyh commented Jan 30, 2022

jackyh commented Jan 30, 2022

jackyh commented Feb 3, 2022

saudet commented Feb 3, 2022

jackyh commented Feb 4, 2022

saudet commented Feb 4, 2022

jackyh commented Feb 4, 2022

saudet commented Feb 4, 2022

jackyh commented Feb 5, 2022

saudet commented Feb 5, 2022

jackyh commented Feb 6, 2022

jackyh commented Feb 6, 2022

saudet commented Feb 6, 2022

jackyh commented Feb 6, 2022

saudet commented Feb 7, 2022

jackyh commented Feb 7, 2022

saudet commented Feb 7, 2022

jackyh commented Apr 7, 2022

jackyh commented Apr 7, 2022

saudet commented Apr 7, 2022

jackyh commented Apr 7, 2022

jackyh commented Apr 7, 2022

saudet commented Apr 7, 2022

jackyh commented Apr 8, 2022

jackyh commented Apr 8, 2022

saudet commented Apr 8, 2022

jackyh commented Apr 8, 2022 • edited Loading

jackyh commented Apr 8, 2022

jackyh commented Apr 8, 2022

saudet commented Apr 8, 2022

jackyh commented Apr 8, 2022

jackyh commented Apr 8, 2022

saudet commented Apr 8, 2022

jackyh commented Apr 8, 2022

jackyh commented Apr 8, 2022 • edited Loading

saudet commented Apr 8, 2022

saudet commented Apr 8, 2022

jackyh commented Apr 8, 2022

jackyh commented Apr 8, 2022

jackyh commented Apr 11, 2022

saudet commented Apr 11, 2022

saudet commented Nov 3, 2022

jackyh commented Jan 28, 2022 •

edited

Loading

jackyh commented Apr 8, 2022 •

edited

Loading

jackyh commented Apr 8, 2022 •

edited

Loading