-
-
Notifications
You must be signed in to change notification settings - Fork 8.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
xgboost4j matrix memory leak #10300
Comments
Hi, is the DMatrix object released after use? |
Yes, I wrote a demo and dispose immediately after new, but the internal memory still shows that xgboost jni has increased. My java version is 11.20 |
I understand that they are all running on jvm, and theoretically it should have nothing to do with the operating system. However, I run the same code on my own jvm on macos, and this problem does not occur. Can this provide you with some tips? |
The operating system where the problem occurred is linux 5.15.0-52 |
There are two places responsible for memory consumption, first is the DMatrix object itself for storing input data, second is the prediction cache. The prediction cache is NOT immediately freed after the destruction of a DMatrix, rather it's checked when a new cache item is requested: xgboost/include/xgboost/cache.h Line 71 in a5a5810
As a result, the result of the memory usage might not look consistent when the caching is being built. |
I understand that if it is a cache, it should not prompt Java_ml_dmlc_xgboost4j_java_XGBoostJNI_XGDMatrixCreateFromCSR. If the cache also prompts this, please tell me how to clear it. My service will continue to grow due to this cache until the memory reaches 100% and then crash. |
I don't think XGBoost will take up that much memory. As mentioned previously, a cache item is evicted as soon as a new cache item is requested and the previous DMatrix is freed: xgboost/include/xgboost/cache.h Line 81 in a5a5810
|
Are you launching new threads for prediction without finishing the thread afterward? |
Yes, I created an xgboost object pool with 20 boosts in it, and then I will use multi-threading to provide prediction capabilities |
But I think even if there are 20 boost objects, there are only 20 caches, because predict is a synchronous method |
Now there is a strange problem. I cannot reproduce this phenomenon of internal memory growing on Mac. It only occurs in the production environment. |
[0x00007fc52398d029] jni_GetIntArrayElements+0x169 [0x00007fc52398daa9] jni_GetFloatArrayElements+0x169 As shown in the log, it is theoretically impossible for malloc memory to be as large as 290MB. |
Hi @huangwei907781034, Could you have the minimal code including the mimiced data to repro it ? and also BTW, How to get the Native Memory Tracking? Thx |
Hi, I have located the problem. The input data contains the value of inf, which causes the matrix to be created abnormally. However, the problem is that the matrix is created abnormally, but the memory will not be released. After I determined the value of inf, my memory was normal. |
Thank you for sharing! That's really helpful. |
problem:
I loaded xgboost model files based on java spring to provide online model prediction services. However, while the service was running, the memory kept increasing. Through JVM NMT, I found that it was Java_ml_dmlc_xgboost4j_java_XGBoostJNI_XGDMatrixCreateFromCSR that caused Internal Memory to keep increasing.
verison:
ml.dmlc
xgboost4j_2.12
1.5.0
code:
dMatrix = new DMatrix(ListUtil.toLongArray(headers), ListUtil.toIntArray(indices), ListUtil.toFloatArray(data), DMatrix.SparseType.CSR, (int) numFeature);
try code:
dMatrix = new DMatrix(ListUtil.toLongArray(headers), ListUtil.toIntArray(indices), ListUtil.toFloatArray(data), DMatrix.SparseType.CSR, (int) numFeature);
dMatrix.dispose();
In order to avoid being affected by my other code, I tried to process the matrix immediately after new, but the memory was still not released. I don’t know if there is any problem with dispose.
NMT log:
Native Memory Tracking:
The text was updated successfully, but these errors were encountered: