Slow prediction problem

[WestbrookGE]

First of all, I want to say this repository is really helpful to my work. It trains really fast and efficient from small dataset to large dataset.
However, I found it really slow when I predicted data using saved model. When I started predicting, the number that shows how mach examples per second is nearly 4k, but the actual number of that is 200. I've checked the process status, it showed that most of the time the CPU usage of every core is nearly 0%! Sometimes it will increase to 20% or 50%, but it just lasted a few seconds.
I've tried prediction on different scaled data, with different parameters, this problem happened all the time.
I'm using google cloud machine with intel CPU and 64GB Memory, I'm not sure if this problem is due to my machine.
Thanks again!

[tomtung]

Are you using the Python API for prediction? In that case you can try calling densify_weights with lower density number (e.g. 0.01) to reduce prediction time at the cost of more memory usage.

[WestbrookGE]

Are you using the Python API for prediction? In that case you can try calling densify_weights with lower density number (e.g. 0.01) to reduce prediction time at the cost of more memory usage.

No, I'm using the CLI app. I've tried to lower the density number to 0.0001(which used 50GB memory, BTW), but this problem still exists. I think the major problem is CPU is not working most of the time, which is really weird. I guess some procedure of prediction is too slow so that CPU has to keep waiting.

Does the prediction part work well on your machine? If yours works fine, I guess it's my own problem.

By the way, the data I used to train/predict is really dense(dimension reduced from 100K to 200). I do not know if it may cause the problem.
Thanks for your help!

PS:
This is the result of my prediction process.

587084 / 587084 [=======================================================] 100.00 % 143.64/s
2020-07-03 09:37:16,940 INFO [omikuji::model::eval] Done testing on 587084 examples; it took 4087.13s

I implemented this code on the origin version of WikiLSHTC-325K, the slow prediction still exists!

[tomtung]

Hmm I couldn't reproduce the low CPU usage. Not sure if the problem is specific to Google cloud machines. Can you try manually setting --n_threads and see if that helps? By default it should detect the number of CPUs automatically, but I'm wondering if there's something non-standard with Google cloud machines.