gpu utilization 100%

[520jefferson]

Firstly, I compile FT4.0 , then i join the enc dec so library into tensorflow serverint 1.15 to compile, but when i start the serving. and access by grpc, then the gpu utilization become 100%.

[byshiue]

Sorry. I don't understand your question.

[520jefferson]

I mean, After i compile the tensorflow serving with custom FT4.0 library libtf_decoding_beamsearch.so libtf_decoding_sampling.so libtf_fastertransformer.so.
Then i start the serving with batch decoding function and load FT4.0 optimized model , then the gpu utilization will suddenly increate to nearly 100%, then the service cannot be access.
I cannot figure out what's wrong.

[byshiue]

We don't test the TF serving although I heard some users use FT in TF serving successfully.
You can try the nsight systems to log the process to check that do we run any FT kernel? Or Add some debug information into FT TF Ops.

[520jefferson]

yes, i call the FT kernel in TF serving. and i can use the FT in TF serving . but once i met the gpu utilization 100% error. someone who have met this error said maybe it is the Deadlock caused by concurrent threads.

[byshiue]

Do you use single thread?

[520jefferson]

Single thread is okay. multi process access is also okay. It is not easy to reappear.

[byshiue]

It may be some bugs of multi-thread. We don't test of multi-threads, and some private resources may be shared under multi-thread.

[520jefferson]

Oh, I forget something, i use fast fp16 then the error may appear, but fp32 is okay.

[calico-niko]

I mean, After i compile the tensorflow serving with custom FT4.0 library libtf_decoding_beamsearch.so libtf_decoding_sampling.so libtf_fastertransformer.so.
Then i start the serving with batch decoding function and load FT4.0 optimized model , then the gpu utilization will suddenly increate to nearly 100%, then the service cannot be access.
I cannot figure out what's wrong.

Hi @520jefferson

I'm trying to compile TF serving with FasterTransformers. As far as I know:

step 1: Build FasterTransformers to obtain the "libtf_xxx.so" file. (I tried this with FT v5.3 but got files named "libtf_bert.so" and others, not libtf_fastertransformer.so.)
step 2: Compile TF serving with the .so file obtained in step 1 to get custom ops.
step 3: Serve the saved_model using the compiled TF serving.

Did I miss anything? Do I need to retrain and export a new saved_model with FasterTransformers, or just train with the official bert repo?

I would really appreciate it if you could provide some help, Thanks!