[Question] How can I pre-calculate the GPU memory required for embedding cache size? #427
Comments
tuanavu
changed the title
|
Regarding 2: Regarding 3: |
|
@tuanavu Regarding the 2rd question, I have some comments here. We already support quantization for fp8 in the static embedding cache from v23.08. HPS will perform fp8 quantization on the embedding vector when reading the embedding table by enable Since the embedding is stored with fp8 type and the GPU memory size will be greatly reduced. However, due to different business use cases, the precision loss caused by quantization/dequantization still needs to be evaluated in the real production. So currently we only have experimental support for static embedding caching for POC verification. If quantization can bring greater benefits to your case, we will add quantization features to dynamics and upcoming lock-free optimized gpu cache. |
Details
My company currently operates a Recommender model trained with TensorFlow 2 (TF2) and served on CPU pods. We are exploring the potential of HugeCTR due to its promising GPU embedding cache capabilities and are considering switching our model to it.
We have successfully retrained our existing TF2 model with the SparseOperationsKit (more info) and created the inference graph with HPS, as demonstrated in these notebooks: sok_to_hps_dlrm_demo.ipynb and demo_for_tf_trained_model.ipynb
Result:
We deployed the model and used Triton's perf_analyzer to test its performance with varying batch sizes. The results were as follows:
Testing Environment:
To maximize throughput, we plan to test the model across different instance types with varying GPU memory sizes. However, optimizing different parameters in config and selecting the best instance type for inference requires a clear understanding of how embedding cache size is calculated.
Details about the current model and embedding tables:
Our current model has various dense, sparse and pre-trained sparse features. After exporting the TF+SOK model to HPS, we have total 42 embedding tables, i.e.:
sparse_filesin hps_config.json. Here’s the stats:hps.Initoutputhps_config.jsonused for inference{ "supportlonglong": true, "volatile_db": { "type": "parallel_hash_map", "allocation_rate": 100000.0, "initial_cache_rate": 1.0 }, "persistent_db": { "type": "disabled" }, "models": [ { "model": "full_model_max_mono", "sparse_files": [ list_of_sparse_files ], "num_of_worker_buffer_in_pool": 2, "instance_group": 4, "embedding_table_names": [ "hps_dense_emb_monolith", "hps_sparse_emb_monolith", "hps_sf_101", "hps_sf_102", "hps_sf_11", "hps_sf_14", "hps_sf_15", "hps_sf_18", "hps_sf_3", "hps_sf_4", "hps_sf_5", "hps_sf_56", "hps_sf_6", "hps_sf_60", "hps_sf_67", "hps_sf_68", "hps_sf_69", "hps_sf_70", "hps_sf_71", "hps_sf_72", "hps_sf_73", "hps_sf_74", "hps_sf_75", "hps_sf_76", "hps_sf_77", "hps_sf_78", "hps_sf_79", "hps_sf_80", "hps_sf_89", "hps_sf_90", "hps_sf_91", "hps_sf_92", "hps_sf_93", "hps_sf_94", "hps_sf_95", "hps_sf_96", "hps_sf_97", "hps_sf_98", "hps_sf_99", "hps_TopicVAEDigestAnswer180DayWDR-sf_2", "hps_TopicVAETopics-sf_1", "hps_TopicVAETopics-sf_7" ], "embedding_vecsize_per_table": [ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 8, 8, 8 ], "maxnum_catfeature_query_per_table_per_sample": [ 221, 7, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 1, 100, 100 ], "default_value_for_each_table": [ 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0 ], "deployed_device_list": [ 0 ], "max_batch_size": 24000, "cache_refresh_percentage_per_iteration": 0.2, "hit_rate_threshold": 1.1, "gpucacheper": 0.8, "gpucache": true } ] }Questions
allocation_rateconfiguration in the abovevolatile_db. I observed that I must reduceallocation_rate = 1e6, or else the default allocation (256 MiB) leads to out-of-memory issue duringhps.init. Could you explain why this happens and provide some insights into this matter?The text was updated successfully, but these errors were encountered: