DLRM - 2 pre-processing questions: what is the "special value and why start at 1?

[s4ayub]

What is the "special value" that you replace infrequent values with when preprocessing?
Why do the remapped contiguous values start at 1?

[PiotrBigajNV]

Hi @s4ayub , which model are you talking about?

[s4ayub]

Hi @PiotrBigajNV, I'm referring to the preproc for pytorch DLRM: https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Recommendation/DLRM

I was also wondering, what benefit does it provide to do the following preprocessing steps:

1. mapping the sparse features to contiguous IDs
2. frequency thresholding - is this just to fit the model better on different sized GPUs?

[PiotrBigajNV]

Hi @s4ayub,

thanks for clarification.

ad.1. To be clear: Sparse (categorical) features are not mapped to continues ID, IDs are discreet, but they are consecutive non-negative integers. This is because they are used in the training to address the embeeding lookup and therefore they are convenient since .This way you can have an embeeding table of size num_categories x emb_dim. Othervise you would need to somehow map particular category value to address embeeding table. Of course this is doable in O(1), but space complexity would be O(n) (n =num_categories)

ad.2. This thresholding is a common practice. It serves 2 purposes:

• Fit model on particular GPU
• Capture all rarely occurring categories into one because otherwise for these categories you would overfit e.g. you have embeeding_dim=128, but just a single sample (value occurrence) for feature1.

Let me know if you need more clarification.

[s4ayub]

@PiotrBigajNV thanks for the response!

What is the "special value" that you replace infrequent values with when preprocessing for frequency thresholding?

[s4ayub]

also, @PiotrBigajNV is the embedding dimension number changed if you apply frequency thresholding? For example for T=0, you have an embedding_dim of 128. Does it remain 128 at T=3?

Really appreciate all the help here!

[PiotrBigajNV]

Hi @s4ayub ,

For NVTabular preprocessing by default the "rare" values are replaced by 0. Take a look at Categorify Op from NVTabular doc you can change it if you need.

Embeddings dimension is a model hyperparameter and is therefore a choice of a model developer. For Criteo dataset for all 3 presented FL the model converges for dim=128. It obviously does not mean that for all datasets dim=128 is a good choice. You can experiment with other dim, there is a flag in model training script.

Happy to help ;) Let me know if you need more support.