[Task] - Incorporate embeddings (pre-trained and encoder-based) into Merlin Training and Inference

[MarkMoTrin]

Problem:

Two+ customer teams would like to leverage encoder-based embeddings as additional features in Transformers4Rec. Currently, there is no clear example/path to pass in extra embeddings. This gap in feature prevents teams from adopting Transformers4Rec in production.

Goal:

• Produce a comprehensive example on how to incorporate embeddings as extra features into Transformers4Rec. Embeddings may be stored in parquet format. Generalizations should be made where possible in examples, especially on defining the schema and other input parameters.

Constraints:

• Embeddings may be stored in parquet files
• Workflow should be able to be deployed in Triton

Starting Point:

[MarkMoTrin]

using-model-embeddings-nvtabular.pdf

[viswa-nvidia]

Removing the milestone 22.09. This is not in the POR yet.

[viswa-nvidia]

@benfred , @jperez999 ,
Review this ticket and add planning details. It is still in needs definition.
Convert the check boxes to tickets in the specific repos ( Please remember to update the initiative field )
Once you are satisfied with the planning change status to RMP review
Refer to this doc for the workflow
There is a link to the video that explains the workflow. Let me know if you have any questions.

[oliverholworthy]

Edit: sorry, this comment is unrelated to this issue (Transformers4Rec). Related Merlin Models (Tensorflow).

For small embedding tables (fit in GPU memory). Using a tensor initializer is the currently supported functionality (in Merlin Models) for pre-trained embedding tables. An example in this notebook:

[karlhigley]

@MarkMoTrin When you say "encoder-based embeddings as additional features," does that imply encoding an item attribute (like "text description") using an external model (like BERT) and supplying the resulting embeddings as features for each item in a session? We're not talking about adding a single embedding as a session-level feature, we're talking about adding a list of embeddings as item-level features, yeah?

[gabrielspmoreira]

When the embedding table are not huge and fit GPU memory, the new PretrainedEmbeddingsInitializer ( NVIDIA-Merlin/Transformers4Rec#572 ) can be used to initialize the embedding matrix with pre-trained embeddings and set them to trainable or not.