Transformer-based NextItemRecommender models#699
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds transformer-based next-item recommenders to Cornac’s NextItemRecommender family (SASRec / BERT4Rec / GPT2Rec), along with shared validation-based checkpoint selection and updated Diginetica split loading semantics to support session-based vs session-aware evaluation.
Changes:
- Introduce new next-item models:
SASRec(native PyTorch),BERT4RecandGPT2Rec(HuggingFace backbones), plusFPMC. - Add shared
val_score()helper and extendGRU4Recto supportmodel_selection="best"based on validation ranking metrics. - Update Diginetica
load_val/load_testto default tomode="session-based"and document/test the new behavior; add new examples and README entries.
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/cornac/datasets/test_diginetica.py | Adjust expected sizes for new default Diginetica val/test mode; assert session-aware mode retains old counts. |
| README.md | Register new models in the project-wide model table (SASRec/BERT4Rec/GPT2Rec/FPMC). |
| examples/transformer_rec_diginetica.py | New end-to-end example training GRU4Rec + transformer-based next-item models on Diginetica. |
| examples/README.md | Add FPMC example entry (but currently missing transformer example entry). |
| examples/fpmc_diginetica.py | New end-to-end example training FPMC on Diginetica. |
| cornac/models/seq_utils/selection.py | New val_score() helper for validation ranking evaluation during training. |
| cornac/models/seq_utils/init.py | Export val_score from seq_utils. |
| cornac/models/sasrec/sasrec.py | SASRec encoder implementation (noted padding attention masking issue). |
| cornac/models/sasrec/requirements.txt | Declare torch dependency for SASRec. |
| cornac/models/sasrec/recom_sasrec.py | SASRec recommender wrapper with training loop + best-on-val selection. |
| cornac/models/sasrec/init.py | Package init for SASRec. |
| cornac/models/gru4rec/recom_gru4rec.py | Add best-on-val model selection to GRU4Rec via val_score(). |
| cornac/models/gpt2rec/requirements.txt | Declare torch + transformers dependency for GPT2Rec. |
| cornac/models/gpt2rec/recom_gpt2rec.py | GPT2Rec recommender wrapper with training loop + best-on-val selection. |
| cornac/models/gpt2rec/gpt2rec.py | GPT-2 backbone module (noted unused tie_weights parameter). |
| cornac/models/gpt2rec/init.py | Package init for GPT2Rec. |
| cornac/models/fpmc/requirements.txt | Declare torch dependency for FPMC. |
| cornac/models/fpmc/recom_fpmc.py | FPMC recommender wrapper with training loop + best-on-val selection. |
| cornac/models/fpmc/fpmc.py | FPMC PyTorch module (noted naming inconsistency FPMC_Model). |
| cornac/models/fpmc/init.py | Package init for FPMC. |
| cornac/models/bert4rec/requirements.txt | Declare torch + transformers dependency for BERT4Rec. |
| cornac/models/bert4rec/recom_bert4rec.py | BERT4Rec recommender wrapper with training loop + best-on-val selection. |
| cornac/models/bert4rec/bert4rec.py | BERT backbone module (noted unused tie_weights parameter). |
| cornac/models/bert4rec/init.py | Package init for BERT4Rec. |
| cornac/models/init.py | Export new models from top-level cornac.models. |
| cornac/datasets/README.md | Document Diginetica session-based vs session-aware loading semantics. |
| cornac/datasets/diginetica.py | Add mode parameter to load_val/load_test, defaulting to session-based. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+132
to
+147
| timeline_mask = (hist_iids == self.pad_idx).to( | ||
| dtype=seqs.dtype, device=seqs.device | ||
| ) | ||
| seqs = seqs * (1.0 - timeline_mask).unsqueeze(-1) | ||
|
|
||
| tl = seqs.shape[1] | ||
| attention_mask = ~torch.tril( | ||
| torch.ones((tl, tl), dtype=torch.bool, device=seqs.device) | ||
| ) | ||
|
|
||
| for i in range(len(self.attention_layers)): | ||
| seqs_t = torch.transpose(seqs, 0, 1) | ||
| Q = self.attention_layernorms[i](seqs_t) | ||
| mha_out, _ = self.attention_layers[i]( | ||
| Q, seqs_t, seqs_t, attn_mask=attention_mask | ||
| ) |
Comment on lines
+32
to
+54
| def __init__( | ||
| self, | ||
| item_num, | ||
| embedding_dim=100, | ||
| maxlen=20, | ||
| n_layers=2, | ||
| n_heads=1, | ||
| dropout=0.1, | ||
| pad_idx=-1, | ||
| tie_weights=False, | ||
| init_std=0.02, | ||
| device="cpu", | ||
| ): | ||
| super().__init__() | ||
| from transformers.models.bert import BertConfig, BertModel | ||
|
|
||
| self.item_num = item_num | ||
| self.pad_idx = pad_idx if pad_idx >= 0 else item_num | ||
| self.maxlen = maxlen | ||
| self.dev = device | ||
| self.init_std = init_std | ||
| self.tie_weights = tie_weights | ||
|
|
Comment on lines
+27
to
+49
| def __init__( | ||
| self, | ||
| item_num, | ||
| embedding_dim=100, | ||
| maxlen=20, | ||
| n_layers=2, | ||
| n_heads=1, | ||
| dropout=0.1, | ||
| pad_idx=-1, | ||
| tie_weights=False, | ||
| init_std=0.02, | ||
| device="cpu", | ||
| ): | ||
| super().__init__() | ||
| from transformers.models.gpt2 import GPT2Config, GPT2Model | ||
|
|
||
| self.item_num = item_num | ||
| self.pad_idx = pad_idx if pad_idx >= 0 else item_num | ||
| self.maxlen = maxlen | ||
| self.dev = device | ||
| self.init_std = init_std | ||
| self.tie_weights = tie_weights | ||
|
|
Comment on lines
127
to
131
| [gru4rec_yoochoose.py](gru4rec_yoochoose.py) - Example of Session-based Recommendations with Recurrent Neural Networks (GRU4Rec). | ||
|
|
||
| [fpmc_diginetica.py](fpmc_diginetica.py) - Example of Factorizing Personalized Markov Chains (FPMC) with Diginetica dataset. | ||
|
|
||
| ---- |
Comment on lines
+20
to
+22
| class FPMC_Model(nn.Module): | ||
| """Factorizing Personalized Markov Chains (Rendle et al., 2010). | ||
|
|
qtuantruong
approved these changes
Jun 4, 2026
hieuddo
force-pushed
the
transformer-models
branch
from
9a4ca93 to
82a5aaa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Transformer-based NextItemRecommender models, including
SASRec,BERT4Rec, andGPT2Rec(BERT4Rec, but GPT2 architecture).Currently based on https://github.com/hieuddo/cornac/tree/fpmc-model, will rebase once #698 is merged.
Quick results, different models would require different
lr:Related Issues
Checklist:
README.md(if you are adding a new model).examples/README.md(if you are adding a new example).datasets/README.md(if you are adding a new dataset).