GPT2 generating repetitive text

[DEBADRIBASAK]

I was trying to use the pretrained GPT2LMHeadModel for generating texts by feeding some initial English words. But it is always generating repetitive texts.
Input: All
Output: All All the same, the same, the same, the same, the same, the same, the same, the same, the same, the same, the same, the same,

Here is my code:
`import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.nn.functional as F
from pytorch_pretrained_BERT.pytorch_pretrained_bert.modeling_gpt2 import GPT2LMHeadModel
from pytorch_pretrained_BERT.pytorch_pretrained_bert.tokenization_gpt2 import GPT2Tokenizer
from pytorch_pretrained_BERT.pytorch_pretrained_bert.optimization_openai import OpenAIAdam
from tqdm import tqdm
import torch.optim as optim
import random
import time
import os
import sys
import random
import argparse
from pathlib import Path
from torch.utils.data import Dataset,TensorDataset,DataLoader,SequentialSampler,RandomSampler

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained("gpt2")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
df = pd.read_csv("gpt_test.csv",sep="\t")
df = df.values

#context = tokenizer.encode(context)

model.to(device)
source = []
generated = []

for l in tqdm(range(len(df))):
source.append(str(df[l,-2]))
context = tokenizer.encode(str(df[l,-2]))
past = None
for i in range(40):
input_ids = torch.tensor([context])
input_ids = input_ids.to(device)
pred,_ = model(input_ids=input_ids)
predictions = torch.argmax(pred[0,-1,:]).item()
context.append(predictions)
if(predictions==2):
break
generated_text = tokenizer.decode(context)
generated.append(generated_text)

df1 = pd.DataFrame({'Source': source,'Generated': generated})
df1.to_csv("./result_with_gpt.csv",sep="\t")`
Can someone point out the mistake? I will be highly grateful if the response is fast.

[thomwolf]

Have you tried the provided GPT-2 generation example? It's here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_gpt2.py

[nkami]

Have you tried the provided GPT-2 generation example? It's here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_gpt2.py

Hey, I encountered the same issue. I tried using the example you provided but it tends to produce repetitive text much more often than earlier versions of the library as well (from around 1-2 months back). Thank you very much for all the work!

[DEBADRIBASAK]

Have you tried the provided GPT-2 generation example? It's here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_gpt2.py

I tried the example. It is working properly. I think with torch.argmax there is chance of repetitive text generation. If we sample using torch.multinomial, there is always some variation.

[HansBambel]

torch.argmax is basically top_k with 1 which is very bad for creating "human like" sentences. A better way to sample is using Nucleus Sampling https://arxiv.org/abs/1904.09751.
Not sure whether this is implemented in PyTorch yet.
EDIT: I have found this code from @thomwolf https://gist.github.com/thomwolf/1a5a29f6962089e871b94cbd09daf317 that implements Nucleus sampling.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[drizzt00s]

generating repetitive text when using GPU, but this not happens when using CPU, anyone knows how to solve this weird issue?

[pallavimitra]

when using GPU, but this not happens when using CPU, anyone knows how to solve this weird issue?

I am facing the same issue. Can someone lead in this problem?