Inference time

[cahya-wirawan]

Hi, I am testing your code, the result looks very good, however when I tried it (several times), your code need around 1 to 4 minutes to generate sentence for one article "data/keyword_to_articles/test_5.txt" (similar times for other articles). Is it expected? I use and old gpu (nvidia 1080), but I think it wouldn't make a big different with newer gpu.

[dapascual]

Hi,
it is expected with the current implementation of the code, which is not optimized for inference speed. The main bottleneck is the computation of the cosine distances, which is done in each iteration in lines 129 and 140 in main. If instead you pre-compute those distances, which you can do since you give in advance the keywords you want to guide towards, the method will be much faster. I did some experiments with that but did not have time to make a proper update, it should speed up by 5-10x.

Pre=computing, basically means calculating the cosine distances in main, e.g. line 741, storing those distances in a large vector and passing it to the conditional_language_generation function, which then does not need to re-compute it at every iteration.

I hope this helps, let me know if it does :)

[cahya-wirawan]

Hi, I did it already actually (https://github.com/cahya-wirawan/K2T/tree/precalc_cosine). It is just a simple update of the cosine calculation. It is actually not really precalculated cosine similarity, but the the cosine of spec keyword is calculated only once during inference. Following the comparison of the inference time:

• data/keyword_to_articles/test_8.txt: original: 1090.27s, precalc_cosine: 185.16s (5.88X)
• data/keyword_to_articles/test_5.txt: original: 215.39s, precalc_cosine: 36.06s (5.97X)
• data/keyword_to_articles/test_10.txt: original: 498.69s, precalc_cosine: 103.86s (4.8X)

which is a nice inference time improvement, but still they are quite slow, since it still call sample_sentence() a lot of time.

[dapascual]

Yes, one limitation of the method as of now is that it's slow, it should still be optimized for inference speed. I'm not sure which configuration you are using, but if you are using beam search, the quality of the generated text is better but it takes considerably longer than using just one beam and nucleus sampling.

[sander-wood]

Hi, I have a quick question. Is this inference time linearly related to the number of keywords? I find that the inference time reaches 10,000 seconds with about 1K keywords (100 words generated). But if I use only 5 keywords the inference time is around 50 seconds.

[dapascual]

Hi, it should be more or less linear since it is applied on GPT-2, which is a sequential model, and the method repeats the same operation until all keywords are generated.

If the keywords are very different from each other such that it is harder to generate a sentence that contains them it may take a bit longer since it will need more conditioned generation steps, and those are more time-consuming that the unconditioned generation ones. But on average this delay should not be very relevant.

Hope this helps.

[sander-wood]

Thanks for your reply! If I understand correctly, K2T seems to be better suited for a smaller bag of words (e.g., 5 tokens), right? For example, if I want to generate text on a certain topic, instead of using all words related to the topic as keywords, would it be more appropriate to select a few representative words as keywords for the K2T generation strategy?