-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Inference time #4
Comments
Hi, Pre=computing, basically means calculating the cosine distances in main, e.g. line 741, storing those distances in a large vector and passing it to the conditional_language_generation function, which then does not need to re-compute it at every iteration. I hope this helps, let me know if it does :) |
Hi, I did it already actually (https://github.com/cahya-wirawan/K2T/tree/precalc_cosine). It is just a simple update of the cosine calculation. It is actually not really precalculated cosine similarity, but the the cosine of spec keyword is calculated only once during inference. Following the comparison of the inference time:
which is a nice inference time improvement, but still they are quite slow, since it still call sample_sentence() a lot of time. |
Yes, one limitation of the method as of now is that it's slow, it should still be optimized for inference speed. I'm not sure which configuration you are using, but if you are using beam search, the quality of the generated text is better but it takes considerably longer than using just one beam and nucleus sampling. |
Hi, I have a quick question. Is this inference time linearly related to the number of keywords? I find that the inference time reaches 10,000 seconds with about 1K keywords (100 words generated). But if I use only 5 keywords the inference time is around 50 seconds. |
Hi, it should be more or less linear since it is applied on GPT-2, which is a sequential model, and the method repeats the same operation until all keywords are generated. If the keywords are very different from each other such that it is harder to generate a sentence that contains them it may take a bit longer since it will need more conditioned generation steps, and those are more time-consuming that the unconditioned generation ones. But on average this delay should not be very relevant. Hope this helps. |
Thanks for your reply! If I understand correctly, K2T seems to be better suited for a smaller bag of words (e.g., 5 tokens), right? For example, if I want to generate text on a certain topic, instead of using all words related to the topic as keywords, would it be more appropriate to select a few representative words as keywords for the K2T generation strategy? |
Hi, I am testing your code, the result looks very good, however when I tried it (several times), your code need around 1 to 4 minutes to generate sentence for one article "data/keyword_to_articles/test_5.txt" (similar times for other articles). Is it expected? I use and old gpu (nvidia 1080), but I think it wouldn't make a big different with newer gpu.
The text was updated successfully, but these errors were encountered: