-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Please provide full narration hyper-parameters #5
Comments
Hi,
Sorry for the late reply. I was quite overwhelmed these days.
For your question:
1. You say you use nucleus sampling with top_p=0.95 and choose k=5 for
having 5 candidates. What temperature do you use?
We used temperature=0.7. Basically, we follow the default haprams setting
of LaViLa, except that we change caption-num-return-sequences to 5 to
accelerate.
[image: image.png]
2. Moreover, in the paper you say that you take the narration with the
largest confidence score as the final caption. I assume you mean by that
taking the candidate with the lowest perplexity (since LaViLa caption model
returns output token ids together with the perplexity values)?
Thanks for pointing it out! I double checked our code for captioning, and
found that we were actually picking up the caption that has the largest
perplexity. While we made fair comparisons in our paper, this is definitely
a suboptimal solution.
3. Besides, I saw you reporting in the paper that you use a temperature=0.0 for
the LLMs, but I see in the readme you provide example commands for the
summarization task with temperature=1.0. So do you use temperature=1.0 for
LLM in summarization task and temperature=0.0 in QA task?
Yes, we only use temperature=1.0 in the summarization task because we found
it better practically. For QA tasks we use temperature = 0.0.
Thanks for your questions, and good luck with your future research!
…On Wed, Apr 24, 2024 at 3:50 AM maximotus ***@***.***> wrote:
Hi ;)
for comparability reasons, it would be beneficial for the community to
have insights into the full hyper-parameter setups.
I am especially interested in the LaViLa captioning config to use with
your provided fair checkpoint for EgoSchema.
In detail, I need the following information:
1. You say you use nucleus sampling with top_p=0.95 and choose k=5 for
having 5 candidates. What temperature do you use?
2. Moreover, in the paper you say that you take the narration with the
largest confidence score as the final caption. I assume you mean by that
taking the candidate with the lowest perplexity (since LaViLa caption model
returns output token ids together with the perplexity values)?
3. Besides, I saw you reporting in the paper that you use a
temperature=0.0 for the LLMs, but I see in the readme you provide
example commands for the summarization task with temperature=1.0. So
do you use temperature=1.0 for LLM in summarization task and
temperature=0.0 in QA task?
Clearification would be much appreciated! :)
Cheers,
Maximotus
—
Reply to this email directly, view it on GitHub
<#5>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKXZD56IABN5UXGWXKRJQHLY65P35AVCNFSM6AAAAABGWMFHOCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DANRSGQ2TGMI>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Thank you very much for the provided information, that is very helpful! May I ask why you do not provide ChatGPT 4 results using your sum + qa pipeline (or have you tested it, and if so, could you share the results)? More insights would be much appreciated! :) Cheers, |
Hi ;)
for comparability reasons, it would be beneficial for the community to have insights into the full hyper-parameter setups.
I am especially interested in the LaViLa captioning config to use with your provided fair checkpoint for EgoSchema.
In detail, I need the following information:
top_p=0.95
and choosek=5
for having 5 candidates. What temperature do you use?temperature=0.0
for the LLMs, but I see in the readme you provide example commands for the summarization task withtemperature=1.0
. So do you usetemperature=1.0
for LLM in summarization task andtemperature=0.0
in QA task?Clearification would be much appreciated! :)
Cheers,
Maximotus
The text was updated successfully, but these errors were encountered: