Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training script for multi context training from ConveRT #14

Merged
merged 1 commit into from Jul 3, 2021

Conversation

infinitylogesh
Copy link
Contributor

Added Training script for multiple-context conversational model mentioned in conveRT paper. The code is adapted from @vasudevgupta7's code search. Updated the below

  • losses for the 3 objectives mentioned in the paper,

    • the interaction between the immediate context and its accompanying response,
    • the interaction of the response with up to N past contexts from the conversation history,
    • the interaction of the full context with the response

    However, the paper doesn't mention how the three losses are combined ( weighted or simple average).
    I have done a simple average for now. If there is a better way to do this (please let me know), can be updated as needed.

  • Past contexts are concatenated ( instead of separated by [SEP] token), as mentioned in the paper
    and as implemented here. Contexts are sorted to have the most recent context first and so on

I have tested this on GPU and the script works. Will update this with multi-context evaluation and sync with other recent changes done to the code-search training script.

Suggestions or feedback on this PR are welcome.

@infinitylogesh infinitylogesh changed the title Training script for multi contest training from ConveRT Training script for multi context training from ConveRT Jul 3, 2021
@nreimers nreimers merged commit 049d30a into nreimers:main Jul 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants