Feature/add integration to deepseek#1176
Merged
chakravarthik27 merged 4 commits intorelease/2.6.0from Feb 26, 2025
Merged
Conversation
…s for reasoning models
Contributor
There was a problem hiding this comment.
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
langtest/utils/custom_types/sample.py:561
- The newly added model_type parameter is passed directly to load_eval_model.model without any validation. Verify that the value from harness_config is compatible with the expected input type of load_eval_model.model.
model, hub, model_type=model_type, **model_parameters
Prikshit7766
approved these changes
Feb 24, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request includes several significant changes to improve the functionality and robustness of the codebase. The most important changes include adding a new method to extract the final answer from text, handling exceptions more effectively, and updating parameters for model evaluation.
Enhancements to exception handling and text processing:
langtest/modelhandler/llm_modelhandler.py: Added theextract_final_answermethod to process the text and extract the final answer for reasoning models, and modified thepredictmethod to utilize this new method. Additionally, improved exception handling by raising aValueErrorwith a specific error message.Updates to model evaluation parameters:
langtest/utils/custom_types/sample.py: Added a new parametermodel_typeto the__update_paramsmethod to support different model types during evaluation.langtest/utils/custom_types/sample.py: Updated the call toload_eval_model.modelwithin the__update_paramsmethod to include the newmodel_typeparameter.