enh(dspy): Move max_tokens explicitally into GPT3 constructor
#734
Conversation
Makes the reason for output truncation clearer as per stanfordnlp#730 Also adds a comment to the predict file where the max tokens are modified before the call is made.
| keys = list(kwargs.keys()) + list(dsp.settings.lm.kwargs.keys()) | ||
| max_tokens_key = "max_tokens" if "max_tokens" in keys else "max_output_tokens" | ||
| new_kwargs = { | ||
| **kwargs, |
There was a problem hiding this comment.
How the value set by the user effects the maximum tokens passed and returned from the model is quite unclear here, would anyone be able to clarify how 101:118 relate to the eventual values passed to different models, I'll have a crack at clearing this up. Might it be possible to do the 75 checking upstream of here?
|
This is mainly during retries etc. It's not very straightforward to change. |
|
Basically, the user's max_tokens is used at first, then max_tokens / 2 if we need more fields, but 75 is sort of like a min_max_tokens. Setting a large enough max_tokens could be all that's required here? |
|
Thanks for taking a look @okhat Yeah the main thing for me is putting it into the init so it's clear to the user that they need to set it. What do you think of this part? In terms of the retries, do you mean if |
|
@harrysalmon just following up on this. I think we can merge the change but leave the existing retry behavior. lmk if it's ready to merge. |
|
Closing for now to avoid breaking cache errors. |
3 participants
Makes the reason for output truncation clearer as per #730
Also adds a comment to the predict file where the max tokens are modified before the call is made.