Bedrock LM produces extraneous key error in unclear circumstances

[JamesScharf]

AWS's Bedrock runtime does not accept the following parameters, however they are still being passed to the LM at some point:
n and max_tokens

At what point is this being done? I haven't specified either value in my code which uses DSPy.

Here's the full error from boto3:

ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: extraneous key [max_tokens] is not permitted#: extraneous key [n] is not permitted, please reformat your input and try again.

[tech4life87]

I am also getting the same exact error when trying to use "Getting Started" example notebook. Here is my code

bedrock = dspy.Bedrock(model='anthropic.claude-v2', region_name="us-west-2") colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts') dspy.settings.configure(lm=bedrock, rm=colbertv2_wiki17_abstracts)
Evaluation
`from dspy.evaluate.evaluate import Evaluate

evaluate_on_hotpotqa = Evaluate(devset=devset, num_threads=1, display_progress=True, display_table=5)

metric = dspy.evaluate.answer_exact_match
evaluate_on_hotpotqa(compiled_rag, metric=metric)`

[okhat]

I see. Does Bedrock support something else in place of n and max_tokens? Both are necessary in DSPy.

i.e., num_completions and maximum_output_tokens

If the former isn't supported, we can do it with a loop... but that's more expensive.

[okhat]

Anyway can fix now by just ignoring n and max_tokens in the request function for Bedrock @JamesScharf if they can't be replaced.

Otherwise, can fix by replacing these values on the fly with the correct kwarg names

[JamesScharf]

Re: n, I don't believe that Bedrock has any equivalent. So, we'd have to loop I think.
The equivalent to max_new_tokens is max_tokens_to_sample; there is no equivalent for max_tokens

[tech4life87]

@okhat It depends with which model one is using with Bedrock. If using Claude then as @JamesScharf correctly points out ismax_tokens_to_sample if using Titan Models it is maxTokenCount . I am kind of curious if using something like LiteLLM or langChain would abstract these model/provider specific nuances.

[JamesScharf]

I'd be hesitant to use LiteLLM for this abstraction. I've used its Sagemaker wrapper and it did not seem to support temperature.

[JamesScharf]

I'll make a PR with the following updates to Bedrock--let me know if I'm missing anything:

• Raise not supported error if user tries to use non-Claude models [for now]
• Implement n as a loop since n is not currently supported by Bedrock (to my knowledge)
• Remap max_tokens to max_tokens_to_sample

[shabie]

Bedrock's parameters are provider specific and Langchain makes no attempt to unify them (which will anyways be a bad idea). So I am not sure how much introduction of Langchain help.

[aazizisoufiane]

Hello @JamesScharf,

The issue persists for me. As a workaround, I implemented a removal of the n key from kwargs in the following manner:
if "n" in kwargs:
if self._batch_n:
del kwargs["n"]
llm_out = self._simple_api_call(formatted_prompt=formatted_prompt, **kwargs)
is that correct?

[JamesScharf]

Good catch, would you mind making a PR? I am away from my computer for a few days and can't do so myself.

[aazizisoufiane]

I'm encountering a permission issues for the PR

[aazizisoufiane]

@JamesScharf I have a new error with bedrock botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: max_tokens_to_sample: range: 1..1,000,000

[aazizisoufiane]

@tech4life87 I got the same error using Litellm: openai.BadRequestError: Error code: 400 - {'error': {'message': 'BedrockException - BedrockException - An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: extraneous key [drop_params] is not permitted, please reformat your input and try again.', 'type': None, 'param': None, 'code': 400}}