[BugFix] - Untyped variadic keyword arguments break during execution

[montezdesousa]

1. Why?

   • Untyped VAR_KEYORD arguments in endpoints like **kwargs are not supported
   • Warning that checks extra_params is raising by mistake whenever extra_params exists

2. What?

   • Warn only if extra_params is of type ExtraParams
   • validate_kwargs allows untyped arguments like **kwargs to pass validation as Any
   • In the API signature wrapper make sure that VAR_KEYWORD arguments always come last
   • In the package builder generalises VAR_KEYWORD argument names to any name and makes sure they always come last

3. Impact

   • We can now use VAR_KEYWORD parameters (**kwargs) in the endpoints, although this is has some non obvious caveats in the API - it will make the VAR_KEYWORD a required URL parameter, since ** parameters cannot have a default, thus FastAPI sees it as required. Will be on the developer side if this is good for the use case or not.

4. Testing Done:

   • Add extra_params: Optional[Dict] = None to any POST method, rebuild, run commands or make API calls
   • Add **any_arg to an endpoint and run commands or make API calls

[deeleeramone]

I still get the same error,

Cannot parse: 593:8:         chart: Annotated[bool, OpenBBCustomParameter(description="Whether to create a chart or not, by default False.")] = False

We very much need this in PackageBuilder:

for k in ["provider", "extra_params", "**kwargs"]

We still need changes in CommandRunner:

ValidationError: 1 validation error for cones
kwargs
  Input should be an instance of _empty [type=is_instance_of, input_value={'chart_params': {'tite': 'some param'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.6/v/is_instance_of

There are two things that happen in my proposed changes to CommandRunner:

• "kwargs" gets popped from the list of parameters that gets validated. This is required so a "kwargs" is not considered as a field for validation.
• "kwargs" is properly unpacked so that it is flat relative to the rest of the params in the POST function. i.e, not dumped as a field name called, "kwargs".

[deeleeramone]

"kwargs" is being forced as a "required" field in the body, and calling it VAR_KEYWORD or VAR_POSITIONAL is not allowed with a default value. Optional[Dict] is not actually optional without a default state. Setting as POSITIONAL_OR_KEYWORD allows you to set a default value of {}, and **kwargs is still in the request body.

It would be ideal if you didn't have to put kwargs in the body, but could just add them to the URL like every other parameter. This behaviour, though, is limited to the API. The Python interface does not require them to be declared within the field "kwargs".

        if parameter.name == "kwargs":
            new_parameter_list.append(
                Parameter(
                    parameter.name,
                    kind=parameter.POSITIONAL_OR_KEYWORD,
                    annotation=Optional[Dict[str, Any]],
                    default={},
                )
            )
        else:
            new_parameter_list.append(
                Parameter(
                    parameter.name,
                    kind=parameter.kind,
                    default=parameter.default,
                    annotation=parameter.annotation,
                )
            )

[montezdesousa]

It's not possible to add undefined URL parameters to API endpoints

[deeleeramone]

Well then we can't get around that "nice to have" thing, but at least they can be unpacked the same as the Python interface.

This: body = json.dumps({"data": data, "kwargs": {"chart_params": {"title": "my chart"}}})

When unpacked becomes the same as the Python Interface "extras":

{'metadata': {'arguments': {'index': 'date',
   'lower_q': 0.25,
   'upper_q': 0.75,
   'model': 'std',
   'is_crypto': False,
   'trading_periods': None,
   'data': {'type': 'List[Data]',
    'columns': ['open',
     'vwap',
     'split_ratio',
     'close',
     'volume',
     'dividend',
     'date',
     'low',
     'high']},
   'chart_params': {'title': 'my chart'}},
  'duration': 109062167,
  'route': '/technical/cones',
  'timestamp': '2024-03-22T12:05:53.906124'}}

[montezdesousa]

Metadata reflects the call that was made to the api, if the call sends 'chart_params': {'title': 'my chart'}} inside a parameter called kwargs/some_kw_args/extra_params it is expected that you see

{'metadata': 
{'arguments': 
        {
         'index': 'date',
         'lower_q': 0.25,
         'data': {
                    'type': 'List[Data]',
                    'columns': ['open', ...],
          },
         'kwargs/some_kw_args/extra_params': {'chart_params': {'title': 'my chart'}}
         },
  'duration': 109062167,
  'route': '/technical/cones',
  'timestamp': '2024-03-22T12:05:53.906124'}}

[deeleeramone]

For this purpose, "extra_params" is a better name for this than "kwargs". Less confusion with everything else that is passed around as "kwargs".

[deeleeramone]

Add extra_params: Optional[Dict] = None to any POST method, rebuid, run commands or make API calls

I get an error when no kwargs are supplied to the POST request:

"'NoneType' object is not iterable"

[montezdesousa]

Add extra_params: Optional[Dict] = None to any POST method, rebuid, run commands or make API calls

I get an error when no kwargs are supplied to the POST request:

"'NoneType' object is not iterable"

thanks, _as_dict was buggy