No CRUD generation support for ARRAY types.

[gmos]

At version 0.29.0 there seems to be no support for ARRAY types in the CRUD generation.
Code is generated, but an ARRAY of VARCHAR comes out as a single Varchar, and the field cannot be properly used.

In the Piccolo Admin the Arrays of Varchar are handled ok. Are there already plans to support Array types in CRUD?

[dantownsend]

@gmos Thanks for reporting this.

PiccoloCRUD should support arrays - there could be a bug somewhere.

In the example below, the years_nominated field is an array. The API should render it like a list.

Is the problem when trying to post a new value back?

[sinisaos]

@gmos @dantownsend POST also work as expected.

[dantownsend]

@sinisaos Thanks for checking that.

[gmos]

Screenshots are from the swagger .../docs page. I expanded the root GET.

Parameters docs:

Here it goes wrong. No array structure to be found. The sensors field is depicted as a simple Varchar.

And this is when I make the actual GET call:

As you can see the response schema is good. the issue is about the generated parameter input for the root GET.

[dantownsend]

@gmos Cool, thanks - that's really useful. We'll investigate.

[gmos]

Here is the column definition from the Table subclass:

    sensors = Array(
        base_column=Varchar(
            length=16,
            null=False,
        )
    )

And here the automatic creation of the lot.

hoses_router = APIRouter()
hoses_crud = PiccoloCRUD(TempHoses, read_only=False)

FastAPIWrapper(
    "/hoses",
    fastapi_app=cast(FastAPI, hoses_router),  # TODO Get away with the cast
    piccolo_crud=hoses_crud,
    fastapi_kwargs=FastAPIKwargs(
        all_routes={"tags": ["TempHoses"]},
    ),
)

Side question: why do I need a cast to get rid of the type warning? AM I doing something wrong assigning a PiccoloCRUD instance to the FastAPIwrapper?

[dantownsend]

@gmos The type error might be because you're using FastAPIWrapper with an APIRouter instance, rather than a FastAPI instance.

We probably just need to update this type annotation to be t.Union[FastAPI, APIRouter]:

piccolo_api/piccolo_api/fastapi/endpoints.py

Line 94 in e4eb74c

fastapi_app: FastAPI,

[sinisaos]

@dantownsend Problem is when you use openapi request url is http://localhost:8000/posts/?tags=tag (which raise Pydantic ValidationError value is not a valid list (type=type_error.list)) and it should be http://localhost:8000/posts/?tags[]=tag like in Piccolo Admin or if you pass request url directly in browser to get correct result.

[dantownsend]

@sinisaos Well remembered - I forgot about that.

I wonder if we can add a custom validator to the Pydantic model:

https://pydantic-docs.helpmanual.io/usage/validators/

[sinisaos]

@dantownsend I think I found a problem in openapi filters. We need to use ModeField.outer_type_

piccolo_api/piccolo_api/fastapi/endpoints.py

Line 401 in 8b66ff3

type_ = _field.type_

because ModeField.type_ shows the wrong type (class int or str insted List[int] or List[str]).
We need to change the line 401 to this type_ = _field.outer_type_ and the result is.

Then I add an extra check in crud endpoints _parse_params method for the array columns so that openapi request url accepts multiple query params without square brackets. Method now looks like this

def _parse_params(self, params: QueryParams) -> t.Dict[str, t.Any]:
  params_map: t.Dict[str, t.Any] = {
      i[0]: [j[1] for j in i[1]]
      for i in itertools.groupby(params.multi_items(), lambda x: x[0])
  }

  array_columns = [
      i._meta.name
      for i in self.table._meta.columns
      if i.value_type == list
  ]

  output = {}

  for key, value in params_map.items():
      if key.endswith("[]") or key.rstrip("[]") in array_columns:
          # Is either an array, or multiple values have been passed in
          # for another field.
          key = key.rstrip("[]")
      elif len(value) == 1:
          value = value[0]

      output[key] = value

  return output

And result is

Piccolo Admin and request directly in browser remain unchanged.

@gmos Can you please check this on your local Piccolo Api installation and confirm that everything works.?

[dantownsend]

@sinisaos That's some good investigative work - thanks.

When we make this change we'll also have to update Piccolo Admin at the same time.

[dantownsend]

@sinisaos Ah, just realised you managed to make it compatible with the current Piccolo Admin, great.

[sinisaos]

@dantownsend Yes, Piccolo Admin works. If you want I can do PR.

[dantownsend]

@sinisaos That would be great, thanks

[dantownsend]

I've released it to PyPI now - version 0.29.2.

Seems to fix the problem. Thanks both.