JSON output reordered by pydantic @validator

[jaddison]

This isn't a big problem, but I'm wondering if it's possible to keep the JSON output key ordering unaffected by the presence of a Pydantic @validator(...)? I like consistency. 🤷‍♂️

I have this pydantic model structure:

class IngredientBase(BaseIngredientMixin):
    name: str = None
    unit: str = None
    unit_price: decimal.Decimal = None
    flavour_warning: bool = None

    @validator('unit')
    def valid_units_only(cls, v):
        choices = set(SingleIngredientTable.unit_choices.keys())
        if v and v not in choices:
            raise ValueError(f'must be one of {choices}')
        return v


class IngredientIn(IngredientBase):
    pass


class IngredientNew(IngredientIn):
    name: str = ...
    unit: str = ...
    unit_price: decimal.Decimal = ...


class IngredientOut(IngredientBase, TestModelMixin, TimestampModelMixin, IDModelMixin):
    pass

This will produce:

{
  "unit": "IU",
  "id": "sing_qOYSyqZhfOcJKHsfVz7tnfP6M",
  "created": 1571357369,
  "updated": 1571418480,
  "is_test": false,
  "name": "Vitamin C",
  "unit_price": 10.568536363534536,
  "flavour_warning": false
}

Without the validator, it produces what I would expect:

{
  "id": "sing_qOYSyqZhfOcJKHsfVz7tnfP6M",
  "created": 1571357369,
  "updated": 1571418480,
  "is_test": false,
  "name": "Vitamin C",
  "unit": "IU",
  "unit_price": 10.568536363534536,
  "flavour_warning": false
}

How can I continue to use @validators and maintain the expected JSON key ordering?

Edit: I can pull out the validator into a 'mixin' class that I apply ONLY to the IngredientIn and IngredientNew models (leaving IngredientOut untouched). But is that my only path here?

[jaddison]

@samuelcolvin, @dmontagu mentioned that I should pose this question to you? Might I ask you to weigh in? Thank you!

[samuelcolvin]

I don't see how validators could effect output order, but I could be wrong.

Could you create a minimal example of the problem and ask on pydantic?

There are numerous fields referenced here that aren't in the example, and there's a complicated inheritance landscape that makes working out what's happening very complicated.

[dmontagu]

@samuelcolvin if you look in BaseModel._iter, it is iterating over self.__dict__, which I believe is built up incrementally during the validation process. I thought having a pre=True validator could result in the key getting set earlier than it might otherwise, and then since dicts get iterated over in the order keys were added, that would explain the ordering change.

The example here doesn’t use pre=True but I figured something related might be going on.

[samuelcolvin]

Maybe I'm being dumb, but I don't think validators effect the order of __fields__.

[dmontagu]

@samuelcolvin Yep, looking through the code I don't see an obvious way this could happen. Thanks for checking.

@jaddison If you can produce a (small) self-contained reproducible example, I can look into this more.

[jaddison]

@dmontagu @samuelcolvin - thank you for looking into this.

It isn't (directly) pydantic-related - I replicated my model structure separately and could not reproduce. However...

Instead, it is related to the response_model parameter; it must force a different method of validation?

To be specific:

• if I remove the response_model parameter from the endpoint and return just an instance of that same pydantic model from the endpoint, the key ordering is as I would expect
• reintroducing the response_model param manifests the key mis-ordering

EDIT: To be clear, you still need to have the validator. However, the model inheritance is not a factor - I believe you can have a single model (no subclassing) with a field validator, and it ought to exhibit this behaviour still (I think)

[jaddison]

Tiny sample, demonstrating the issue:

from fastapi import FastAPI
from pydantic import BaseModel, validator

app = FastAPI()

class ValidatedModel(BaseModel):
    one: str = None
    two: str = None
    three: str = None

    @validator('two')
    def validate_two(cls, v):
        if v is '1':
            raise ValueError
        return v

# @app.get("/")
@app.get("/", response_model=ValidatedModel)
async def get():
    return {
        "one": "one",
        "two": "two",
        "three": "three"
    }

Note that I am on Python 3.7.4.

[dmontagu]

Confirmed the issue, and that the problem doesn't occur for pydantic.json() calls. It looks like this is related to the creation of the secured cloned model; if you drop the response_model argument to the decorator the order comes out the same with validator or without:

@app.get("/", response_model=None)
async def get():
    return ValidatedModel(**{
        "one": "one",
        "two": "two",
        "three": "three"
    })

I think this will be addressed by pydantic/pydantic#812 (which I'll finish up once pydantic v1 is released) and we can refactor fastapi to make use of that functionality. (That change should also result in a >=2x speedup of serialization when specifying a response_model, assuming you return a valid BaseModel instance.)

[dmontagu]

Yep, I found the problem -- it is in fastapi.utils:

def create_cloned_field(field: Field) -> Field:
    original_type = field.type_
    if is_dataclass(original_type) and hasattr(original_type, "__pydantic_model__"):
        original_type = original_type.__pydantic_model__  # type: ignore
    use_type = original_type
    if lenient_issubclass(original_type, BaseModel):
        original_type = cast(Type[BaseModel], original_type)
        use_type = create_model(
            original_type.__name__,
            __config__=original_type.__config__,
            __validators__=original_type.__validators__,  # type: ignore
        )
        for f in original_type.__fields__.values():
            use_type.__fields__[f.name] = f
    ...

It passes the validator fields first, then tacks on the non-validator fields. Actually, it overwrites them all, but because the use_type.__fields__ already has the validated fields as keys, they remain at the start of dictionary (because of how dictionaries are ordered in python 3.6+).

The PR I linked above should help do away with the cloned field, which would resolve this issue.

@jaddison It should be a relatively simple fix to get the field ordering right, and you could create a PR doing that if you really cared; that could probably be merged more quickly.