Provide typed objects using pydantic and json schema

[dazza-codes]

See

• https://pydantic-docs.helpmanual.io/usage/schema/
• Generate pydantic model python code from jsonschema pydantic/pydantic#643
• https://github.com/koxudaxi/datamodel-code-generator

This a different feature request for better typing support in botocore (and hence, boto3). A related issue that was auto-closed was at #1940

[kdaily]

Hi @dazza-codes, marking as a feature request.

[dazza-codes]

See also the example in cloudtools/troposphere#1879 e.g. this targets CloudFormation template schema, but the general idea is about creating python models with typing support using code generators from json schema and creating service instances by parsing service descriptions with these models.

pip install datamodel-code-generator[http]

wget https://schema.cloudformation.us-east-1.amazonaws.com/CloudformationSchema.zip
mkdir cfn_schemas
mv CloudformationSchema.zip cfn_schemas/
cd cfn_schemas/
unzip CloudformationSchema.zip 
cd ..
datamodel-codegen  --input cfn_schemas/aws-s3-bucket.json --input-file-type jsonschema --output aws_s3_bucket.py
cat aws_s3_bucket.py 

[ericbn]

aws-cdk uses attrs and cattrs (via jsii), so maybe it will be a better choice to use it here too.

@kdaily, would you accept a PR with this change? And in the positive case, how do you think it's the best way to implement it while keeping complete backwards compatibility? The first idea that comes to my mind is making all classes be subclasses of a Base that implements dict, e.g.:

class Base(dict):
    def __setitem__(self, key, item):
        self.__dict__[key] = item

    def __getitem__(self, key):
        return self.__dict__[key]
        
    # ... and all other dict methods. See https://stackoverflow.com/a/23976949/2654518


@attr.define(slots=False)
class APIGatewayModel(Base):
  model_id: str
  name: str
  description: str
  schema: str
  content_type: str


# both api_gateway_model.model_id and api_gateway_model['model_id'] work,
# given api_gateway_model is an instance of APIGatewayModel.

[jmgreg31]

I had a similar want and in thinking how to scale this out with all the exiting resources teams have developed, started a rough local project leveraging mypy_boto3 type defs to create pydantic models. I'm currently constraining the mypy_boto3 submodules, but this theoretically can scale to all type definitions defined. There's still a lot to consider, but works for the general purpose I had intended. Thought it might be somewhat related to this thread, and came up when I was orignially searching for existing solutions.

https://github.com/jmgreg31/mypydantic
example cloudformation model

[ericbn]

To solve the type checking, there's a solution for this already in https://pypi.org/project/boto3-stubs and the boto organization is working on https://github.com/boto/botostubs

These only solve for checking with type checkers like mypy, not for checking during runtime. Maybe adding the runtime check as proposed here can be dropped in favor of the type checking.

[jmgreg31]

Thanks @ericbn

The driver was more around model manipulation than checking. stubs look to rely primarily on TypedDict. I wanted to be able to access request/response key/values like attributes using dot notation and handling null definitions. Maybe not something common folks are looking to do, or a there's a usage pattern of stubs I was just missing.

from mypy_boto3_wafv2.client import WAFV2Client
client: WAFV2Client = boto3.client("wafv2")

# CaptchaConfig is optional, so this can theoretically throw an error
get_web_acl_response = client.get_web_acl(Name="example", Scope="REGIONAL", Id="1234")
get_web_acl_response["WebACL"]["CaptchaConfig"]

# Dot Notation and CaptchaConfig can be None
from mypydantic.models.wafv2 import GetWebACLResponse
get_web_acl_response: GetWebACLResponse = GetWebACLResponse(
    **client.get_web_acl(Name="example", Scope="REGIONAL", Id="1234")
)
get_web_acl_response.web_acl.captcha_config

[ericbn]

Right @jmgreg31, mypy won't complain if you access an item from a non-total TypedDict with []:

You may need to use get() to access items of a partial (non-total) TypedDict, since indexing using [] could fail at runtime. However, mypy still lets use [] with a partial TypedDict – you just need to be careful with it, as it could result in a KeyError. Requiring get() everywhere would be too cumbersome.

(From https://mypy.readthedocs.io/en/stable/typed_dict.html#totality)

Maybe you could consider asking for the feature in mypy: a configurable option that enables forcing the get() in this scenario.

Other scenarios should be covered, like missing a key from a total TypedDict when creating a new instance of it.

As for the dot notation, IMO it might be overkill to implement a new feature just to enable that.

[ericbn]

A cool feature of https://pypi.org/project/boto3-stubs is that you can use it as a dev dependency only and deploy your code without it. Just install it during the linting phase. This works if you use boto3-stubs instead of boto3-stubs-lite, i.e. if you use implicit type annotations. Then your code can simply be:

import boto3

client = boto3.client("wafv2")

get_web_acl_response = client.get_web_acl(Name="example", Scope="REGIONAL", Id="1234")
get_web_acl_response["WebACL"]["CaptchaConfig"]

[jmgreg31]

Really appreciate the additional info @ericbn ! I was originally using the get() method but it just didn’t provide the same editor completion capabilities I wanted to achieve. I completely recognize that it’s something almost certainly out of scope as a niche capability. Just wanted to throw in an idea I had for folks looking to create pydantic models around boto3 services. It was great to dig into the core libraries and learn a lot from folks far smarter than me !

[OriPoria]

You can use this package:
https://github.com/OriPoria/boto-model-py