Make meta verify objects are JSON serialisable before insertion.

[0Hughman0]

If you insert non JSON data into a meta dict, everything freaks out because whenever meta tries to write to disk, it can't.

We should verify something is serialisable prior to insertion.

[0Hughman0]

This should be done using pydantic.

I think it should be possible to enforce errors for incorrect types in meta, although I can see this becoming a bit of a mess.

[0Hughman0]

Type hints introduced in #53 should help a bit. Hints suggest values must be JSONType

Cassini/cassini/accessors.py

Line 27 in 7116ee0

JSONType = Union[dict, list, tuple, JSONPrimative]

[0Hughman0]

Defining an OpenAPI schema and then generating a pydantic model looks like the best approach:

https://koxudaxi.github.io/datamodel-code-generator/

Then I can use similar tools on the gui side to generate schema for the front-end, which is important because it also deals with meta files! (Possibly should have full control).

Finding the best way to integrate the pydantic model with the Meta implementation may be a challenge.

[0Hughman0]

Adding model validate checks might work:

https://docs.pydantic.dev/latest/api/base_model/#pydantic.BaseModel.model_validate

I don't think we want to be creating millions of instances of the model. The models are kinda designed to be wrappers around dictionaries... I guess I could make _cache a Model? Seems Models are mutable by default:

https://docs.pydantic.dev/latest/concepts/models/#faux-immutability

Then for refreshing can dump:

https://docs.pydantic.dev/latest/api/base_model/#pydantic.BaseModel.model_dump_json

[0Hughman0]

... this could be incredibly easy! 🤞

[0Hughman0]

This maybe needs a little more thought...

We are encouraging users to insert whatever they like into the meta, which will have all sorts of types.

How do we assure those types?

The hope was also that these types could be communicated to the front end for validation, how can this happen if they are dynamic?

One option is to set up MetaAttr so that these are strictly typed, but then attributes defined outside of this are not.

The only problem with that is that we have this slightly messy thing that things you might want to store in meta are not universal across all instances of the same tier. For example, some Samples might require temperature information, and others concentrations.

[0Hughman0]

Ok I think here's the best way...

So, it looks like models can generate schema of themselves:

https://docs.pydantic.dev/latest/api/base_model/#pydantic.BaseModel.model_dump_json

And we can allow arbitrary attributes (I presume they must be json serialisable, otherwise dump won't work!)

https://docs.pydantic.dev/latest/concepts/models/#extra-fields

For 'core' attributes, we need to ensure they're consistent. I think the best way to do this is using the MetaAttr idea above i.e. generate some parts of the schema from that. This could be done during __initsubclass__. Doing so would make setting these attributes strict Python side. Users could add additional MetaAttr as they wish to benefit from that additional type safety.

For 'extra' attributes, I am fairly sure Pydantic will make sure they're serialisable, which was the original concern.

Then, at the server side for the front end, we can provide a schema using model_dump_json or something. That way we get type safety for the front end, which is what we wanted.

[0Hughman0]

We could even do something really fancy to find the schema of tiers with the same parents and kinda combine them together to ensure consistency... this might be too magic... but equally, we are doing the same thing at the front end!

[0Hughman0]

I think let's not do this, but instead in 0.4.x if we can do #116, we can allow per sibling setting of the form of Meta using specialised MetaAttr.

[0Hughman0]

The current, somewhat convoluted implementation involving the MetaManagers, which is frankly kinda confusing...

anyway, one thing is that we pass fields to Meta and then we allow Meta to build it's model class. Perhaps we should get MetaManager to build the ModelClass and pass that directly into the Meta object.

I think this makes a lot more sense and makes it explicit that Meta of the same Tier Class should have the same model.

This will be helpful for later when we want to serve up the models for meta.