data and metadata tables advice

[joamatab]

Thank you Josh for your work with pydbantic,

I want to store both data and metadata for my data

How do you recommend creating tables that can store pandas dataframes with arbitrary columns?

How about metadata where the metadata has some JSON fields?

[codemation]

@joamatab thanks for opening an issue here for these questions.

• Q - How do you recommend creating tables that can store pandas dataframes with arbitrary columns?
• A - DataBaseModel s are a sub class of pydantic BaseModel and thus share the same constraints that you might face when creating a model. Fortunately custom classes and subsequently pandas DataFrame type objects can be used combined with the Config class attribute and arbitrary_types_allowed = True as documented here

from pandas import DataFrame
from pydbantic import DataBaseModel, PrimaryKey

class MyModel(DataBaseModel):
    id: str = PrimaryKey()
    frame: DataFrame

    class Config:
        arbitrary_types_allowed = True

In future releases I am considering applying a default arbitrary_types_allowed to True, to avoid the need to set this yourself. Since DataFrame type objects are serializable via pickle, they will be stored as bytes in the database within a Binary Field in the database.

• Q: How about metadata where the metadata has some JSON fields?
• A : If you expect the metadata fields to change or vary, then set the field type to dict, if it will always follow a set schema, consider creating a BaseModel for the metadata, both will be stored on Binary database fields ( of course de-serialized / serialized for you).

[codemation]

@joamatab - anything else on this issue?