Skip to content

Catch errors when reading serialized schemas also for collections #230

@MoritzPotthoffQC

Description

@MoritzPotthoffQC

If the format of serialized schema or collections change, previously serialized schemas might not be readable with new dataframely versions (e.g. primary_keys vs. primary_key). For schemas, this is already handled by catching errors in the dataframely.schema.deserialize_schema method if strict is False.

It would be nice if this could also be done in collections (dataframely.collection.collection._deserialize_types).

Additionally, in both cases, when these methods are called transitively via a scan_parquet call, the strict option should be set to False whenever the validation option allows that.

That would allow reading data that was serialized with a previous version (which then might require validation).

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions