[MAINTENANCE] Fluent Datasources - don't register "private" Datasource classes #7124
Conversation
✅ Deploy Preview for niobium-lead-7998 ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
👇 Click on the image for a new way to code review
Legend |
| print(f"✅ {name} - {schema_path.name} schema updated") | ||
| print(f"🔃 {name} - {schema_path.name} schema updated") |
There was a problem hiding this comment.
Changed the symbol here so it's easier to tell at-a-glance if schemas have changed
| if issubclass(model, Datasource): | ||
| print(f"🙈 {name} - is a Datasource; skipping") | ||
| continue |
There was a problem hiding this comment.
Stop skipping Datasource schema generation
| @@ -80,7 +79,6 @@ class _PandasDatasource(Datasource): | |||
| asset_types: ClassVar[List[Type[DataAsset]]] = list(_ASSET_MODELS.values()) | |||
|
|
|||
| # instance attributes | |||
| type: str = pydantic.Field("_pandas") | |||
There was a problem hiding this comment.
This field serves no purpose because this class is not meant to be instantiated. Was only defined to prevent a crash during Metadatasource.__new__ registration. Now that Metadatasource skips registration for _Datasources we can remove it.
Kilo59
changed the title
| @@ -49,7 +48,6 @@ class _SparkDatasource(Datasource): | |||
| asset_types: ClassVar[List[Type[DataAsset]]] = [CSVSparkAsset] | |||
|
|
|||
| # instance attributes | |||
| type: str = pydantic.Field("_spark") | |||
There was a problem hiding this comment.
@Kilo59 I may need some help next week figuring out the right way to create SparkFilesystemDatasource, SparkS3Datasource, and so on in terms of the type field as well as the Schema file naming and generation. Thank you.
There was a problem hiding this comment.
I would think this should just be 'spark_file' or 'spark_filesystem' and 'spark_s3'.
| @@ -22,10 +22,16 @@ | |||
| "additionalProperties": { | |||
| "$ref": "#/definitions/CSVSparkAsset" | |||
| } | |||
| }, | |||
There was a problem hiding this comment.
@Kilo59 I may need some help next week figuring out the right way to create SparkFilesystemDatasource, SparkS3Datasource, and so on in terms of the type field as well as the Schema file naming and generation. Most likely, this particular file will have to attain a different name (e.g., SparkFilesystemDatasource.json). Thank you.
Changes proposed in this pull request:
Datasourceclasses (ex_PandasDatasource)invoke schema --syncto be more transparent about when and why schemas are updated (or not updated)Datasourceschemas in addition toDataAssetschemasDefinition of Done