You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've made rowCount field in OutputStatisticsOutputDatasetFacet optional to address the case then there is no information about rows count, but only for files count.
Purpose:
This section gives the context of the proposal. It explains why this is needed.
Please describe the corresponding use cases.
Consider adding "fileCount" field to DataQualityMetricsInputDatasetFacet and OutputStatisticsOutputDatasetFacet:
For example, this allows to track Spark jobs which created many small files in S3 or HDFS. There is no need to store file names, only count.
Proposed implementation
This section describes how you propose to model it.
If you are you proposing a new facet, please mention its name and schema.
The text was updated successfully, but these errors were encountered: