You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added target_validations class, a subclass of hub_validations designed for target (truth) data validation results (#265).
New functions: new_target_validations() and as_target_validations()
combine.target_validations() method for concatenating target validation objects
Print method displays full file paths instead of basenames for better clarity when working with target data files
Added utilities for working with hive-partitioned data file paths:
extract_hive_partitions() for extracting key value pairs from paths to hive-partitioned data files.
is_hive_partitioned_path() for checking if a path is hive-partitioned.
Added read_target_file() function for reading in individual target data files.
Added target data utilities:
get_target_task_id() for extracting the name of the task ID(s) containing targets from the target metadata of a hub's config.
Added target data validation checks:
check_target_file_name(): that a hive-partitioned target data file name can be correctly parsed.
check_target_dataset_exists(): that a target dataset exists and can be detected. Required before running any other checks.
check_target_dataset_unique(): that a single unique target dataset exists for a given target type.
check_target_dataset_file_ext_unique(): that file(s) in a target dataset (e.g. time-series or oracle-output) share a single unique file extension.
check_target_dataset_rows_unique(): that each row in a target dataset is unique. Function designed to be used as part of overall target data integrity check.
check_target_file_ext_valid(): that the file extension of a single target data file is valid.
check_target_file_read(): that a single target data file can be read in without errors.
check_target_tbl_colnames(): that column names of a target data file match expected column names for the target type.
check_target_tbl_coltypes(): that column types of a target data file match expected column types for the target type.
check_target_tbl_values(): that values in a target data file match valid values/value combinations for model tasks specified n the tasks.json hub config file.
check_target_tbl_ts_targets(): Check that targets contained in a time-series target data file or implied through hub config are valid time-series targets.
check_target_tbl_output_type_ids(): Check that each observation (as defined by the observable unit) in an oracle-output target data file matches the expected output_type_ids.
check_target_tbl_rows_unique(): Check that each observation in a target data file is unique.
check_target_tbl_oracle_value(): Check that the oracle_value values in an oracle-output target data file for cdf and pmf output types conform to expectations. Specifically it verifies that oracle values are either 0 or 1, pmf oracle values sum to 1 for each observation unit and cdf oracle values are non-decreasing for each observation unit when sorted by the output_type_id set defined in the hub config.
Added validate_target_file() function for validating file level properties of a target data file (#250).
Added validate_target_dataset() function for validating dataset level properties of a target dataset (#229).
Added validate_target_data() function for validating the contents of a submitted target data file (#249).
Added validate_target_submission() function for validating a single target data file (#263).
Added validate_target_pr() function for validating all target data files in a pull request (#264).
Improved performance of check_tbl_values_required().