The GTMF (Ground Truth Maturity Framework) aims to assist teams in improving the quality of their ground truth (GT) data. To do this, it provides a range of methodologies, metrics, and tools that allow users to measure and understand their GT data more effectively. Please find more details in this blog post.
In this repository, we introduce the GTMF library, a toolkit we developed for the measurement and improvement of ground truth data. The library covers multiple dimensions including representativity, accuracy, reliability, metric variance, and efficiency. It provides APIs for general metrics in each dimension and allows for customized parameters and flexible measurement granularity. It also allows teams to build their own workflows wrapping up the GTMF metric APIs.
Available APIs of each dimensions are included in the following files:
- Representativity: representativity.py. We leverage Balance(A Meta open sourced package for balancing biased data samples) to calculate the sample weights internally. It also covers several metrics that we include in GTMF library.
- Auccuracy: accuracy.py
- Reliability: reliability.py
- Aggregated Variance: metric_variance.py
- Cost Efficiency: efficiency.py
Jupyter Notebook Examples are under the folder jupyter_notebook_example
Installation Requirements You may find out the requirements here.
You may find out more about the license here.