-
Notifications
You must be signed in to change notification settings - Fork 3
User Defined Functions
One of the key features of StatisticsTool is its ability to work with user-defined functions. These functions allow users to customize the tool to their specific project characteristics, making it a highly flexible and adaptable solution.
By allowing users to define these functions, StatisticsTool can be tailored to a wide range of use cases and applications. Whether you're working on a computer vision project, a natural language processing project, or any other type of machine learning project, StatisticsTool can be customized to meet your specific needs.
User-defined functions (UDFs) in reports can have parameters that are set by the user who creates the report. This can be done either through the Web UI or by manipulating report's configuration file.
To add parameters to a UDF, the programmer can add an optional function to the function's file called "get_function_arguments()". This function allows the programmer to specify the number and names of the parameters. For example, the function can return a dictionary with the parameter names and their types, like this: {"variable1": "string", "variable2": "string"}.
In the example provided in the repo unter example's statistics tool external library, there is for each user defined function a readme.txt file, and there is also an example for each of the user defined functions.
In the Report Creation Phase, StatisticsTool offers several user-defined functions that can be implemented to customize the tool for specific project characteristics. These functions include:
This function is used to parse the prediction log files and the ground truth annotations and return them as a dataframe. The reading function is called twice, once for the predictions and once for the ground truth. The reading function should mark each prediction/annotation with a key, and in the association function, it tries to associate between all the predictions and annotations with the same key. If we have few detections in the same prediction (e.g. bounding boxes in frame) they all should be with the same key same for the annotations related to same prediction (frame).
In addition, any other information that will be added in reading phase will be passed through the rest of the evaluation pipeline and can be used in other user defined functions.
There are some special columns that can be added in reading function and the Statistics Tool can use them (The full list in the repo unser DataFrameTokens type):
LABELS_GROUP_KEY = 'frame_id' - this is the key to all related detections.
VIDEO_TOKEN = "video" - contains the information about the related video file or images folder. This value is filled by the tool if it is not provided in the reading function.
The Association function in StatisticsTool is used to match different results in the same inference. Once the predictions and ground truth annotations have been read into dataframes, the Association function is called to associate them based on the unique keys assigned in the Reading function. The function should be designed to handle cases where there are multiple predictions and annotations with the same key, but it is optional for cases where there is only one prediction/annotation for each key. The Association function should match the predictions and annotations based on the key and create a new dataframe with the matched pairs. The matched pairs can be used in the next phase to generate a report of the algorithm's performance. The Association function can be customized to handle different types of data and association methods depending on the specific project requirements.
This function is an optional function that the user can use to reshape the dataframe with the data. After the association function processes the data, StatisticsTool creates one dataframe with the predictions and annotations that are associated with each other in the same line. In the transform function, the user can reshape the dataframe as needed. For example, they can group predictions/annotations into events if they need to evaluate algorithm behavior in events.
In the Report viewing phase of StatisticsTool, there are several user-defined functions available. These functions allow users to customize the report generated by StatisticsTool to suit their specific needs. The user-defined functions available in the Report viewing phase are:
- Partitioning function
- Statistics function
- Confusion function
The Partitioning function is used to partition the data in the report created in the previous phase. The user can specify which rows belong to which partition by returning a dictionary of dictionaries of masks. This allows for the creation of multiple partitions. The function takes the dataframe from the previous phase with all the data that was added in the reading function. With this data, the user can return a dictionary of dictionary of masks that each mask should point to the rows in the dataframe that belongs to the partition with masks keys. Because it is a dictionary of dictionary we can have multiple partitions.
The Statistics function is used to calculate some statistics about the accuracy of all the samples. It may be some partitioned sample or all the sample. The function gets masks with the same keys that were provided in the confusion function by the user-defined function and the masks may be only partitioned data. The function should return a dictionary with statistics results as the user wants. For example, precision/recall.
The Confusion function is used to calculate the accuracy of the algorithm's predictions for each row in the partitioned data. The function marks each row as TP, FP, FN, or TN and returns a dictionary with keys and masks that the user can choose. These keys and masks are then presented in the report. The function input is the dataframe and the output is dictionary with key and mask the user can choose the keys and they will be presented in the report. The user can choose what values to use to calculate accuracy and what keys and masks to use for the output dictionary.