ABCRaster stands for Accuracy assessment of Binary Classified Raster. It is a package for performing validation, accuracy assessment, or comparing classification results (.tif) versus a reference (.shp, .tif) e.g. CEMS. Can be used to assess other binary classification (presence/absence) maps. Computes accuracy assessment metrics e.g. User, Producer’s accuracy, Kappa, etc. Also creates ‘confusion map’ with pixels marked as TP, TN, FP, and FN.
- reference shapefile can be in any projection (built-in reprojection and rasterization)
- (stratified) random sampling support (*based on reference file)
- applying raster or vector masks
- creates confusion (difference) tiff file
First, a conda environment containing GDAL needs to be created. Here, we suggest to use mamba for the sake of speed:
mamba create -n abcraster -c conda-forge python=3.11 gdal geopandas
conda activate abcraster
The package itself can be installed by pip (from source or a repository):
pip install abcraster
The abcraster.base
module provides the Validation
class, which carries the main functionality as
dedicated methods. One can build a self-defined validation workflow by importing the class and calling
the needed methods. An example of a self-defined workflow is given here:
# initialize validation object
val = Validation(input_data_filepath=input_path, ref_data_filepath=ref_path, out_dirpath=out_dirpath)
val.apply_mask(aoi_path, invert_mask=True) # apply an area-of-interest (.tif or .shp)
val.apply_mask(mask_path) # apply a general mask (.tif or .shp)
val.accuracy_assessment() # calculate confusion matrix/map
val.write_confusion_map(os.path.join(out_dirpath, 'val_diff.tif')) # write confusion map to file
print(val.calculate_accuracy_metric(critical_success_index)) # print the CSI value
The calculate_accuracy_metric
method takes in all predefined functions of the abcraster.metrics
module,
but allows for self-written function as well. The function will receive a dictionary representing the confusion
matrix and cotnaining the values for the keys: 'TP', 'TN', 'FP' and 'FN'.
An already pre-defined workflow can be utilized in a Python script when using the run
function of the
abcraster.base
module. An example of a call of the run
function is given here:
run(input_data_filepaths=[input_path], ref_data_filepath=ref_path, out_dirpath=out_dirpath, metrics_list=['CSI', 'OA'],
samples_filepath=os.path.join(out_dirpath, 'sampling.tif'), sampling=(200, 200))
The same pre-defined worklfow can be called through the command line by:
python -m abcraster.base
or
abcraster
Furhter details can be defined using the following arguments:
-in
or --input_filepath
-- Full file path to the binary raster data 1= presence, 0=absennce, for now 255=nodata.
-ex
or --exclusion_filepath
-- Full file path to the binary exclusion data 1=exclude,
for now 255=nodata
-ref
or --reference_file
-- Full file path to the validation shapefile (.tif or .shp, in any projection)
-out
or --output_raster
-- Full file path to the final difference raster
-csv
or --output_csv
-- Full file path to the csv results (optional!)
-del
or --delete_tmp
-- Option to delete temporary files (optional!)
-ns
or --num_samples
-- Number of total samples if sampling will be applied (optional!)
-stf
or --stratify
-- Stratification flag (no input required) based on reference data (optional!)
-nst
or --no_stratify
-- No stratification flag option (optional!)
-sfp
or --samples_filepath
-- Full file path to the sampling raster dataset (.tif ), if num samples not specified,
assumes samples will be read from this path (optional!)
-all
or --all_metrics
-- Flag to indicate to compute all metrics, Default true. (optional!)
-na
or --not_all_metrics
-- Flag to indicate not to compute all metrics,
metrics should be specified if activated. (optional!)
-mts
or --metrics
-- Optional list of metrics (keys) to run e.g. OA, UA, K. See metrics in ( )
above list.
All metrics are based on the confusion matrix of all the pixels that are within the common extent between a reprojected and rasterized version of the shapefile, less excluded pixels (exclusion tiff file), if present and nodata values (currently assumed to be 255).
TP - True Positive, FP - False Positive, TN - True Negative, and FP - False Negative
Overall accuracy (OA) is computed as follows:
Cohen's Kappa Coefficient (K) is computed from:
where
User's Accuracy (UA) or Precision is computed by:
Producer's Accuracy (PA) or Recall is computed by:
Critical Success Index (CSI) is computed by:
F1 Score (F1) is computed by:
Penalization function is computed by:
Success Rate (SR) is computed by:
Bias (B) is computed by:
Prevalence (P) is computed by:
True negative rate (TNR) is computed by:
False positive rate (FPR) is computed by:
Negative predictive value (NPV) is computed by:
False omission rate (FOR) is computed by:
Module added for random and stratified sampling methods. Sampling module includes stand-alone CLI for creating raster encoded samples. Optional to enable sampling in Accuracy assessment workflow either by providing a preselected samples raster or number of samples e.g. int for class independent sampling or an iterable for (reference) class defined values e.g. [n, m] where n and m are int.