Our work towards a Structured Evaluation Methodology for Artificial Intelligence Technology (SEMAIT) aims to provide plots, tools, methods, and strategies to extract insights out of various machine learning (ML) and Artificial Intelligence (AI) data.
Included in this software is the MIg analyZeR (mizr) R software package that produces various plots. It was initially developed within the Multimodal Information Group (MIG) at the National Institute of Standards and Technology (NIST).
A quickstart is below. Full documentations (docs and vignettes) are available pre-build at mizr/docs/index.html
For ease of updating, all of the paths and inputs required are at the top of each R or Rmd file.
The R code to run the SEMAIT analysis on a sample dataset
is provided in scripts_semait/semait_analysis_sdms_script_code.R of this repository. A documented R Markdown of the same script is available as the vignette semait_analysis_sdms_documented.Rmd
Please see our SEMAIT paper for more information (paper coming soon), and please cite our paper with the citation (citation coming soon)
This version of mizr is currently built and tested on version 4.2.2 of R with ggplot 3.4.1 and will work on any OS and configuration that supports R 4.2.2 and ggplot 3.4.1. It has been tested both on Mac OS and Linux. As it has been partially implemented on earlier versions of R and ggplot, some functions may have depreciation warnings.
The main mizr plotting methods are named mizr_YYY_plot().
Every mizr_YYY_plot() plot method produces a plot as a ggplot object and returns that ggplot object. The benefit of this design decision is that many changes (including theme changes) can be customized by altering or augmenting the returned ggplot object (such as <plot_object> + theme(...)) without having to examine the mizr method source. A second design feature is that every ``mizr_YYY_plot()is designed to call two sub-methods in sequence: Azzz_compute_df()` method that computes the data frame; and a `zzz_produce_plot()` method that takes that computed dataframe and renders the plot. This design feature provides a way to get the raw data frame of any plot's data. More details are in the MIg analyZeR (mizr) Plot Library Requirements vignette.
Please contact mizr@nist.gov with questions, comments, feedback, or issues.
Contributors to this code repository:
- Peter Fontana (NIST)
Here is a quick start to get up and running with the mizr software package.
If using a terminal, mizr can be installed by going into the root directory of this repository and running
R CMD INSTALL --no-multiarch --with-keep.source mizrAnother way to install it is via RStudio. In RStudio, make a new project (or open a project) within the package folder mizr. That means that it is the <repository_directory>/mizr directory. After making a project, use the menu Build -> Install and Restart option. This will install the package mizr.
This installs the minimum packages to run mizr. To run the test suite and documentation code, you will need additional packages. To install these packages, run in a shell
Rscript -e 'install.packages(c("covr", "DT", "devtools", "foreign", "htmltools", "knitr", "lintr", "pkgdown", "purrr", "rmarkdown", "roxygen2", "styler", "testthat", "tools"),dependencies=TRUE,repos="http://cran.rstudio.com", quiet=TRUE)'or in an R console
install.packages(c("covr", "DT", "devtools", "foreign", "htmltools", "knitr", "lintr", "pkgdown", "purrr", "rmarkdown", "roxygen2", "styler", "testthat", "tools"),dependencies=TRUE,repos="http://cran.rstudio.com", quiet=TRUE)First, load the R package in an R environment with
library(mizr)Then, if you wish to run a few plots to check, you can use one of the examples in the data directory. The code below loads an example set, processes it, and runs a few plots. To run this example, please change data_dir to be the location of the data directory of this repository. In this case the data dir is stored in a shell environment variable MIG_ANALYZER_DATA_DIR, Although this example retrieves the directory from an environment variable, a direct path will work.
data_dir <- Sys.getenv("MIG_ANALYZER_DATA_DIR")Then load the supporting libraries
Then run the code below
library(mizr)
data_dir <- Sys.getenv("MIG_ANALYZER_DATA_DIR")
old_theme <- mizr_set_theme()
sdms_fpath <- normalizePath(file.path(data_dir, "raw", "semait_sdms_1.csv"), mustWork = TRUE)
sdms_df <- read.csv(sdms_fpath, stringsAsFactors = TRUE)
mizr_tile_table_count_plot(sdms_df, c("dataset", "metric"), c("system"))
mizr_block_plot(sdms_df, c("dataset", "metric"), "system", "score") %>%
eh_title("Block Plot of Systems over different Datasets and Metrics")It takes the sdms, which is a data frame in experimental data format, loads it, the accompanying metadata file, and calls a selection of the plotting methods. This gives a quick check of the installation. The next subsection gives a quick view of some additional plots. The output of the Installation Check is provided in the Vignette MIg analyZeR (mizr) README Rendering.
This block of code takes the sdms_df, which is a data frame in experimental data format, loads it, and provides a few plots. This gives an overview of some of the plotting functions currently implemented. The output of the Quick Run is provided in the Vignette MIg analyZeR (mizr) README Rendering.
This code uses the previously assigned directory data_dir. O
library(ggplot2)
library(dplyr)
library(mizr)
data_dir <- Sys.getenv("MIG_ANALYZER_DATA_DIR")
old_theme <- mizr_set_theme()
sdms_fpath <- normalizePath(file.path(data_dir, "raw", "semait_sdms_1.csv"), mustWork = TRUE)
sdms_df <- read.csv(sdms_fpath, stringsAsFactors = TRUE)
mizr_tile_table_count_plot(sdms_df, c("dataset", "metric"), c("system"))
# As mizr_..._plot() methods return ggplot objects, we can augment them like ggplots.
mizr_histogram_plot(sdms_df, "score", binwidth = 0.05)
mizr_histogram_plot(sdms_df, "score", binwidth = 0.05) +
coord_cartesian(xlim = c(0, 1), clip = "off")
# We can also pipe them to some of our mizr methods in enhance_plots.R
mizr_block_plot(sdms_df, c("dataset", "metric"), "system", "score") %>%
eh_title("Block Plot of Systems over different \nDatasets and Metrics")
mizr_tile_table_plot(sdms_df, c("dataset", "metric"), "system", "score", use_margins = TRUE)
mizr_box_plot(sdms_df, c("system"), "score")
mizr_order_plot(sdms_df, c("dataset", "metric", "system"), "score")
mizr_scatter_doe_plot(sdms_df, "system", "score", doe_plot_style = "enhanced")To make fully-worked through examples more accessible with the package documentation, we have placed the R Markdown source scripts in mizr/vignettes, calling these examples vignettes. The making and placing of these vignettes allows these complete examples to be bundled with the package documentation and can be referred to in the "Articles" menu of the rendered documentation.
Rendered vignettes can be found in the "Articles" section of the rendered documentation displayable in the repo at mizr/docs/index.html. The version rendered is the version on master branch, and the package has its version number. Included is the vignette which provides the SEMAIT code to run the software as well as a documented analysis on that code for an example data.
The example data is a self-constructed set of tuples where we ran baseline machine learning systems on publicly-available datasets scored on a variety of metrics. The scripts to produce this data are in the scripts_data folder for those who wish to have the source code for this data (code is a combination of bash and R scripts).
Rendered vignettes can be produced with the R command:
devtools::build_vignettes()
and those rendered vignettes (and html files) will be in /mizr/inst/docs.
The package mizr has a custom ggplot theme that is used. Calling mizr_set_theme() will set the
mizr theme as the default them in the document for all plots. To use that theme, run
old_theme <- mizr_set_theme()
Otherwise your default ggplot theme will be used to produce the mizr plots.
All of the main plot methods start with mizr and can be found in the file mizr_plots.R. By
design, each mizr_method is split into two submethods: one that produces the computed data frame
and one that produces the plots. This way, if you wish to get the data frame with the numbers
of any plot, simply run the first submethod that calls the data frame.
There is a test suite each inside test directories inside the folders.
To test, please set the environment variable MIG_ANALYZER_DATA_DIR to point to the location of the data in this repository. For testing and some production, this is the data directory of your clone of this repository. This environment variable will be loaded by the R test suite to find the test data. To load that environment value in R, add it to your .Renviron file, typically found in ~/.Renviron
To run all of the tests, run the file
./test_R_packages.sh
in the root directory. This will run the R tests and coverage metrics. The commands that are being run to do the testing and coverage are below for those that do not wish to run a shell script.
Currently the tests are only a stub. However, to run the R current tests, go into the ./mizr directory and run in an terminal
Rscript -e 'library(testthat);devtools::test(reporter="progress")'
If one wishes to test it in an R terminal, one can use this
library(testthat)
devtools::test(reporter="progress")
to test the package, or the command below to test
test_dir("./tests/testthat", reporter="progress")
the directory of tests.
To get code coverage, go into the ./mizr directory and run in an terminal
Rscript -e 'library(covr);package_coverage(combined_types=FALSE)'
In R, the commands are
library(covr)
package_coverage(combined_types=FALSE)
A report with
library(covr)
report()
gives an interactive report for in-depth coverage and can be saved with
library(covr)
report(package_coverage(combined_types=FALSE),file="R_coverage_report.html",browse=FALSE)
This package conforms to the lintr package, excluding the documented vignettes (because knitr produces blocks of commented code, which fail the lintr). To run the code checks, within the mizr directory, in an R terminal run
Rscript -e "library(lintr);lintr::lint_package()"
This lintr uses the configurations defined in mizr/.lintr
This code conforms to the automated code styling package styler. To run the styler, run
Rscript -e "library(styler);styler:::style_pkg()"
Package documentation is partially complete as documentation is in progress. However, there is a tutorial vignette available.
R documentation is generated to .Rd files through roxygen2 with,
devtools::document()
and is converted to an .html page with
library(pkgdown)
pkgdown::build_site()
To embed the README.md into the home page of the site, copy the README.md to the mizr directory with the command below and then update the site links.
cp README.md mizr/README.md
Since the vignettes require the mizr package to be installed, please install the package prior to
building the vignettes. To build the vignettes, run in R.
devtools::build_vignettes()
.Rd files are in /mizr/man and index.html is in /mizr/docs, and vignettes will be in /mizr/inst/docs which will have the html files.
The license is documented in the LICENSE file and on the NIST website.
Certain equipment, instruments, software, or materials are identified in this paper in order to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement of any product or service by NIST, nor is it intended to imply that the materials or equipment identified are necessarily the best available for the purpose.