Skip to content
A tool to help improve data quality standards in observational data science.
JavaScript R HTML TSQL Other
Branch: master
Clone or download
Latest commit dba5fe9 Oct 4, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
R Fixes issue #55 Sep 30, 2019
docs Added missing step for npm web server deployment Sep 16, 2019
extras Added screenshot Aug 23, 2019
inst Fixes #51 by using absolute value when checking numeric datatypes Sep 17, 2019
man Added tablesToExclude parameter to allow skipping of tables if known … Aug 28, 2019
tests Added missing dbType in test Aug 23, 2019
.Rbuildignore updated R package wrappers Jul 23, 2019
.gitignore Fixes #30, making autocommit setting specific to connectionDetails db… Aug 23, 2019
.travis.yml Added devtools to travis. Added travis and codecov statuses to readme… Aug 23, 2019
DESCRIPTION Added pkgdown documentation. Added one testthat test. Aug 23, 2019
DataQualityDashboard.Rproj updated R package wrappers Jul 23, 2019
LICENSE Initial commit Jun 25, 2019
NAMESPACE
README.md Update README.md Oct 4, 2019
_pkgdown.yml Added pkgdown documentation. Added one testthat test. Aug 23, 2019
compare_versions Added pkgdown documentation. Added one testthat test. Aug 23, 2019

README.md

DataQualityDashboard

Build Status codecov.io

The DataQualityDashboard is a tool to help improve data quality standards in observational data science.

Introduction

An R package for characterizing the data quality of a person-level data source that has been converted into the OMOP CDM 5.3.1 format.

Features

  • Utilizes configurable data check thresholds
  • Analyzes data in the OMOP Common Data Model format for all data checks
  • Produces a set of data check results with supplemental investigation assets.

Technology

DataQualityDashboard is an R package

System Requirements

Requires R (version 3.2.2 or higher). Requires DatabaseConnector and SqlRender.

R Installation

install.packages("devtools")
devtools::install_github("OHDSI/DataQualityDashboard")

Executing Data Quality Checks

# fill out the connection details -----------------------------------------------------------------------
connectionDetails <- DatabaseConnector::createConnectionDetails(dbms = "", 
                                                              user = "", 
                                                              password = "", 
                                                              server = "", 
                                                              port = "", 
                                                              extraSettings = "")

cdmDatabaseSchema <- "yourCdmSchema" # the fully qualified database schema name of the CDM
resultsDatabaseSchema <- "yourResultsSchema" # the fully qualified database schema name of the results schema (that you can write to)
cdmSourceName <- "Your CDM Source" # a human readable name for your CDM source

# determine how many threads (concurrent SQL sessions) to use ----------------------------------------
numThreads <- 1 # on Redshift, 3 seems to work well

# specify if you want to execute the queries or inspect them ------------------------------------------
sqlOnly <- FALSE # set to TRUE if you just want to get the SQL scripts and not actually run the queries

# where should the logs go? -------------------------------------------------------------------------
outputFolder <- "output"

# logging type -------------------------------------------------------------------------------------
verboseMode <- FALSE # set to TRUE if you want to see activity written to the console

# write results to table? ------------------------------------------------------------------------------
writeToTable <- TRUE # set to FALSE if you want to skip writing to a SQL table in the results schema

# if writing to table and using Redshift, bulk loading can be initialized -------------------------------

# Sys.setenv("AWS_ACCESS_KEY_ID" = "",
#            "AWS_SECRET_ACCESS_KEY" = "",
#            "AWS_DEFAULT_REGION" = "",
#            "AWS_BUCKET_NAME" = "",
#            "AWS_OBJECT_KEY" = "",
#            "AWS_SSE_TYPE" = "AES256",
#            "USE_MPP_BULK_LOAD" = TRUE)

# which DQ check levels to run -------------------------------------------------------------------
checkLevels <- c("TABLE", "FIELD", "CONCEPT")

# which DQ checks to run? ------------------------------------

checkNames <- c() # Names can be found in inst/csv/OMOP_CDM_v5.3.1_Check_Desciptions.csv

# run the job --------------------------------------------------------------------------------------
DataQualityDashboard::executeDqChecks(connectionDetails = connectionDetails, 
                                    cdmDatabaseSchema = cdmDatabaseSchema, 
                                    resultsDatabaseSchema = resultsDatabaseSchema,
                                    cdmSourceName = cdmSourceName, 
                                    numThreads = numThreads,
                                    sqlOnly = sqlOnly, 
                                    outputFolder = outputFolder, 
                                    verboseMode = verboseMode,
                                    writeToTable = writeToTable,
                                    checkLevels = checkLevels,
                                    checkNames = checkNames)

# inspect logs ----------------------------------------------------------------------------
ParallelLogger::launchLogViewer(logFileName = file.path(outputFolder, cdmSourceName, 
                                                      sprintf("log_DqDashboard_%s.txt", cdmSourceName)))

# (OPTIONAL) if you want to write the JSON file to the results table separately -----------------------------
jsonFilePath <- ""
DataQualityDashboard::writeJsonResultsToTable(connectionDetails = connectionDetails, 
                                            resultsDatabaseSchema = resultsDatabaseSchema, 
                                            jsonFilePath = jsonFilePath)
                                            

Viewing Results

Launching Dashboard as Shiny App

DataQualityDashboard::viewDqDashboard(jsonPath = file.path(getwd(), outputFolder, cdmSourceName, sprintf("results_%s.json", cdmSourceName)))

Launching on a web server

If you have npm installed:

  1. Install http-server:
npm install -g http-server
  1. Rename the json file to results.json and place it in inst/shinyApps/www

  2. Go to inst/shinyApps/www, then run:

http-server

A results JSON file for the Synthea synthetic dataset will be shown. You can view your results by replacing the results.json file with your file (with name results.json).

View checks

To see description of checks using R, execute the command bellow:

View(read.csv(system.file("csv","OMOP_CDMv5.3.1_Check_Descriptions.csv",package="DataQualityDashboard"),as.is=T))

Support

License

DataQualityDashboard is licensed under Apache License 2.0

Development status

In early development phase. Not ready for use.

Acknowledgements

This project is supported in part through the National Science Foundation grant IIS 1251151.

You can’t perform that action at this time.