stigmatizing_variables_validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c887371e",
   "metadata": {},
   "source": [
    "# BioData Catalyst Powered by PIC-SURE: Validate stigmatizing variables\n",
    "\n",
    "The purpose of this notebook is to validate stigmatizing variables in [BioData Catalyst Powered by PIC-SURE](https://picsure.biodatacatalyst.nhlbi.nih.gov/). Specifically, this notebook will ensure the stigmatizing variables identified were removed from PIC-SURE Open Access.\n",
    "\n",
    "For more information about stigmatizing variables, please view the [README.md](https://github.com/hms-dbmi/biodata_catalyst_stigmatizing_variables#biodata_catalyst_stigmatizing_variables)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73c49668",
   "metadata": {},
   "source": [
    "### Install packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6045077e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install --upgrade --force-reinstall git+https://github.com/hms-dbmi/pic-sure-python-client.git\n",
    "!{sys.executable} -m pip install --upgrade --force-reinstall git+https://github.com/hms-dbmi/pic-sure-python-adapter-hpds.git\n",
    "!{sys.executable} -m pip install --upgrade --force-reinstall git+https://github.com/hms-dbmi/pic-sure-biodatacatalyst-python-adapter-hpds.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3214eb18",
   "metadata": {},
   "outputs": [],
   "source": [
    "import PicSureClient\n",
    "import PicSureBdcAdapter\n",
    "from python_lib.utils import get_multiIndex_variablesDict, joining_variablesDict_onCol\n",
    "from python_lib.stig_utils import validate_stig_vars\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d222ec62",
   "metadata": {},
   "source": [
    "### Connect to PIC-SURE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59d787ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "PICSURE_network_URL = \"https://picsure.biodatacatalyst.nhlbi.nih.gov/picsure\" \n",
    "resource_id = \"70c837be-5ffc-11eb-ae93-0242ac130002\" # Be sure to use Open Access resource id\n",
    "token_file = \"token.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcabd434",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(token_file, \"r\") as f:\n",
    "    my_token = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65ee519d",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = PicSureClient.Client()\n",
    "connection = client.connect(PICSURE_network_URL, my_token, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd7faa48",
   "metadata": {},
   "source": [
    "### Get concept paths from PIC-SURE Open Access"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a52ffd",
   "metadata": {},
   "source": [
    "To ensure that all stigmatizing variables were removed from PIC-SURE Open Access, we will compare the previously identified stigmatizing variables to a list of all variables in Open Access. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83f24204",
   "metadata": {},
   "outputs": [],
   "source": [
    "bdc = PicSureBdcAdapter.Adapter(PICSURE_network_URL, my_token)\n",
    "dictionary = bdc.useDictionary().dictionary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "149f148b",
   "metadata": {},
   "outputs": [],
   "source": [
    "variables = dictionary.find() # Input with study or list of studies that if needed\n",
    "variables = variables.dataframe()\n",
    "\n",
    "#variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec085839",
   "metadata": {},
   "source": [
    "### Validation testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48aa4900",
   "metadata": {},
   "source": [
    "`validate_stig_vars` is a function that compares the list of previously identified stigmatizing variables to the variables in PIC-SURE Open Access. If stigmatizing variables are found in Open Access, it will save the variables to a specified output file. \n",
    "\n",
    "| Function | Arguments / Input | Output|\n",
    "|--------|-------------------|-------|\n",
    "| `validate_stig_vars()` | (1) fullVariableDict of Open Access variables, (2) tab-delimited list of stigmatizing variables - output from identify_stigmatizing_variables.ipynb, (3) output file name | list of stigmatizing variables found in Open Access, if any |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5661d271",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_file = 'stigmatizing_variable_results/REVAMP_stigmatizing_variables.txt'\n",
    "output_file = 'stigmatizing_variable_results/validation1.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c4acb65",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = validate_stig_vars(variables, input_file, output_file)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}