Life insurance underwriting considers an applicant’s medical risk factors, usually provided inside free-text documents. New insurance-specific Natural Language Processing (NLP) models can automatically extract material medical history and risk factors from such documents. This joint Solution Accelerator with John Snow Labs makes it easy to implement this in practice – enabling a faster, more consistent, and more scalable underwriting experience. This tutorial will cover:
- The end-to-end solution architecture on Databricks, from data ingestion to dashboarding
- Easily analyze free-text documents to extract medical history & risk factors using NLP
- Executable Python notebooks implementing the solution that you can start from today
We will get the following list of medical risk factors from unstructured clinical notes using Spark NLP models and tools and make analysis on the results.
- Basic Profile
- ✅ Age
- ✅ Gender
- ✅ Weight
- ✅ Height
- ✅ Race/Ethnicity
- ✅ Disability
- Personal History
- ✅ Medical records (ICD-10-CM)
- ✅ Prescription history (RxNorm)
- ✅ Actions of prescriptions (Action Mapper)
- ✅ Family health history (ICD-10-CM + Assertion)
- Lifestyle
- ✅ Profession
- ✅ Marital status
- ✅ Housing
- ✅ Smoking
- ✅ Alcohol
- ✅ Substance
- Diseases
- ✅ Asthma and breathing problems
- ✅ Heart disease, including heart attacks and angina
- ✅ High cholesterol
- ✅ High blood pressure
- ✅ Hypertension
- ✅ Cancer
- ✅ Strokes, including mini-strokes and brain haemorrage
- ✅ Anxiety
- ✅ Depression
- ✅ Diabetes
- ✅ Obesity
- ✅ Epilepsy
- ✅ Cerebral palsy and other neurological conditions
- ✅ Kidney diseases
Copyright / License info of the notebook. Copyright [2023] the Notebook Authors. The source in this notebook is provided subject to the Apache 2.0 License. All included or referenced third party libraries are subject to the licenses set forth below.
| Author |
|---|
| Databricks Inc. |
| John Snow Labs Inc. |
Databricks Inc. (“Databricks”) does not dispense medical, diagnosis, or treatment advice. This Solution Accelerator (“tool”) is for informational purposes only and may not be used as a substitute for professional medical advice, treatment, or diagnosis. This tool may not be used within Databricks to process Protected Health Information (“PHI”) as defined in the Health Insurance Portability and Accountability Act of 1996, unless you have executed with Databricks a contract that allows for processing PHI, an accompanying Business Associate Agreement (BAA), and are running this notebook within a HIPAA Account. Please note that if you run this notebook within Azure Databricks, your contract with Microsoft applies.
To run this accelerator, set up JSL Partner Connect AWS, Azure and navigate to My Subscriptions tab. Make sure you have a valid subscription for the workspace you clone this repo into, then install on cluster as shown in the screenshot below, with the default options. You will receive an email from JSL when the installation completes.
Once the JSL installation completes successfully, clone this repo into a Databricks workspace. Attach the RUNME notebook to any cluster and execute the notebook via Run-All. A multi-step-job describing the accelerator pipeline will be created, and the link will be provided. Execute the multi-step-job to see how the pipeline runs.