This repository provides examples of using pre-trained BERT models from SparkNLP with PySpark for Natural Language Processing task.

Our joint Solution Accelerator with John Snow Labs automates the detection of sensitive information contained within unstructured data using NLP models for healthcare. Extracted data is stored within the Lakehouse, where teams can use the pre-trained models to easily remove, obfuscate or mask data for downstream analytics at massive scale.

Web application to detect emotion in text

Media diversity shapes perspectives, yet biased news distorts reality, fostering misinformation. 'Political Ideologies Prediction in News Articles' aims to forecast bias using PySpark, NLP, and ML for adaptable, swift inference. Integrated with NYT API, it predicts bias in top political articles, fostering better understanding of subjective content

Media diversity shapes perspectives, yet biased news distorts reality, fostering misinformation. 'Political Ideologies Prediction in News Articles' aims to forecast bias using PySpark, NLP, and ML for adaptable, swift inference. Integrated with NYT API, it predicts bias in top political articles, fostering better understanding of subjective content

This repository contains application code for the Wizeline Data Engineering Bootcamp (DEB) 2023. It is one of two repositories for the DEB. The other houses the infrastructure code.

Completed professional certificates and specializations via Coursera

Build a lakehouse for all your gamer data and use natural language processing techniques to flag questionable comments for moderation.

Databricks and John Snow Labs Solution Accelerator for Medicare Risk Adjustment automates the extraction of undiagnosed member conditions from unstructured clinical notes with NLP models, improving downstream reimbursements.

Drawing a Company Ecosystem Graph

Automated Extraction of Medical Risk Factors For Life Insurance Underwriting

To ensure ongoing drug safety, pharma companies need to monitor and report adverse drug events post-market launch. This accelerator extracts, processes and analyzes adverse drug events from real-world text data using NLP

Generate oncology insights from real-world data using NLP. Once extracted, oncology data is enriched with useful information like ICD-10 codes and used to build powerful visualizations

Twitter based stock market prediction using Pyspark, project for Big Data PW 2020L

This project focuses on the use of big data platforms, specifically Spark (PySpark, SparkML, Spark NLP). We will use the comment text a user posted, categorize the sentiment and predict scores of each comment. Our objective is to understand the dynamics of the Reddit online community and how the way people communicate online leads to different r…

Micro project on big data technologies via spark

My notebook on using Python with Jupyter Notebook, PySpark etc

Text summarization algorithms using PySpark