Reproducible and flexible example notebooks that may be used by researchers using Databricks within the NHS England SDE.
This repository is maintained by NHS England Data Wranglers team.
To contact us raise an issue on Github or via email.
See our (and our colleagues') other work here:
The NHS England SDE is a data and research analysis platform, which is built to uphold the highest standards of privacy and security of NHS health and social care data when used for research and analysis.
Within the SDE, users have access to Databricks. Some users may be new to Databricks, so we provide example notebooks and useful functions for what they can do with Databricks inside the SDE. In addition, the analysis explores machine learning (ML) capabilities in the SDE.
pyspark_code_example: PySpark is the Python API for Aparche Spark - it can handle Big Data better than pandas.
Usually, if a dataframe has more than one million rows, it is big data. However, a dataset may have less rows than this and still be classed as "big" due to complexity.
This folder contains notebooks exploring the machine learning capabilities of PySpark in the SDE. Examples include:
- decision_tree_regressor
- random_forest_classifier
This folder contains analysis for smaller data sets, using libraries like pandas and sklearn. Examples include:
- regression_multivariable
- regression_simple
SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. The example notebook demonstrates what you can do with SparkR.
Useful functions and code, including code to create and save toy data sets.
The codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.
Any HTML or Markdown documentation is © Crown copyright and available under the terms of the Open Government 3.0 licence.
Data Wranglers, Data & Analytics, NHS England
For individuals involved see CONTRIBUTORS.