Substra is an open source federated learning (FL) software. It enables the training and validation of machine learning models on distributed datasets. It provides a flexible Python interface and a web application to run federated learning training at scale. This specific repository is about SubstraFL the high-level federated learning Python library based on the low-level Substra python library. SubstraFL is used to run complex federated learning experiments at scale.
Substra's main usage is in production environments. It has already been deployed and used by hospitals and biotech companies (see the MELLODDY project for instance). Substra can also be used on a single machine to perform FL simulations and debug code.
Substra was originally developed by Owkin and is now hosted by the Linux Foundation for AI and Data. Today Owkin is the main contributor to Substra.
Join the discussion on Slack and subscribe here to our newsletter.
pip install substrafl
Have a look at our documentation.
Try out our MNIST example.
If you need support, please either raise an issue on Github or ask on Slack.
Substra warmly welcomes any contribution. Feel free to fork the repo and create a pull request.
Install substrafl
in editable mode with developper dependencies.
In addition, install substra
and substra-tools
in editable mode.
It is recommended to install all the libraries in a Python virtual env.
git clone git@github.com:Substra/substrafl.git
pip install -e "substrafl[dev]"
git clone git@github.com:Substra/substra.git
pip install -e substra
git clone git@github.com:Substra/substra-tools.git
pip install -e substra-tools
Now you can use the following command from subtrafl
top level directory to run tests:
cd substrafl
make test-subprocess
Substra can be used in three different modes: using Python subprocesses (subprocess
),
using Docker (docker
) and using Kubernetes (remote
).
The command make test-subprocess
runs the test suite in subprocess mode. It's lightweight and perfect to start.
To test with the Docker mode, you will need Docker installed and running on your machine. If necessary, install it using Docker Desktop.
The following command runs the test suites in subprocess and Docker mode:
make test-local
``
Please be warned that some of these tests are slow and the whole test suite might require a couple hours to complete.
To try out a local deployment with Kubernetes, please follow the [installation instructions](https://docs.substra.org/en/stable/contributing/local-deployment.html) provided in the documentation.
The following command runs the remote tests:
```sh
make test-remote
The changelog is managed with towncrier.
To add a new entry in the changelog, add a file in the changes
folder. The file name should have the following structure:
<unique_id>.<change_type>
.
The unique_id
is a unique identifier, we currently use the PR number.
The change_type
can be of the following types: added
, changed
, removed
, fixed
.
To generate the changelog (for example during a release), use the following command (you must have the dev dependencies installed):
towncrier build --version=<x.y.z>
You can use the --draft
option to see what would be generated without actually writing to the changelog (and without removing the fragments).
The API documentation is generated from the SubstraFL repository thanks to the auto doc module. It is automatically built by https://github.com/Substra/substra-documentation and integrated into the general documentation here.
You can build the API documentation locally to see the changes made by your PR.
You need to have substrafl.dev installed on your machine and some extra requirements. From the SubstraFL repo:
pip install -e '.[dev]'
cd docs
pip install -r requirements.txt
You can build the documentation to see if your changes are well taken into account. From the ./docs folder :
make clean html
No warning should be thrown by this command.
Then open the ./docs/_build/index.html
file to see the results.
You can also generate the documentation live so each of your changes are taken into account on the fly:
make livehtml
NB: Sometimes make livehtml
does not take changes into account so running the make html
command in parallel might be needed.