Auto-generate requirements.txt to sync with pyproject.toml automatically

.github/workflows/flower.yml

@@ -33,5 +33,7 @@ jobs:
         run: python -m poetry install --extras "simulation"
       - name: Check if protos need recompilation
         run: ./dev/check-protos.sh
+      - name: Check if example requirements.txt files need regeneration
+        run: ./dev/check-requirements-txt.sh
       - name: Lint + Test (isort/black/docformatter/mypy/pylint/flake8/pytest)
         run: ./dev/test.sh

dev/check-requirements-txt.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+
+# Copyright 2023 Flower Labs GmbH. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+./generate-requirements-txt.sh
+
+set -e
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"/../
+
+# Fail if user forgot to sync requirements.txt and pyproject.toml
+CHANGED=$(git diff --name-only HEAD examples)
+
+if [ -n "$CHANGED" ]; then
+    echo "Changes detected, requirements.txt and pyproject.toml is not synced. Please run the script dev/generate-requirements-txt."
+    exit 1
+fi
+
+echo "No changes detected"
+exit 0

dev/generate-requirements-txt.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+# Copyright 2023 Flower Labs GmbH. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+set -e
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"/../
+
+# Purpose of this script is to regenerate requirements.txt
+for path in $(find ./examples -type f -name 'pyproject.toml' | sed -E 's|/[^/]+$||' |sort -u)
+do
+    pipreqs --mode 'compat' --force $path
+done

examples/advanced_pytorch/README.md

@@ -21,13 +21,18 @@ This will create a new directory called `advanced_pytorch` containing the follow
 
 ```shell
 -- pyproject.toml
+-- requirements.txt
 -- client.py
 -- server.py
 -- README.md
 -- run.sh
 ```
 
-Project dependencies (such as `pytorch` and `flwr`) are defined in `pyproject.toml`. We recommend [Poetry](https://python-poetry.org/docs/) to install those dependencies and manage your virtual environment ([Poetry installation](https://python-poetry.org/docs/#installation)), but feel free to use a different way of installing dependencies and managing virtual environments if you have other preferences.
+### Installing Dependencies
+
+Project dependencies (such as `torch` and `flwr`) are defined in `pyproject.toml` and `requirements.txt`. We recommend [Poetry](https://python-poetry.org/docs/) to install those dependencies and manage your virtual environment ([Poetry installation](https://python-poetry.org/docs/#installation)) or [pip](https://pip.pypa.io/en/latest/development/), but feel free to use a different way of installing dependencies and managing virtual environments if you have other preferences.
+
+#### Poetry
 
 ```shell
 poetry install
@@ -42,6 +47,14 @@ poetry run python3 -c "import flwr"
 
 If you don't see any errors you're good to go!
 
+#### pip
+
+Write the command below in your terminal to install the dependencies according to the configuration file requirements.txt.
+
+```shell
+pip install -r requirements.txt
+```
+
 # Run Federated Learning with PyTorch and Flower
 
 The included `run.sh` will start the Flower server (using `server.py`), 
@@ -55,4 +68,3 @@ poetry run ./run.sh
 The `run.sh` script starts processes in the background so that you don't have to open eleven terminal windows. If you experiment with the code example and something goes wrong, simply using `CTRL + C` on Linux (or `CMD + C` on macOS) wouldn't normally kill all these processes, which is why the script ends with `trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT` and `wait`. This simply allows you to stop the experiment using `CTRL + C` (or `CMD + C`). If you change the script and anything goes wrong you can still use `killall python` (or `killall python3`) to kill all background processes (or a more specific command if you have other Python processes running that you don't want to kill).
 
 You can also manually run `poetry run python3 server.py` and `poetry run python3 client.py` for as many clients as you want but you have to make sure that each command is ran in a different terminal window (or a different computer on the network).
-

examples/advanced_pytorch/requirements.txt

@@ -0,0 +1,3 @@
+flwr~=1.4.0
+torch~=2.0.1
+torchvision~=0.15.2

examples/advanced_tensorflow/README.md

@@ -21,13 +21,18 @@ This will create a new directory called `advanced_tensorflow` containing the fol
 
 ```shell
 -- pyproject.toml
+-- requirements.txt
 -- client.py
 -- server.py
 -- README.md
 -- run.sh
 ```
 
-Project dependencies (such as `tensorflow` and `flwr`) are defined in `pyproject.toml`. We recommend [Poetry](https://python-poetry.org/docs/) to install those dependencies and manage your virtual environment ([Poetry installation](https://python-poetry.org/docs/#installation)), but feel free to use a different way of installing dependencies and managing virtual environments if you have other preferences.
+### Installing Dependencies
+
+Project dependencies (such as `tensorflow` and `flwr`) are defined in `pyproject.toml` and `requirements.txt`. We recommend [Poetry](https://python-poetry.org/docs/) to install those dependencies and manage your virtual environment ([Poetry installation](https://python-poetry.org/docs/#installation)) or [pip](https://pip.pypa.io/en/latest/development/), but feel free to use a different way of installing dependencies and managing virtual environments if you have other preferences.
+
+#### Poetry
 
 ```shell
 poetry install
@@ -42,6 +47,14 @@ poetry run python3 -c "import flwr"
 
 If you don't see any errors you're good to go!
 
+#### pip
+
+Write the command below in your terminal to install the dependencies according to the configuration file requirements.txt.
+
+```shell
+pip install -r requirements.txt
+```
+
 ## Run Federated Learning with TensorFlow/Keras and Flower
 
 The included `run.sh` will call a script to generate certificates (which will be used by server and clients), start the Flower server (using `server.py`), sleep for 2 seconds to ensure the the server is up, and then start 10 Flower clients (using `client.py`). You can simply start everything in a terminal as follows:

examples/advanced_tensorflow/requirements.txt

@@ -0,0 +1,2 @@
+flwr~=1.4.0
+tensorflow~=2.12.0

examples/android/README.md

@@ -16,7 +16,11 @@ Start by cloning the example project. We prepared a single-line command that you
 git clone --depth=1 https://github.com/adap/flower.git && mv flower/examples/android . && rm -rf flower && cd android
 ```
 
-Project dependencies (such as `tensorflow` and `flwr`) are defined in `pyproject.toml`. We recommend [Poetry](https://python-poetry.org/docs/) to install those dependencies and manage your virtual environment ([Poetry installation](https://python-poetry.org/docs/#installation)), but feel free to use a different way of installing dependencies and managing virtual environments if you have other preferences.
+### Installing Dependencies
+
+Project dependencies (such as `tensorflow` and `flwr`) are defined in `pyproject.toml` and `requirements.txt`. We recommend [Poetry](https://python-poetry.org/docs/) to install those dependencies and manage your virtual environment ([Poetry installation](https://python-poetry.org/docs/#installation)) or [pip](https://pip.pypa.io/en/latest/development/), but feel free to use a different way of installing dependencies and managing virtual environments if you have other preferences.
+
+#### Poetry
 
 ```shell
 poetry install
@@ -31,6 +35,14 @@ poetry run python3 -c "import flwr"
 
 If you don't see any errors you're good to go!
 
+#### pip
+
+Write the command below in your terminal to install the dependencies according to the configuration file requirements.txt.
+
+```shell
+pip install -r requirements.txt
+```
+
 # Run Federated Learning on Android Devices
 
 The included `run.sh` will start the Flower server (using `server.py`). You can simply start it in a terminal as follows:

examples/android/requirements.txt

@@ -0,0 +1 @@
+numpy~=1.21.1

examples/dp-sgd-mnist/README.md

@@ -18,13 +18,17 @@ This will create a new directory called `dp-sgd-mnist` containing the following
 
 ```shell
 -- pyproject.toml
+-- requirements.txt
 -- client.py
 -- server.py
 -- common.py
 -- README.md
 ```
+### Installing Dependencies
 
-Project dependencies (such as `tensorflow` and `tensorflow-privacy`) are defined in `pyproject.toml`. We recommend [Poetry](https://python-poetry.org/docs/) to install those dependencies and manage your virtual environment ([Poetry installation](https://python-poetry.org/docs/#installation)), but feel free to use a different way of installing dependencies and managing virtual environments if you have other preferences.
+Project dependencies (such as `tensorflow` and `tensorflow-privacy`) are defined in `pyproject.toml` and `requirements.txt`. We recommend [Poetry](https://python-poetry.org/docs/) to install those dependencies and manage your virtual environment ([Poetry installation](https://python-poetry.org/docs/#installation)) or [pip](https://pip.pypa.io/en/latest/development/), but feel free to use a different way of installing dependencies and managing virtual environments if you have other preferences.
+
+#### Poetry
 
 ```shell
 poetry install
@@ -39,6 +43,14 @@ poetry run python3 -c "import flwr"
 
 If you don't see any errors you're good to go!
 
+#### pip
+
+Write the command below in your terminal to install the dependencies according to the configuration file requirements.txt.
+
+```shell
+pip install -r requirements.txt
+```
+
 # Run Federated Learning with TensorFlow/Keras/Tensorflow-Privacy and Flower
 
 Afterwards you are ready to start the Flower server as well as the clients. You can simply start the server in a terminal as follows:
@@ -47,6 +59,7 @@ Afterwards you are ready to start the Flower server as well as the clients. You
 # terminal 1
 poetry run python3 server.py
 ```
+
 Now you are ready to start the Flower clients which will participate in the learning. To do so simply open two more terminals and run the following command in each:
 
 ```shell
@@ -78,7 +91,6 @@ poetry run python3 client.py --num-clients 3 --partition 1 &
 poetry run python3 client.py --num-clients 3 --partition 2 --dpsgd True
 ```
 
-
 Additional training parameters for the client and server can be referenced by passing `--help` to either script.
 
 Other things to note is that when all clients are running `dpsgd`, either train for more rounds or increase the local epochs to achieve optimal performance. You shall need to carefully tune the hyperparameters to your specific setup.

examples/dp-sgd-mnist/requirements.txt

@@ -0,0 +1,4 @@
+flwr~=1.4.0
+numpy~=1.21.1
+tensorflow~=2.12.0
+tensorflow_privacy~=0.8.9

examples/embedded_devices/README.md

@@ -36,11 +36,13 @@ The only requirement for the server is to have flower installed. You can do so b
 2. Extract the imgae (~14GB) and flash it onto the uSD card using Etcher (or equivalent).
 3. Follow [the instructions](https://developer.nvidia.com/embedded/learn/get-started-jetson-xavier-nx-devkit) to setup the device.
 4. Installing Docker: Docker comes pre-installed with the Ubuntu image provided by NVIDIA. But for convinience we will create a new user group and add our user to it (with the idea of not having to use `sudo` for every command involving docker (e.g. `docker run`, `docker ps`, etc)). More details about what this entails can be found in the [Docker documentation](https://docs.docker.com/engine/install/linux-postinstall/). You can achieve this by doing:
+
     ``` bash
     $ sudo usermod -aG docker $USER
     # apply changes to current shell (or logout/reboot)
     $ newgrp docker
     ```
+
 5. The minimal installation to run this example only requires an additional package, `git`, in order to clone this repo. Install `git` by:
 
     ```bash
@@ -50,6 +52,7 @@ The only requirement for the server is to have flower installed. You can do so b
 6. (optional) additional packages:
         <img align="right" style="padding-top: 40px; padding-left: 15px" width="575" height="380" src="media/tmux_jtop_view.gif">
      * [jtop](https://github.com/rbonghi/jetson_stats),  to monitor CPU/GPU utilization, power consumption and, many more.
+
         ```bash
         # First we need to install pip3
         $ sudo apt-get install python3-pip -y 
@@ -58,7 +61,9 @@ The only requirement for the server is to have flower installed. You can do so b
         # finally, install jtop
         $ sudo -H pip3 install -U jetson-stats
         ```
+
      * [TMUX](https://github.com/tmux/tmux/wiki), a terminal multiplexer.
+
         ```bash
         # install tmux
         $ sudo apt-get install tmux -y
@@ -67,6 +72,7 @@ The only requirement for the server is to have flower installed. You can do so b
         ``` 
 
 7. Power modes: The Jetson devices can operate at different power modes, each making use of more or less CPU cores clocked at different freqencies. The right power mode might very much depend on the application and scenario. When power consumption is not a limiting factor, we could use the highest 15W mode using all 6 CPU cores. On the other hand, if the devices are battery-powered we might want to make use of a low power mode using 10W and 2 CPU cores. All the details regarding the different power modes of a Jetson Xavier-NX can be found [here](https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fpower_management_jetson_xavier.html%23wwpID0E0NO0HA). For this demo we'll be setting the device to the high performance mode:
+
     ```bash
     $ sudo /usr/sbin/nvpmodel -m 2 # 15W with 6cpus @ 1.4GHz
     ```
@@ -76,6 +82,7 @@ The only requirement for the server is to have flower installed. You can do so b
 1. Install Ubuntu server 20.04 LTS 64-bit for Rapsberry Pi. You can do this by using one of the images provided [by Ubuntu](https://ubuntu.com/download/raspberry-pi) and then use Etcher. Alternativelly, astep-by-step installation guide, showing how to download and flash the image onto a uSD card and, go throught the first boot process, can be found [here](https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview). Please note that the first time you boot your RPi it will automatically update the system (which will lock `sudo` and prevent running the commands below for a few minutes)
 
 2. Install docker (+ post-installation steps as in [Docker Docs](https://docs.docker.com/engine/install/linux-postinstall/)):
+
     ```bash
     # make sure your OS is up-to-date
     $ sudo apt-get update
@@ -104,6 +111,7 @@ For this demo we'll be using [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.h
 ## Server
 
 Launch the server and define the model you'd like to train. The current code (see `utils.py`) provides two models for CIFAR-10: a small CNN (more suitable for Raspberry Pi) and, a ResNet18, which will run well on the gpu. Each model can be specified using the `--model` flag with options `Net` or `ResNet18`. Launch a FL training setup with one client and doing three rounds as:
+
 ```bash
 # launch your server. It will be waiting until one client connects
 $ python server.py --server_address <YOUR_SERVER_IP:PORT> --rounds 3 --min_num_clients 1 --min_sample_size 1 --model ResNet18

examples/ios/README.md

@@ -4,7 +4,11 @@ FLiOS is a sample application for testing and benchmarking the Swift implementat
 
 ## Project Setup
 
-Project dependencies (`flwr`) are defined in `pyproject.toml`. We recommend [Poetry](https://python-poetry.org/docs/) to install those dependencies and manage your virtual environment ([Poetry installation](https://python-poetry.org/docs/#installation)), but feel free to use a different way of installing dependencies and managing virtual environments if you have other preferences.
+### Installing Dependencies
+
+Project dependencies (`flwr`) are defined in `pyproject.toml` and `requirements.txt`. We recommend [Poetry](https://python-poetry.org/docs/) to install those dependencies and manage your virtual environment ([Poetry installation](https://python-poetry.org/docs/#installation)) or [pip](https://pip.pypa.io/en/latest/development/), but feel free to use a different way of installing dependencies and managing virtual environments if you have other preferences.
+
+#### Poetry
 
 ```shell
 poetry install
@@ -14,11 +18,19 @@ poetry shell
 Poetry will install all your dependencies in a newly created virtual environment. To verify that everything works correctly you can run the following command:
 
 ```shell
-python3 -c "import flwr"
+poetry run python3 -c "import flwr"
 ```
 
 If you don't see any errors you're good to go!
 
+#### pip
+
+Write the command below in your terminal to install the dependencies according to the configuration file requirements.txt.
+
+```shell
+pip install -r requirements.txt
+```
+
 # Run Federated Learning on iOS Clients
 
 To start the server, write the following command in the terminal in the ios folder (with the conda environment created above):
@@ -27,7 +39,7 @@ To start the server, write the following command in the terminal in the ios fold
 python3 server.py
 ```
 
-Open the FlowerCoreML.xcodeproj with XCode, wait until the dependencies are fetched, then click build and run with iPhone 13 Pro Max as target, or you can deploy it in your own iOS device by connecting your Mac, where you run your XCode, and your iPhone.
+Open the FLiOS.xcodeproj with XCode, wait until the dependencies are fetched, then click build and run with iPhone 13 Pro Max as target, or you can deploy it in your own iOS device by connecting your Mac, where you run your XCode, and your iPhone.
 
 When the iOS app runs, load both the training and test dataset first. Then enter the hostname and port of your server in the TextField provided. Finally press `Start` which will start the federated training.
 

examples/ios/requirements.txt

@@ -0,0 +1,2 @@
+flwr~=1.4.0
+numpy~=1.21.1

examples/mt-pytorch/requirements.txt

@@ -0,0 +1,4 @@
+flwr~=1.4.0
+torch~=2.0.1
+torchvision~=0.15.2
+tqdm~=4.65.0