Merge pull request #304 from scaleoutsystems/develop

Release v0.3.0

.github/workflows/build-sphinx-pages.yaml

@@ -0,0 +1,50 @@
+name: Sphinx to GitHub Pages
+
+on:
+  # Trigger the workflow on push or pull request,
+  # but only for the main branch
+  push:
+    branches: [ develop ]
+  pull_request:
+    branches: [ develop ]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+
+    - name: Checkout current repository
+      uses: actions/checkout@v2
+      with:
+        fetch-depth: 0 # otherwise, you will failed to push refs to dest repo
+        token: ${{ secrets.PAT }}
+
+    - name: Run sphinx-apidoc
+      run: |
+        pip install sphinx fedn
+        cd docs
+        echo "Remove any existing fedn.*rst"
+        rm -v source/fedn.*rst || true
+        echo "Generate API documentation"
+        sphinx-apidoc -f -o source/ ../fedn/fedn/
+        git status
+    
+    - name: Commit and push workflow changes back into repo
+      uses: stefanzweifel/git-auto-commit-action@v4
+      with:
+        file_pattern: docs/source/*.rst
+        commit_message: Apply sphinx action changes
+        branch: develop
+        push_options: --force
+
+    - name: Run sphinx-build and commit to gh-pages
+      uses: sphinx-notes/pages@master
+      with:
+        documentation_path: docs/source
+
+    - name: Push sphinx-build to gh-pages
+      uses: ad-m/github-push-action@master
+      with:
+        github_token: ${{ secrets.SPHINX_GH_TOKEN }}
+        branch: gh-pages

README.md

@@ -1,15 +1,15 @@
 ![alt text](https://thumb.tildacdn.com/tild6637-3937-4565-b861-386330386132/-/resize/560x/-/format/webp/FEDn_logo.png)
 
-FEDn is an open-source, modular and model agnostic framework for Federated Machine Learning. Scale seamlessly from pseudo-distributed development to real-world production networks in distributed, heterogeneous environments.
+FEDn is a modular and model agnostic framework for Hierarchical Federated Machine Learning. Scale seamlessly from pseudo-distributed development to real-world production networks in distributed, heterogeneous environments.
 
-[Scaleout Discord server](https://discord.gg/KMg4VwszAd) 
+Community support: [Scaleout Discord server](https://discord.gg/KMg4VwszAd) 
 
 ## Core Features
 
 - **ML-framework agnostic**. Model updates are treated as black-box computations. This means that it is possible to support virtually any ML model type or framework. Support for Keras and PyTorch is available out-of-the-box, and support for many other model types, including models from SKLearn, are in active development.
-- **Horizontally scalable through a tiered aggregation scheme**. FEDn allows for massive horizontal scaling. This is achieved by a tiered architecture where multiple combiners divide up the work to coordinate client updates and aggregation. Recent benchmarks show high performance both for thousands of clients in a cross-device setting and for large model updates (1GB) in a cross-silo setting, see https://arxiv.org/abs/2103.00148.
-- **Built for real-world production scenarios**. The implementation is based on proven design patterns in distributed computing and incorporates enterprise security features. A key feature is that data clients do not have to expose any ingress ports.
-- **WebUI to manage alliances, track training progress and follow client validations in real time**. The FEDn frontend lets you efficiently manage and track events and training progress in the alliance, helping you monitor both client and server performance.   
+- **Hierarchical Federated Learning for high scalability**. FEDn allows for massive horizontal scaling. This is achieved by a hierarchical architecture where multiple combiners divide up the work to coordinate client updates and aggregation. Recent benchmarks show high performance both for thousands of clients in a cross-device setting and for large model updates (1GB) in a cross-silo setting, see https://arxiv.org/abs/2103.00148.
+- **Built for real-world production scenarios**. The implementation is based on proven design patterns in distributed computing and incorporates enterprise security features. A key feature is that clients do not have to expose any ingress ports.
+- **UI to manage alliances, track training progress and follow client validations in real time**. FEDn has a UI that lets you efficiently track events, training progress, visualize model validations accross the federation, and monitor both client and server performance.   
 
 ## Documentation 
 More details about the architecture and implementation:  
@@ -42,6 +42,8 @@ docker-compose -f config/base-services.yaml -f config/private-network.yaml up
 Make sure you can access the following services before proceeding to the next steps: 
  - Minio: http://localhost:9000
  - Mongo Express: http://localhost:8081
+
+(You can update username and password for the services in config/base-services.yaml, default values: user=fedn_admin, password=password)
 
 3. Start the Reducer  
 
@@ -73,24 +75,20 @@ The deployment, sizing of nodes, and tuning of a FEDn network in production depe
 Scaleout Studio is a cloud-native SaaS for MLOps for Decentralized AI applications. Studio lets you deploy, manage and monitor FEDn networks as apps deployed to Kubernetes, all from a graphical interface. In addtion to FEDn, Studio provides developer tools (e.g. Jupyter Labs and VSCode), storage managmement (Kubernetes volumes, minio, MongoDB etc), and model serving for the federated model (Tensorflow Serving, TorchServe, MLflow or custom serving). End-to-end example here: https://www.youtube.com/watch?v=-a_nIzkSumI
 
 - Sign up for private-beta access at https://scaleoutsystems.com/.   
-- [Deploy STACKn on your own infrastructure] (https://github.com/scaleoutsystems/stackn) 
+- [Deploy STACKn on your own infrastructure](https://github.com/scaleoutsystems/stackn) 
 
 
 ## Where to go from here
-Explore additional projects/clients:
-
-- PyTorch version of the MNIST getting-started example: https://github.com/scaleoutsystems/examples/tree/main/mnist-pytorch
-- Sentiment analysis with a Keras CNN-lstm trained on the IMDB dataset (cross-silo): https://github.com/scaleoutsystems/FEDn-client-imdb-keras 
-- Sentiment analysis with a PyTorch CNN trained on the IMDB dataset (cross-silo): https://github.com/scaleoutsystems/FEDn-client-imdb-pytorch.git 
-- VGG16 trained on cifar-10 with a PyTorch client (cross-silo): https://github.com/scaleoutsystems/FEDn-client-cifar10-pytorch 
-- Human activity recognition with a Keras CNN based on the casa dataset (cross-device): https://github.com/scaleoutsystems/FEDn-client-casa-keras 
-- Fraud detection with a Keras auto-encoder (ANN encoder): https://github.com/scaleoutsystems/FEDn-client-fraud_keras  
+- Explore additional projects/clients: https://github.com/scaleoutsystems/examples 
+- Understand the architecture: https://scaleoutsystems.github.io/fedn/#/architecture 
+- Understand the compute package: https://github.com/scaleoutsystems/fedn/blob/master/docs/tutorial.md
+- Deploy a fully distributed network: https://github.com/scaleoutsystems/fedn/blob/master/docs/deployment.md
 
 ## Community support 
 Join the [Scaleout Discord Server](https://discord.gg/KMg4VwszAd) to engage with other users and developers. If you have a bug report or a feature request, start a ticket directly here on GitHub. 
 
 ## Commercial support
-Scaleout offers flexible support agreements, reach out at (https://scaleoutsystems.com) to inquire about Enterprise support.
+Scaleout offers flexible support agreements, reach out at (https://www.scaleoutsystems.com/contact) to inquire about Enterprise support.
 
 ## Making contributions
 All pull requests will be considered and are much appreciated. Reach out to one of the maintainers if you are interested in making contributions, and we will help you find a good first issue to get you started. 
@@ -110,7 +108,13 @@ If you use FEDn in your research, please cite:
 ```
 
 ## Organizational collaborators, contributors and supporters 
-<img src="docs/img/logos/Scaleout.png" width="250"> <img src="docs/img/logos/UU.png" width="200"> <img src="docs/img/logos/Scania.png" width="250">
+<p align="middle">
+ <img src="docs/source/img/logos/Scaleout.png" width="19%">
+ <img src="docs/source/img/logos/UU.png" width="19%"> 
+ <img src="docs/source/img/logos/ai-sweden-logo.png" width="19%">
+ <img src="docs/source/img/logos/zenseact-logo.png" width="19%">
+ <img src="docs/source/img/logos/Scania.png" width="19%"> 
+</p>
 
 ## License
 FEDn is licensed under Apache-2.0 (see LICENSE file for full information).

config/base-services.yaml

@@ -36,6 +36,8 @@ services:
   mongo-express:
     image: mongo-express:latest
     restart: always
+    depends_on:
+      - "mongo"
     environment:
       - ME_CONFIG_MONGODB_SERVER=mongo
       - ME_CONFIG_MONGODB_PORT=6534

config/combiner-dev.yaml

@@ -16,5 +16,4 @@ services:
       - ${PWD}/config:/app/client/config
       - ${PWD}/tmp/c1-models:/tmp/models
     ports:
-      - 12080:12080
-
+      - 12080:12080

config/settings-combiner.yaml.template

@@ -8,4 +8,4 @@ combiner:
     name: combiner
     host: combiner
     port: 12080
-    max_clients: 30
+    max_clients: 30

docs/.nojekyll

@@ -0,0 +1 @@
+

docs/Makefile

@@ -0,0 +1,35 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# html alias
+html:
+	@echo "Remove any existing fedn.*.rst"
+	@rm -v $(SOURCEDIR)/fedn.*rst || true
+
+	@echo "Generate API documentation"
+	@sphinx-apidoc -f -o ./source ../fedn/fedn
+
+	@echo "Generate HTML documentation"
+	@$(SPHINXBUILD) -b html "$(SOURCEDIR)" "$(BUILDDIR)"
+
+	@echo
+	@echo "Build finished."
+	@echo
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/source/_static/custom.css

@@ -0,0 +1,32 @@
+html {
+  scroll-behavior: smooth;
+}
+
+a {
+  color: #0298fb;
+}
+
+.vp-sidebar .toctree-l1 a:link, .vp-sidebar .toctree-l2 a:link {
+   color: #000;
+}
+
+.vp-sidebar .toctree-l1 a:active, .vp-sidebar .toctree-l2 a:active {
+   font-weight: 500;
+}
+
+
+.vp-sidebar .toctree-l1 a:hover, .vp-sidebar .toctree-l2 ,a:hover {
+   color: #0298fb;
+}
+
+.vp-sidebar .toctree-l1.current a {
+    border-left: .5rem solid #02bbff;
+}
+
+.vp-sidebar .toctree-l1 a.current {
+    border-left-color: #0076ff;
+}
+
+.highlight pre {
+  color: #48b685;
+}

docs/source/_static/docs_theme.css

@@ -0,0 +1,5 @@
+@import url("theme.css");
+
+.wy-nav-content {
+    max-width: 90%;
+}