Merge pull request #340 from mlf-core/release/1.11.0

release/1.11.0

.cookietemple.yml

@@ -15,7 +15,7 @@ full_name: Lukas Heumos
 email: lukas.heumos@posteo.net
 project_name: mlf-core
 project_short_description: Reproducible machine learning pipelines using mlflow.
-version: 1.10.0
+version: 1.11.0
 license: Apache2.0
 command_line_interface: Click
 testing_library: pytest

.github/create.sh

@@ -15,10 +15,6 @@ case "$1" in
     echo -e "\n\033[B\033[B\n\n\n\nn\n\n\n\nn" | mlf-core create
     ;;
 
-  xgboost_dask)
-    echo -e "\n\033[B\033[B\033[B\n\n\n\nn\n\n\n\nn" | mlf-core create
-    ;;
-
   *)
     echo -n "unknown"
     ;;

.github/workflows/create_mlflow.yml

@@ -10,7 +10,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        template: [ 'pytorch', 'tensorflow', 'xgboost', 'xgboost_dask' ]
+        template: [ 'pytorch', 'tensorflow', 'xgboost']
 
     steps:
       - name: Reclaim space

.github/workflows/publish_docs.yml

@@ -31,7 +31,7 @@ jobs:
 
       - name: Deploy
         if: ${{ github.ref == 'refs/heads/master' && github.event_name == 'push' }}
-        uses: peaceiris/actions-gh-pages@v3
+        uses: peaceiris/actions-gh-pages@v3.8.0
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           publish_dir: ./docs/_build/html

.github/workflows/run_cookietemple_lint.yml

@@ -18,7 +18,7 @@ jobs:
 
       - name: Install cookietemple
         run: |
-          pip install cookietemple
+          pip install cookietemple==1.2.4
 
       - name: Run cookietemple lint
         run: |

AUTHORS.rst

@@ -6,8 +6,9 @@ Development Lead
 ----------------
 
 * Lukas Heumos <lukas.heumos@posteo.net>
+* Philipp Ehmele <philipp_ehm@protonmail.com>
 
 Contributors
 ------------
 
-None yet. Why not be the first?
+* Edmund Miller <edmund.miller@utdallas.edu>

CHANGELOG.rst

@@ -6,6 +6,24 @@ Changelog
 
 This project adheres to `Semantic Versioning <https://semver.org/>`_.
 
+1.11.0 (2021-03-30)
+-------------------
+
+**Added**
+
+**Fixed**
+
+* [MLFLOW-PYTORCH] MNIST data loading (#298)
+* Improved console output info in case the repos sync secret could not be set automatically and added it to FAQ docs
+
+**Dependencies**
+
+**Deprecated**
+
+* Removed mlflow-xgboost_dask. It was never deterministic to begin with.
+  If you are interested in using mlf-core and XGBoost with multiple GPUs please refer to the old releases.
+
+
 1.10.0 (2021-03-11)
 -------------------
 

README.rst

@@ -44,6 +44,7 @@ Overview
    :alt: mlf-core soverview
 
    mlf-core provides CPU and GPU deterministic machine learning templates based on MLflow, Conda, Docker and a strong Github integration.
+   Templates are available for PyTorch, TensorFlow and XGBoost.
    A custom linter ensures that projects stay deterministic in all phases of development and deployment.
 
 Installing

cookietemple.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.10.0
+current_version = 1.11.0
 
 [bumpversion_files_whitelisted]
 setup_file = setup.py

docs/available_templates/available_templates.rst

@@ -9,8 +9,7 @@ cookietemple currently has the following templates available:
 1. `mlflow-pytorch`_
 2. `mlflow-tensorflow`_
 3. `mlflow-xgboost`_
-4. `mlflow-xgboost_dask`_
-5. `package-prediction`_
+4. `package-prediction`_
 
 In the following every template is devoted its own section, which explains its purpose, design, included frameworks/libraries, usage and frequently asked questions.
 A set of frequently questions, which all templates share see here: :ref:`all_templates_faq` FAQ.
@@ -19,7 +18,6 @@ It is recommended to use the sidebar to navigate this documentation, since it is
 .. include:: mlflow_pytorch.rst
 .. include:: mlflow_tensorflow.rst
 .. include:: mlflow_xgboost.rst
-.. include:: mlflow_xgboost_dask.rst
 .. include:: package_prediction.rst
 
 .. _all_templates_faq:

docs/conf.py

@@ -53,9 +53,9 @@
 # the built documents.
 #
 # The short X.Y version.
-version = '1.10.0'
+version = '1.11.0'
 # The full version, including alpha/beta/rc tags.
-release = '1.10.0'
+release = '1.11.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/faq.rst

@@ -28,3 +28,10 @@ How can you claim that your templates are GPU deterministic?
 
 We performed extensive tests on various GPU architectures and can confidently say that given our settings GPU deterministic results can be expected on the same GPU architecture.
 Hence, all mlf-core templates feature `system-intelligence <https://github.com/mlf-core/system-intelligence>`_, which queries your system for the used GPUs and their architecture.
+
+What to do if my sync secret could not be set during project creation process?
+-------------------------------------------------------------------------------
+
+You may encounter (rarely) the console info, that your projects sync secret could not be set during project creation. This is mostly caused by
+GitHubs API not being reachable for a few seconds. You can now either recreate your project or simply head to your projects repo and set a repo secret called ``CT_SYNC_TOKEN`` with its
+value being a valid personal access token for your account.

docs/requirements.txt

@@ -1,4 +1,4 @@
-Sphinx==3.5.2
-sphinx_rtd_theme==0.5.1
+Sphinx==3.5.4
+sphinx_rtd_theme==0.5.2
 sphinx-automodapi==0.13
 -r ../requirements.txt

mlf_core/__init__.py

@@ -2,4 +2,4 @@
 
 __author__ = """Lukas Heumos"""
 __email__ = 'lukas.heumos@posteo.net'
-__version__ = '1.10.0'
+__version__ = '1.11.0'

mlf_core/create/domains/mlflow_creator.py

@@ -30,7 +30,6 @@ def __init__(self):
         self.MLFLOW_PYTORCH_TEMPLATE_VERSION = load_mlf_core_template_version('mlflow-pytorch', self.AVAILABLE_TEMPLATES_PATH)
         self.MLFLOW_TENSORFLOW_TEMPLATE_VERSION = load_mlf_core_template_version('mlflow-tensorflow', self.AVAILABLE_TEMPLATES_PATH)
         self.MLFLOW_XGBOOST_TEMPLATE_VERSION = load_mlf_core_template_version('mlflow-xgboost', self.AVAILABLE_TEMPLATES_PATH)
-        self.MLFLOW_XGBOOST_DASK_TEMPLATE_VERSION = load_mlf_core_template_version('mlflow-xgboost_dask', self.AVAILABLE_TEMPLATES_PATH)
 
     def create_template(self, path: Path, dot_mlf_core: dict or None):
         """
@@ -39,7 +38,7 @@ def create_template(self, path: Path, dot_mlf_core: dict or None):
 
         self.cli_struct.language = mlf_core_questionary_or_dot_mlf_core(function='select',
                                                                         question='Choose the project\'s primary framework',
-                                                                        choices=['pytorch', 'tensorflow', 'xgboost', 'xgboost_dask'],
+                                                                        choices=['pytorch', 'tensorflow', 'xgboost'],
                                                                         default='pytorch',
                                                                         dot_mlf_core=dot_mlf_core,
                                                                         to_get_property='language')
@@ -52,7 +51,6 @@ def create_template(self, path: Path, dot_mlf_core: dict or None):
             'pytorch': self.mlflow_pytorch_options,
             'tensorflow': self.mlflow_tensorflow_options,
             'xgboost': self.mlflow_xgboost_options,
-            'xgboost_dask': self.mlflow_xgboost_dask_options,
         }
         switcher.get(self.cli_struct.language)(dot_mlf_core)
 
@@ -72,7 +70,6 @@ def create_template(self, path: Path, dot_mlf_core: dict or None):
             'pytorch': self.MLFLOW_PYTORCH_TEMPLATE_VERSION,
             'tensorflow': self.MLFLOW_TENSORFLOW_TEMPLATE_VERSION,
             'xgboost': self.MLFLOW_XGBOOST_TEMPLATE_VERSION,
-            'xgboost_dask': self.MLFLOW_XGBOOST_DASK_TEMPLATE_VERSION
         }
         self.cli_struct.template_version, self.cli_struct.template_handle = switcher_version.get(
             self.cli_struct.language), f'mlflow-{self.cli_struct.language.lower()}'
@@ -91,7 +88,3 @@ def mlflow_tensorflow_options(self, dot_mlf_core: dict or None):
     def mlflow_xgboost_options(self, dot_mlf_core: dict or None):
         """ Prompts for mlflow-xgboost specific options and saves them into the MlflowTemplateStruct """
         pass
-
-    def mlflow_xgboost_dask_options(self, dot_mlf_core: dict or None):
-        """ Prompts for mlflow-xgboost_dask specific options and saves them into the MlflowTemplateStruct """
-        pass

mlf_core/create/github_support.py

@@ -235,7 +235,11 @@ def create_sync_secret(username: str, repo_name: str, token: str) -> None:
     :param token: The PAT of the user with repo scope
     """
     public_key_dict = get_repo_public_key(username, repo_name, token)
-    create_secret(username, repo_name, token, public_key_dict['key'], public_key_dict['key_id'])
+    try:
+        create_secret(username, repo_name, token, public_key_dict['key'], public_key_dict['key_id'])
+    except KeyError:
+        print('[bold red]Could not create sync secret for repository due to GitHubs API currently unavailable. See '
+              'https://mlf-core.readthedocs.io/en/latest/faq.html for more!')
 
 
 def get_repo_public_key(username: str, repo_name: str, token: str) -> dict:

mlf_core/create/templates/available_templates.yml

@@ -23,15 +23,6 @@ mlflow:
         short description: Fully CPU and GPU reproducible MLflow based XGBoost template.
         long description: A fully CPU and GPU reproducible MLflow based XGBoost template with CPU, single-GPU and
                         system-intelligence support. The initialized project contains a Covertype example.
-    xgboost_dask:
-        name: XGBoost-Dask
-        handle: mlflow-xgboost_dask
-        version: 1.2.2
-        available libraries: XGBoost, Dask
-        short description: MLflow and Dask based XGBoost template.
-        long description: A MLflow based XGBoost template with CPU, single-GPU, multi-GPU and
-                        system-intelligence support. The initialized project contains a Covertype example.
-                        Due to Dask's non-deterministic shuffling the template may not always be reproducible!
 package:
     prediction:
         name: Prediction

mlf_core/create/templates/common_mlflow_files/{{ cookiecutter.common_mlflow }}/.github/workflows/publish_docker.yml

@@ -7,26 +7,26 @@ name: Publish Container to Docker Packages
 
 on:
   release:
-      types: [published]
+    types: [published]
   push:
-      branches:
-        - development
+    branches:
+      - development
+
 jobs:
   publish_release:
     runs-on: ubuntu-latest
     steps:
       - name: Reclaim space
         run: |
-             sudo rm -rf /usr/share/dotnet
-             sudo rm -rf /opt/ghc
-
-    - uses: actions/checkout@master
-    - name: Publish to Registry
-      uses: elgohr/Publish-Docker-Github-Action@master
-      with:
-        name: {{ cookiecutter.github_username }}/{{ cookiecutter.project_slug_no_hyphen }}
-        username: {{ cookiecutter.github_username }}
-        password: {% raw %}'${{ secrets.MLF_CORE_SYNC_TOKEN}}'{% endraw %}
-        registry: ghcr.io
-        tags: "latest,{{ cookiecutter.version }}"
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /opt/ghc
 
+      - uses: actions/checkout@master
+      - name: Publish to Registry
+        uses: elgohr/Publish-Docker-Github-Action@master
+        with:
+          name: {{ cookiecutter.github_username }}/{{ cookiecutter.project_slug_no_hyphen }}
+          username: {{ cookiecutter.github_username }}
+          password: {% raw %}'${{ secrets.MLF_CORE_SYNC_TOKEN}}'{% endraw %}
+          registry: ghcr.io
+          tags: "latest,{{ cookiecutter.version }}"