Add Dockerfiles and helper scripts (#1511)

* Add official Dockerfiles

* Install locales package for UI

* Add jupyter notebook images

* Add interactive script

* Fix jupyter launcher

* Remove outdated helper scripts

* Update docs

* Fix missing bind for notebook mode

* Fix default password setting for notebook

* Add helper functions for packaging

* Build python package for helper scripts

* Update Dockerfile.base

* Fix missing quotes in pip install

* Pin package versions: numpy, tf, decorator

* Consistent naming UI -> GUI

* Remove unused log file

* Add documentation for docker images

* Adapt docker developer docs

* Add examples/ to jupyter image, minor fixes

* Allow to run docker in sudo mode (previously yielded error message)

* Add a "custom" mode for developers in deeplabcut_docker.sh

* Update PyPI package

* Fix _build() in deeplabcut_docker.sh

* Add experimental macOS support; variable nb port

* Update PyPI package

* Update docker.md

* Update installation.md

* Update docs/installation.md

* Update docs/installation.md

* fix older mistake

* Add maintainer information for deeplabcut-docker

* Update _toc.yml

* Update README.md

* Update PyPI description with current docs

* Exclude docker group error message for macOSX

* Update deeplabcut_docker.py

* Fix if user group exists in container

* Final fixes for MacOS containers

* Remove debug comment

_toc.yml

@@ -4,8 +4,9 @@ parts:
 - caption: Main Documentation
   chapters:
   - file: docs/README
+  - file: docs/UseOverviewGuide
   - file: docs/installation
-  - file: docs/UseOverViewGuide
+  - file: docs/docker
   - file: docs/standardDeepLabCut_UserGuide
   - file: docs/maDLC_UserGuide
   - file: docs/PROJECT_GUI

docker/.gitignore

@@ -0,0 +1,2 @@
+examples/
+PYPI_README.md

docker/Dockerfile.base

@@ -0,0 +1,12 @@
+FROM nvidia/cuda:11.4.0-cudnn8-runtime-ubuntu20.04
+
+RUN DEBIAN_FRONTEND=noninteractive apt-get update -yy \ 
+    && apt-get install -yy --no-install-recommends python3 python3-pip ffmpeg libsm6 libxext6 \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN pip install --no-cache-dir --upgrade pip \
+ && pip install --no-cache-dir --upgrade "deeplabcut>=2.2.0.2" numpy==1.19.5 decorator==4.4.2 tensorflow==2.5.0
+
+# TODO required to fix permission errors when running the container with limited permission.
+RUN chmod a+rwx -R /usr/local/lib/python3.8/dist-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained

docker/Dockerfile.latest-core

@@ -0,0 +1,6 @@
+FROM deeplabcut/deeplabcut:base
+
+ENV DLClight True
+
+COPY motd.sh /home/motd.sh
+RUN echo "source /home/motd.sh" >> /etc/profile

docker/Dockerfile.latest-gui

@@ -0,0 +1,15 @@
+FROM deeplabcut/deeplabcut:base
+
+RUN DEBIAN_FRONTEND=noninteractive apt-get update -yy \ 
+    && DEBIAN_FRONTEND=noninteractive \
+         apt-get install -yy --no-install-recommends \
+         libgtk-3-dev python3-wxgtk4.0 locales \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* \
+    && locale-gen en_US.UTF-8 en_GB.UTF-8
+
+RUN pip install --no-cache-dir --upgrade pip \
+ && pip install --no-cache-dir --upgrade "deeplabcut[gui]>=2.2.0.2" numpy==1.19.5 decorator==4.4.2 tensorflow==2.5.0
+
+ENV DLClight=False
+CMD ["python3", "-m", "deeplabcut"]

docker/Dockerfile.latest-gui-jupyter

@@ -0,0 +1,7 @@
+FROM deeplabcut/deeplabcut:latest-gui
+RUN pip install --no-cache-dir jupyter
+COPY ./examples /app/examples
+CMD ["jupyter", "notebook", \
+     "--no-browser", \
+     "--NotebookApp.password", "sha3_224:019dc9326f7c:689d6c568f840f6126fb438ba0cdb8ce94ddd0769399e152caf79931", \
+     "--ip", "0.0.0.0" ]

docker/LICENSE

@@ -0,0 +1,166 @@
+                   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
+

docker/MANIFEST.in

@@ -0,0 +1,4 @@
+include pyproject.toml
+include PYPI_README.md
+include LICENSE
+include deeplabcut_docker.sh

docker/Makefile

@@ -0,0 +1,16 @@
+clean:
+	rm -rf dist/
+	rm -f PYPI_README.md
+
+prepare_build:
+	python3 -m pip install --upgrade twine build
+	cp ../docs/docker.md PYPI_README.md
+
+build: clean prepare_build  
+	python3 -m build
+
+upload_test: prepare_build
+	python3 -m twine upload --repository testpypi dist/*
+
+upload: prepare_build
+	python3 -m twine upload dist/*

docker/README.md

@@ -0,0 +1,110 @@
+# DeepLabCut Dockerfiles
+
+**Note that this README is mainly intended for DeepLabCut developers. The main documentation contains its own user documentation on the provided docker images.**
+
+This repo contains build routines for the following official DeepLabCut docker images:
+- `deeplabcut/deeplabcut:base`: Base image with TF2.5, cuDNN8 and DLC
+- `deeplabcut/deeplabcut:latest-core`: DLC in light mode
+- `deeplabcut/deeplabcut:latest-gui`: DLC in GUI mode
+- `deeplabcut/deeplabcut:latest-gui-jupyter`: DLC in GUI mode, with jupyter installed
+
+All images are based on Python 3.8.
+The images are synced to DockerHub: https://hub.docker.com/r/deeplabcut/deeplabcut
+
+## Quickstart
+
+You can use the images fully standalone, without the need of cloning the DeepLabCut repo.
+A helper package called `deeplabcut-docker` is available on PyPI and can be installed by running
+
+``` bash
+pip install deeplabcut-docker
+```
+
+*Note: Advanced users can also directly download and use the `deeplabcut-docker.sh` script if this is preferred over a python helper script.*
+
+We provide docker containers for three different use cases outlined below.
+
+In all cases, your current directory will be mounted in the container, and the container
+will be started with your current username and group.
+
+- To launch the DLC GUI directly, run
+  ```bash
+  deeplabcut-docker gui
+  ```
+- Interactive console with DLC in light mode
+  ```bash
+  deeplabcut-docker bash
+  ```
+- A Jupyter notebook server can be launched with
+  ```bash
+  deeplabcut-docker notebook
+  ```
+
+## For developers
+
+Make sure your docker daemon is running and navigate to the repository root directory.
+You can build the images by running
+
+```
+docker/build.sh build
+```
+
+Note that this assumes that you have rights to execute `docker build` and `docker run` commands which requires either `sudo` access or membership in the `docker` group on your local machine. If you are not in the `docker` group, run the script with the environment variable `DOCKER="sudo docker"` set to override the default docker command.
+
+Images can be verified by running
+
+```
+docker/build.sh test
+``` 
+
+Built images can be pushed to DockerHub by running
+
+```
+docker/build.sh push
+``` 
+
+## Prerequisites (if you don't have Docker installed already)
+
+**(1)** Install Docker. See https://docs.docker.com/install/ & for Ubuntu: https://docs.docker.com/install/linux/docker-ce/ubuntu/
+Test docker: 
+
+    $ sudo docker run hello-world
+    
+ The output should be: ``Hello from Docker! This message shows that your installation appears to be working correctly.``
+
+*if you get the error ``docker: Error response from daemon: Unknown runtime specified nvidia.`` just simply restart docker: 
+  
+       $ sudo systemctl daemon-reload
+       $ sudo systemctl restart docker
+
+    
+**(2)** Add your user to the docker group (https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user)
+Quick guide  to create the docker group and add your user: 
+Create the docker group.
+
+    $ sudo groupadd docker
+Add your user to the docker group.
+
+    $ sudo usermod -aG docker $USER
+
+(perhaps restart your computer (best) or (at min) open a new terminal to make sure that you are added from now on)
+
+## Acknowledgements
+
+Ascii art in the MOTD is adapted from https://ascii.co.uk/art/mice and https://patorjk.com/software/taag/#p=display&f=Small%20Slant&t=DeepLabCut.
+
+```
+                    .--,       .--,
+                    ( (  \.---./  ) )
+                     '.__/o   o\__.'
+                       `{=  ^  =}´
+                         >  u  <
+ ____________________.""`-------`"".______________________  
+\   ___                   __         __   _____       __  /
+/  / _ \ ___  ___  ___   / /  ___ _ / /  / ___/__ __ / /_ \
+\ / // // -_)/ -_)/ _ \ / /__/ _ `// _ \/ /__ / // // __/ /
+//____/ \__/ \__// .__//____/\_,_//_.__/\___/ \_,_/ \__/  \
+\_________________________________________________________/
+                       ___)( )(___ `-.___. 
+                      (((__) (__)))      ~`
+```