Adding script to setup GPU usage (#82)

* Adding script to setup GPU usage with Tensorflow 2.4

Co-authored-by: Temesghen Kahsai <teme@gretel.ai>

.github/workflows/integration-tests.yml

@@ -17,10 +17,12 @@ jobs:
       uses: actions/setup-python@v1
       with:
         python-version: 3.6
-    - name: Install
-      run:
-        pip install -e '.[tf]'
-        pip install -r test-requirements.txt
+    - name: Install TF 
+      run:  pip install tensorflow==2.4
+    - name: Install gretel-synthetics
+      run: pip install -e .
+    - name: Install Test Requirements
+      run: pip install -r test-requirements.txt
     - name: Test
       run: pytest -s -vv --cov src --cov-report term-missing tests-integration/
     - uses: 8398a7/action-slack@v2

.github/workflows/unit-tests.yml

@@ -11,10 +11,12 @@ jobs:
       uses: actions/setup-python@v1
       with:
         python-version: 3.6
-    - name: Install
-      run:
-        pip install -e '.[tf]'
-        pip install -r test-requirements.txt
+    - name: Install TF 
+      run:  pip install tensorflow==2.4
+    - name: Install gretel-synthetics
+      run: pip install -e .
+    - name: Install Test Requirements
+      run: pip install -r test-requirements.txt
     - name: Lint
       run: |
         # stop the build if there are Python syntax errors or undefined names

README.md

@@ -30,19 +30,16 @@ Check out additional examples [here](https://github.com/gretelai/gretel-syntheti
 ## Getting Started
 
 By default, we do not install Tensorflow via pip as many developers and cloud services such as Google Colab are
-running customized versions for their hardware. If you wish to pip install Tensorflow along with gretel-synthetics,
-use the [tf] commands below instead.
+running customized versions for their hardware. 
 
 ```
-pip install -U .                     # Do not install Tensorflow by default (assuming you have built a distro for your hardware)
-pip install -U -e ".[tf]"            # Install a pinned version of Tensorflow"
+pip install -U .                    
 ```
 
 _or_
 
 ```
-pip install gretel-synthetics        # Do not install Tensorflow by default (assuming you have built a distro for your hardware)
-pip install gretel-synthetics[tf]    # Install a pinned version of Tensorflow
+pip install gretel-synthetics        
 ```
 
 _then..._
@@ -54,6 +51,24 @@ $ jupyter notebook
 
 When the UI launches in your browser, navigate to `examples/synthetic_records.ipynb` and get generating!
 
+If you want to install `gretel-synthetics` locally and use a GPU (recommended):
+
+1. Create a virtual environment (e.g. using `conda`) 
+
+```
+$ conda create --name tf --python=3.8
+```
+
+2. Activate the virtual environment
+
+```
+$ conda activate tf
+```
+
+3. Run the setup script `./setup-utils/setup-gretel-synthetics-tensorflow24-with-gpu.sh`
+
+The last step will install all the necessary software packages for GPU usage, `tensorflow=2.4` and `gretel-synthetics`. 
+Note that this script works only for Ubuntu 18.04. You might need to modify it for other OS versions.
 
 ## Overview
 

setup-utils/setup-gretel-synthetics-tensorflow24-with-gpu.sh

@@ -0,0 +1,41 @@
+#NOTE: Initialize conda (pyenv or virtualenv) before running the script!!!
+#conda create --name tf --yes python=3.8
+#conda activate tf
+
+echo 'Adding NVIDIA package repositories'
+wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
+sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
+sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
+sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
+sudo apt-get update
+
+wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
+
+sudo apt install -y ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
+sudo apt-get update
+
+echo 'Installing NVIDIA driver'
+sudo apt-get install --no-install-recommends nvidia-driver-450
+
+wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/libnvinfer7_7.1.3-1+cuda11.0_amd64.deb
+sudo apt install ./libnvinfer7_7.1.3-1+cuda11.0_amd64.deb
+sudo apt-get update
+
+echo 'Installing development and runtime libraries (~4GB)'
+sudo apt-get install --no-install-recommends \
+    cuda-11-0 \
+    libcudnn8=8.0.4.30-1+cuda11.0  \
+    libcudnn8-dev=8.0.4.30-1+cuda11.0
+
+echo 'Installing TensorRT. Requires that libcudnn8 is installed above.'
+sudo apt-get install -y --no-install-recommends libnvinfer7=7.1.3-1+cuda11.0 \
+    libnvinfer-dev=7.1.3-1+cuda11.0 \
+    libnvinfer-plugin7=7.1.3-1+cuda11.0
+
+
+echo 'Installing tensorflow=2.4 and gretel-synthetics'
+pip3 install --upgrade pip
+pip3 install tensorflow==2.4 gretel-synthetics
+
+echo 'Check if tensorflow with gpu support is installed'
+python -c "import tensorflow as tf; tf.test.gpu_device_name()"

setup.py

@@ -34,9 +34,6 @@
         'dataclasses==0.7;python_version<"3.7"',
         'loky==2.8.0'
     ],
-    extras_require={
-        'tf': ['tensorflow==2.4.0rc1']
-    },
     classifiers=[
         "Programming Language :: Python :: 3",
         "License :: OSI Approved :: Apache Software License",