Pre-0.1 release

.gitignore

@@ -13,7 +13,15 @@ bin/copy.sh
 .vagrant/
 .DS_Store
 *.log
+*.csr
+*.crt
+*.key
 lcm/coord/tmp.etcd
 tmp
 .venv/
 .idea/
+venv/
+bin/dind_scripts/config.json
+bin/dind_scripts/go1.10.1.linux-amd64.tar.gz
+bin/dind_scripts/helm-v2.8.2-linux-amd64.tar.gz
+bin/dind_scripts/linux-amd64/

.helmignore

@@ -21,3 +21,5 @@ demos/
 samples/
 cli/
 community/
+persistentvol/
+design/

.travis.yml

@@ -31,14 +31,22 @@ script:
   # fail fast
   - set -e
   - export MAKE_ARGS=--no-print-directory
+  # Open SSH
+  #  - echo travis:$sshpassword | sudo chpasswd
+  #  - sudo sed -i 's/ChallengeResponseAuthentication no/ChallengeResponseAuthentication yes/' /etc/ssh/sshd_config
+  #  - sudo service ssh restart
+  #  - sudo apt-get install sshpass
+  #  - sshpass -p $sshpassword ssh -R 9999:localhost:22 -o StrictHostKeyChecking=no travis@$bouncehostip
   # compile and build Docker images
   - glide -q install
+  - make $MAKE_ARGS docker-build-base
+  - make $MAKE_ARGS gen-certs
   - make $MAKE_ARGS build
   - make $MAKE_ARGS docker-build
   # deploy services
   - make $MAKE_ARGS deploy
   # submit a test job
-  - make $MAKE_ARGS test-submit
+  - make $MAKE_ARGS test-submit-minikube-ci
 
 notifications:
   email: false

CONTRIBUTING.md

@@ -28,6 +28,7 @@ via code, tests, or documentation:
 * Michael Kalantar
 * Tommy Li
 * Vinod Muthusamy
+* Falk Pollok
 * Florian Rosenberg
 * Animesh Singh
 * Atin Sood

MAINTAINERS.md

@@ -7,6 +7,7 @@ The maintainers are listed in alphabetical order.
 - Scott Boag ([sboagibm](https://github.com/sboagibm))
 - Waldemar Hummer ([whummer](https://github.com/whummer))
 - Tommy Li ([Tomcli](https://github.com/Tomcli))
+- Falk Pollok ([fplk](https://github.com/fplk))
 - Animesh Singh ([animeshsingh](https://github.com/animeshsingh))
 
 ## Methodology

README.md

@@ -14,7 +14,7 @@ FfDL is a collaboration platform for:
 
 ![ffdl-architecture](docs/images/ffdl-architecture.png)
 
-To know more about the architectural details, please read [here](design/design_docs.md)
+To know more about the architectural details, please read the [design document](design/design_docs.md).
 
 ## Prerequisites
 
@@ -26,12 +26,12 @@ To know more about the architectural details, please read [here](design/design_d
 
 * `S3 CLI`: The [command-line interface](https://aws.amazon.com/cli/) to configure your Object Storage
 
-* An existing Kubernetes cluster (e.g., [Minikube](https://github.com/kubernetes/minikube) for local testing).
-  For Minikube, use the command `make minikube` to start Minikube and set up local network routes. Minikube **v0.25.1** is tested with Travis CI.
+* An existing Kubernetes cluster (e.g., [Kubeadm-DIND](https://github.com/kubernetes-sigs/kubeadm-dind-cluster#using-preconfigured-scripts) for local testing).
+  <!-- For Minikube, use the command `make minikube` to start Minikube and set up local network routes. Minikube **v0.25.1** is tested with Travis CI. -->
 
 * Follow the appropriate instructions for standing up your Kubernetes cluster using [IBM Cloud Public](https://github.com/IBM/container-journey-template/blob/master/README.md) or [IBM Cloud Private](https://github.com/IBM/deploy-ibm-cloud-private/blob/master/README.md)
 
-* The minimum recommended capacity for FfDL is 4GB Memory and 3 CPUs.
+* The minimum capacity requirement for FfDL is 4GB Memory and 3 CPUs.
 
 ## Usage Scenarios
 
@@ -46,9 +46,8 @@ To know more about the architectural details, please read [here](design/design_d
 ## Steps
 
 1. [Quick Start](#1-quick-start)
-  - 1.1 [Installation using Minikube](#11-installation-using-minikube)
+  - 1.1 [Installation using Kubeadm-DIND](#11-installation-using-kubeadm-dind)
   - 1.2 [Installation using Kubernetes Cluster](#12-installation-using-kubernetes-cluster)
-  - 1.3 [Installation using IBM Cloud Kubernetes Cluster](#13-installation-using-ibm-cloud-kubernetes-cluster)
 2. [Test](#2-test)
 3. [Monitoring](#3-monitoring)
 4. [Development](#4-development)
@@ -62,47 +61,42 @@ To know more about the architectural details, please read [here](design/design_d
 
 ## 1. Quick Start
 
-There are multiple installation paths for installing FfDL locally ("1-click-install") or into an existing Kubernetes cluster.
+There are multiple installation paths for installing FfDL locally ("1-click-install") or into an existing Kubernetes cluster. You can visit [Step 5](#5-detailed-installation-instructions) for more details on the deployment instructions.
 
-> Note: If your Kubernetes Cluster version is 1.7 or below, please go to the [values.yaml](values.yaml) and change `k8s_1dot8_or_above` to **false**.
+### 1.1 Installation using Kubeadm-DIND
 
-### 1.1 Installation using Minikube
-
-If you have Minikube installed on your machine, use these commands to deploy the FfDL platform:
+If you have [Kubeadm-DIND](https://github.com/kubernetes-sigs/kubeadm-dind-cluster#using-preconfigured-scripts) installed on your machine, use these commands to deploy the FfDL platform:
 ``` shell
-export VM_TYPE=minikube
-make minikube
-make deploy
+export VM_TYPE=dind
+export PUBLIC_IP=localhost
+export SHARED_VOLUME_STORAGE_CLASS="";
+make deploy-plugin
+make quickstart-deploy
 ```
 
 ### 1.2 Installation using Kubernetes Cluster
 
-To install FfDL to a proper Kubernetes cluster, make sure `kubectl` points to the right namespace,
+To install FfDL to any proper Kubernetes cluster, make sure `kubectl` points to the right namespace,
 then deploy the platform services:
-> Note: For PUBLIC_IP, put down one of your Cluster Public IP that can access your Cluster's NodePorts.
+> Note: For PUBLIC_IP, put down one of your Cluster Public IP that can access your Cluster's NodePorts. For IBM Cloud, you can get your Public IP with `bx cs workers <cluster_name>`.
 
 ``` shell
 export VM_TYPE=none
 export PUBLIC_IP=<Cluster Public IP>
-make deploy
-```
 
-### 1.3 Installation using IBM Cloud Kubernetes Cluster
-
-To install FfDL to a proper IBM Cloud Kubernetes cluster, make sure `kubectl` points to the right namespace
-and your machine is logged in with `bx login`, then deploy the platform services:
-``` shell
-export VM_TYPE=ibmcloud
-export CLUSTER_NAME=<Your Cluster Name> # Replace <Your Cluster Name> with your IBM Cloud Cluster Name
-make deploy
+# Change the storage class to what's available on your Cloud Kubernetes Cluster.
+export SHARED_VOLUME_STORAGE_CLASS="ibmc-file-gold";
+make deploy-plugin
+make quickstart-deploy
 ```
 
 ## 2. Test
 
 To submit a simple example training job that is included in this repo (see `etc/examples` folder):
 
-```
-make test-submit
+``` shell
+make test-push-data-s3
+make test-job-submit
 ```
 
 ## 3. Monitoring
@@ -115,6 +109,9 @@ Please refer to the [developer guide](docs/developer-guide.md) for more details.
 
 ## 5. Detailed Installation Instructions
 
+0. If you don't have a Kubernetes Cluster, you can create a [Kubeadm-DIND](https://github.com/kubernetes-sigs/kubeadm-dind-cluster#using-preconfigured-scripts) Kubernetes Cluster on your local machine. We recommend you give at least 4 CPUs and 8GB of memory to your Docker.
+> For Mac users, visit the instructions on the [Docker website](https://docs.docker.com/docker-for-mac/#advanced) and learn how to give more memory to your Docker.
+
 1. First, clone this repository and install the helm tiller on your Kubernetes cluster.
 ``` shell
 helm init
@@ -124,11 +121,46 @@ kubectl get pods --all-namespaces | grep tiller-deploy
 # kube-system   tiller-deploy-fb8d7b69c-pcvc2              1/1       Running
 ```
 
-2. Now let's install all the necessary FfDL components using helm install.
-> Note: If your Kubernetes Cluster version is 1.7 or below, please go to the [values.yaml](values.yaml) and change `k8s_1dot8_or_above` to **false**.
+2. Define the necessary environment variables.
+  * 2.a. For Kubeadm-DIND Cluster only
+  ```shell
+  export FFDL_PATH=$(pwd)
+  export SHARED_VOLUME_STORAGE_CLASS=""
+  ```
+
+  * 2.b. For Cloud Kubernetes Cluster
+  ```shell
+  # Change the storage class to what's available on your Cloud Kubernetes Cluster.
+  export SHARED_VOLUME_STORAGE_CLASS="ibmc-file-gold"
+  ```
+
+3. Install the Object Storage driver using helm install.
+  * 3.a. For Kubeadm-DIND Cluster only
+  ```shell
+  export FFDL_PATH=$(pwd)
+  ./bin/s3_driver.sh
+  helm install storage-plugin --set dind=true,cloud=false
+  ```
+
+  * 3.b. For Cloud Kubernetes Cluster
+  ```shell
+  helm install storage-plugin
+  ```
+
+4. Create a static volume to store any metadata from FfDL.
+
+```shell
+pushd bin
+./create_static_volumes.sh
+./create_static_volumes_config.sh
+# Wait while kubectl get pvc shows static-volume-1 in state Pending
+popd
+```
+
+5. Now let's install all the necessary FfDL components using helm install.
 
 ``` shell
-helm install .
+helm install . --set lcm.shared_volume_storage_class=$SHARED_VOLUME_STORAGE_CLASS
 ```
 > Note: If you want to upgrade an older version of FfDL, run
 > `helm upgrade $(helm list | grep ffdl | awk '{print $1}' | head -n 1) .`
@@ -153,32 +185,44 @@ helm status $(helm list | grep ffdl | awk '{print $1}' | head -n 1) | grep STATU
 # STATUS: DEPLOYED
 ```
 
-3. Run the following script to configure Grafana for monitoring FfDL using the logging information from prometheus.
-> Note: If you are using a IBM Cloud Cluster, make sure you are logged in with `bx login`.
+6. Obtain the necessary port for Grafana, FfDL Web UI, local object storage, and FfDL restapi.
+```shell
+grafana_port=$(kubectl get service grafana -o jsonpath='{.spec.ports[0].nodePort}')
+ui_port=$(kubectl get service ffdl-ui -o jsonpath='{.spec.ports[0].nodePort}')
+restapi_port=$(kubectl get service ffdl-restapi -o jsonpath='{.spec.ports[0].nodePort}')
+s3_port=$(kubectl get service s3 -o jsonpath='{.spec.ports[0].nodePort}')
+```
+
+* For Kubeadm-DIND Cluster, we need to run the below script to forward the port to the localhost machine since we don't want to exec into the docker image and install various dependencies.
+  ```shell
+  ./bin/dind-port-forward.sh
+  ```
 
+7. Run the following commands to configure Grafana for monitoring FfDL using the logging information from prometheus.
+  * 7.a. For Kubeadm-DIND Cluster only
   ```shell
-# If your Cluster is running on Minikube, replace "ibmcloud" to "minikube"
-# If your Cluster is not running on Minikube or IBM Cloud, replace "ibmcloud" to "none"
-export VM_TYPE=ibmcloud
+  export VM_TYPE=none
+  export PUBLIC_IP=localhost
+
+  ./bin/grafana.init.sh
+  ```
 
-# Replace <Your Cluster Name> with your IBM Cloud Cluster Name if your cluster is on IBM Cloud.
-# Use export PUBLIC_IP if you are using a none VM_TYPE. A Cluster Public IP that can access your Cluster's NodePorts.
-export CLUSTER_NAME=<Your Cluster Name>
+
+  * 7.b. For Cloud Kubernetes Cluster.
+  > Note: If you are using IBM Cloud Cluster, you can obtain your k8s public ip using `bx cs workers <cluster-name>`.
+
+  ``` shell
+  export VM_TYPE=none
   export PUBLIC_IP=<Cluster Public IP>
 
   ./bin/grafana.init.sh
   ```
 
-4. Lastly, run the following commands to obtain your Grafana, FfDL Web UI, and FfDL restapi endpoints.
+8. Lastly, run the following commands to obtain your Grafana, FfDL Web UI, and FfDL restapi endpoints.
 ``` shell
 # Note: $(make --no-print-directory kubernetes-ip) simply gets the Public IP for your cluster.
 node_ip=$(make --no-print-directory kubernetes-ip)
 
-# Obtain all the necessary NodePorts for Grafana, Web UI, and RestAPI.
-grafana_port=$(kubectl get service grafana -o jsonpath='{.spec.ports[0].nodePort}')
-ui_port=$(kubectl get service ffdl-ui -o jsonpath='{.spec.ports[0].nodePort}')
-restapi_port=$(kubectl get service ffdl-restapi -o jsonpath='{.spec.ports[0].nodePort}')
-
 # Echo statements to print out Grafana and Web UI URLs.
 echo "Monitoring dashboard: http://$node_ip:$grafana_port/ (login: admin/admin)"
 echo "Web UI: http://$node_ip:$ui_port/#/login?endpoint=$node_ip:$restapi_port&username=test-user"
@@ -191,8 +235,6 @@ Congratulation, FfDL is now running on your Cluster. Now you can go to [Step 6](
 In this example, we will run some simple jobs to train a convolutional network model using TensorFlow and Caffe. We will download a set of
 MNIST handwritten digit images, store them with Object Storage, and use the FfDL CLI to train a handwritten digit classification model.
 
-> Note: For Minikube, make sure you have the latest TensorFlow Docker image by running `docker pull tensorflow/tensorflow`
-
 ### 6.1. Using FfDL Local S3 Based Object Storage
 
 1. Run the following commands to obtain the object storage endpoint from your cluster.
@@ -232,8 +274,19 @@ binary).
 ```shell
 restapi_port=$(kubectl get service ffdl-restapi -o jsonpath='{.spec.ports[0].nodePort}')
 export DLAAS_URL=http://$node_ip:$restapi_port; export DLAAS_USERNAME=test-user; export DLAAS_PASSWORD=test;
+```
 
-# Obtain the correct CLI for your machine and run the training job with our default TensorFlow model
+Replace the default object storage path with your s3_url. You can skip this step if your already modified the object storage path with your s3_url.
+```shell
+if [ "$(uname)" = "Darwin" ]; then
+  sed -i '' s#"http://s3.default.svc.cluster.local"#"$s3_url"# etc/examples/tf-model/manifest.yml
+else
+  sed -i s#"http://s3.default.svc.cluster.local"#"$s3_url"# etc/examples/tf-model/manifest.yml
+fi
+```
+
+Define the FfDL command line interface and run the training job with our default TensorFlow model
+```shell
 CLI_CMD=$(pwd)/cli/bin/ffdl-$(if [ "$(uname)" = "Darwin" ]; then echo 'osx'; else echo 'linux'; fi)
 $CLI_CMD train etc/examples/tf-model/manifest.yml etc/examples/tf-model
 ```
@@ -284,8 +337,6 @@ you can simply run `$CLI_CMD logs <MODEL_ID>`
 
 In this section we will demonstrate how to run a TensorFlow job with training data stored in Cloud Object Storage.
 
-> Note: This also can be done with other Cloud providers' Object Storage, but we will demonstrate how to use IBM Cloud Object Storage in this instructions.
-
 1. Provision an S3 based Object Storage from your Cloud provider. Take note of your Authentication Endpoints, Access Key ID and Secret.
 
 > For IBM Cloud, you can provision an Object Storage from [IBM Cloud Dashboard](https://console.bluemix.net/catalog/infrastructure/cloud-object-storage?taxonomyNavigation=apps) or from [SoftLayer Portal](https://control.softlayer.com/storage/objectstorage).
@@ -351,16 +402,27 @@ $CLI_CMD train etc/examples/tf-model/manifest.yml etc/examples/tf-model
 ```
 
 ## 7. Clean Up
-If you want to remove FfDL from your cluster, simply use the command below or run `helm delete <your FfDL release name>`
+If you want to remove FfDL from your cluster, simply use the following commands.
 ```shell
 helm delete $(helm list | grep ffdl | awk '{print $1}' | head -n 1)
 ```
 
+If you want to remove the storage driver and pvc from your cluster, run:
+```shell
+kubectl delete pvc static-volume-1
+helm delete $(helm list | grep ibmcloud-object-storage-plugin | awk '{print $1}' | head -n 1)
+```
+
+For Kubeadm-DIND, you need to kill your forwarded ports. Note that the below command will kill all the ports that are created with `kubectl`.
+```shell
+kill $(lsof -i | grep kubectl | awk '{printf $2 " " }')
+```
+
 ## 8. Troubleshooting
 
 * FfDL has only been tested under Mac OS and Linux
 
-* The default Minikube driver under Mac OS is VirtualBox, which is known for having issues with networking.
+<!-- * The default Minikube driver under Mac OS is VirtualBox, which is known for having issues with networking.
   We generally recommend Mac OS users to install Minikube using the xhyve driver.
 
 * Also, when testing locally with Minikube, make sure to point the `docker` CLI to Minikube's Docker daemon:
@@ -369,7 +431,7 @@ helm delete $(helm list | grep ffdl | awk '{print $1}' | head -n 1)
    eval $(minikube docker-env)
    ```
 * If you run into DNS name resolution issues using Minikube, make sure that the system uses only `10.0.0.10`
-  as the single nameserver. Using multiple nameservers can result in problems, in particular under Mac OS.
+  as the single nameserver. Using multiple nameservers can result in problems, in particular under Mac OS. -->
 
 * If `glide install` fails with an error complaining about non-existing paths (e.g., "Without src, cannot continue"),
   make sure to follow the standard Go directory layout (see [Prerequisites section]{#Prerequisites}).
@@ -378,6 +440,8 @@ helm delete $(helm list | grep ffdl | awk '{print $1}' | head -n 1)
 
 * When using the FfDL CLI to train a model, make sure your directory path doesn't have slashes `/` at the end.
 
+* If your job is stuck in pending stage, you can try to redeploy the plugin with `helm install storage-plugin --set dind=true,cloud=false` for Kubeadm-DIND and `helm install storage-plugin` for general Kubernetes Cluster. Also, double check your training job manifest file to make sure you have the correct object storage credentials.
+
 ## 9. References
 
 Based on IBM Research work in Deep Learning.

bin/addnew_march_15_2018.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+
+git add lcm/lcmconfig/lcmconfig_test.go
+git add lcm/service/lcm/helper/
+git add lcm/service/lcm/learner/
+git add lcm/service/lcm/nonSplitTraining.go
+git add lcm/service/lcm/policies/
+git add lcm/service/lcm/splitTraining.go
+git add metrics/log_collectors/tensorboard/Dockerfile
+git add metrics/log_collectors/training_data_service_client/match_log_file.py
+git add metrics/log_collectors/training_data_service_client/scan_log_dirs.py
+git add metrics/log_collectors/training_data_service_client/states.py
+git add metrics/log_collectors/training_data_service_client/training_data_buffered.py
+git add trainer/glide.yaml
+git add trainer/trainer/mongo.go
+git add trainer/trainer/queue.go
+git add trainer/trainer/queue_test.go