Skip to content
Permalink
Browse files

Primarily installation instructions and readme cleanup (#37)

Primarily installation instructions and readme cleanup, and update the cloud sdk version in the DockerFile. 
Expose Flask port in docker image.
  • Loading branch information
amygdala committed Jan 10, 2017
1 parent bf067ac commit 41f0f72e6584d2f1f4cdb6b3ac0c03edd0b4d30e
@@ -19,6 +19,8 @@
- [Transfer learning example](#transfer-learning-example)
- [Optional: Clone/Download the TensorFlow repo from GitHub](#optional-clonedownload-the-tensorflow-repo-from-github)

* * *

You can set up for the workshop in two different, mutually-exclusive ways:

- [Running in a docker container](#docker-based-installation).
@@ -64,8 +66,8 @@ Create a directory (called, say, `workshop-data`) to mount as a volume when you
Once you've downloaded the container image, you can run it like this:

```sh
$ docker run -v `pwd`/workshop-data:/root/tensorflow-workshop-master/workshop-data -it \
-p 6006:6006 -p 8888:8888 gcr.io/google-samples/tf-workshop:v5
docker run -v `pwd`/workshop-data:/root/tensorflow-workshop-master/workshop-data -it \
-p 6006:6006 -p 8888:8888 -p 5000:5000 gcr.io/google-samples/tf-workshop:v5
```

Edit the path to the directory you're mounting as appropriate. The first component of the `-v` arg is the local directory, and the second component is where you want to mount it in your running container.
@@ -103,27 +105,48 @@ $ docker exec -it <container_id> bash

### Running the Docker container on a VM

It is easy to set up a Google Compute Engine (GCE) VM in which to run the Docker container.
It is easy to set up a Google Compute Engine (GCE) VM on which to run the Docker container. We sketch the steps below, or see [TLDR_CLOUD_INSTALL.md](TLDR_CLOUD_INSTALL.md) for more detail.

First, make sure that your project has the GCE API enabled. An easy way to do this is to go to the [Cloud Console](https://console.cloud.google.com/), and visit the Compute Engine panel. It should display a button to enable the API.

- Connect to your project's [Cloud Shell](https://cloud.google.com/shell/).
- From the Cloud Shell, create a container-optimized image as described [here](https://cloud.google.com/container-optimized-os/docs/how-to/create-configure-instance).
- SSH to that image. You can do this from the Cloud Console by visiting the Compute Engine panel, and clicking on the 'SSH' pulldown to the right of your instance.

- From the Cloud Shell, create a container-optimized image as described [here](https://cloud.google.com/container-optimized-os/docs/how-to/create-configure-instance). E.g.:

```
gcloud compute instances create mlworkshop \
--image-family gci-stable \
--image-project google-containers \
--zone us-central1-b --boot-disk-size=100GB \
--machine-type n1-standard-1
```

- Set up a default network firewall rule for: tcp:8888, tcp:6006, and tcp:5000. (This opens the ports we'll use for jupyter, Tensorboard, and Flask.) E.g:

```shell
gcloud compute firewall-rules create mlworkshop --allow tcp:8888,tcp:6006,tcp:5000
```

- SSH to your image. You can do this from the Cloud Console by visiting the Compute Engine panel, and clicking on the 'SSH' pulldown to the right of your instance.

Then, once you've ssh'd to the VM, follow the instructions above to download and run the Docker container there.
Note: **Docker is already installed** on the 'container-optimized' VMs.

When you're done with the workshop, you may want to *stop* or *delete* your VM instance.

You can also delete your firewall rule:

```shell
gcloud compute firewall-rules delete mlworkshop
```

* * *
## Virtual environment-based installation

(These steps are not necessary if you have already completed the instructions for running the Docker image.)

We highly recommend that you use a virtual environment for your TensorFlow installation rather than a direct install onto your machine. The instructions below walk you through a `conda` install, but a `virtualenv` environment will work as well.

Note: The 'preprocessing' stage in the [Cloud ML transfer learning](workshop_sections/transfer_learning/cloudml)
example requires installation of the Cloud ML SDK, which requires Python 2.7. Otherwise, Python 3 should likely work.

### Install Conda + Python 2.7 to use as your local virtual environment

Anaconda is a Python distribution that includes a large number of standard numeric and scientific computing packages. Anaconda uses a package manager called "conda" that has its own environment system similar to Virtualenv.
@@ -196,32 +219,14 @@ Because we have limited workshop time, we've saved a set of
generated as part of the [Cloud ML transfer learning](workshop_sections/transfer_learning/cloudml)
example. To save time, copy them now to your own bucket as follows.

Copy a zip of the generated records to some directory on your local machine:

```shell
gsutil cp gs://oscon-tf-workshop-materials/transfer_learning/cloudml/hugs_preproc_tfrecords.zip .
```

and then expand the zip:

```shell
unzip hugs_preproc_tfrecords.zip
```

Set the `BUCKET` variable to point to your GCS bucket (replacing `your-bucket-name` with the actual name):
Set the `BUCKET` variable to point to your GCS bucket (replacing `your-bucket-name` with the actual name), then copy the records to your bucket. Then, set the GCS_PATH variable to the newly copied subdir.

```shell
BUCKET=gs://your-bucket-name
```

Then set the `GCS_PATH` variable as follows, and copy the unzipped records to a `preproc` directory under that path:

```shell
gsutil cp -r gs://tf-ml-workshop/transfer_learning/hugs_preproc_tfrecords $BUCKET
GCS_PATH=$BUCKET/hugs_preproc_tfrecords
gsutil cp -r hugs_preproc_tfrecords/ $GCS_PATH/preproc
```

Once you've done this, you can delete the local zip and `hugs_preproc_tfrecords` directory.

## Optional: Clone/Download the TensorFlow repo from GitHub

@@ -7,10 +7,11 @@ The accompanying slides are

Contributions are not currently accepted. This is not an official Google product.

This document points to more information for each step in the workshop.
This document points to more information for each workshop lab.

- [Installation](INSTALL.md)
- [Building a small starter TensorFlow graph](workshop_sections/starter_tf_graph/README.md)
- [XOR: A minimal training example](workshop_sections/xor/README.md)
- [Introducing MNIST, and building a simple linear classifier in TensorFlow](workshop_sections/mnist_series/01_README_mnist_simple.md).
- [Using TensorFlow's high-level APIs to build an MNIST DNN Classifier, and introducing TensorBoard](workshop_sections/mnist_series/02_README_mnist_tflearn.md).
- [Building the same classifier using TensorFlow's "low-level" APIs](workshop_sections/mnist_series/the_hard_way).
@@ -0,0 +1,148 @@

# TensorFlow on Cloud Machine Learning: Setup in a cloud-based environment

If you’re unfamiliar with Google Cloud Platform, and/or would rather do the exercises in this workshop in a cloud environment, follow these steps for initial setup.

Alternative setup flows for local installation can be found [here](INSTALL.md).

## Initial setup

### 1. Set Up Your GCP Project

#### 1.1 Create a Cloud Platform project

Create a Google Cloud Platform (GCP) account by [signing up for the free trial](https://cloud.google.com/free-trial/).
You will be asked to enter a credit card number, but you will get $300 of credits, and won't be billed.

If you already have an account, and have run through your trial credits, see one of the workshop instructors.

#### 1.2 Enable the necessary APIs

1. Click on the “hamburger” menu at upper-left, and then “API Manager”.
1. Click on “+ Enable API” (top-middle of page).
1. Enter “Compute Engine API” in the search box and click it when it appears in the list of results.
1. Click on “Enable” (top-middle of page).
1. Repeat steps 2 through 4 for: “Cloud Machine Learning” and “Dataflow”.

![Hamburger menu](./assets/hamburger.png)

![API Manager](./assets/api_manager.png)


### 2. Connect to your project's Cloud Shell

Click on the Cloud Shell icon (leftmost icon in the set of icons at top-right of the page).

![Cloud Shell](./assets/cloudshell.png)


Run commands 3-6 below in the Cloud Shell.

### 3. Initialize Cloud ML for your project

```shell
gcloud beta ml init-project
```

### 4. Set up your Cloud Storage Bucket

```shell
PROJECT_ID=$(gcloud config list project --format "value(core.project)")
BUCKET_NAME=${PROJECT_ID}-ml
gsutil mb -l us-central1 gs://$BUCKET_NAME
```

### 5. Create a container-optimized image in GCE

```shell
gcloud compute instances create mlworkshop \
--image-family gci-stable \
--image-project google-containers \
--zone us-central1-b --boot-disk-size=100GB \
--machine-type n1-standard-1
```

### 6. Set up a firewall rule for your project that will allow access to the web services we will run

```shell
gcloud compute firewall-rules create mlworkshop --allow tcp:8888,tcp:6006,tcp:5000
```

### 7. SSH into the new GCE instance, in a new browser window

- Click on the “hamburger” menu at upper-left, and then “Compute Engine”
- Find your instance in the list (mid-page) and click on the “SSH” pulldown menu on the right. Select “Open in browser window”.
- A new browser window will open, with a command line into your GCE instance.

### 8. Start the Docker container in the GCE image (in the newly opened SSH browser window):

```shell
docker pull gcr.io/google-samples/tf-workshop:v5
mkdir workshop-data
docker run -v `pwd`/workshop-data:/root/tensorflow-workshop-master/workshop-data -it \
-p 6006:6006 -p 8888:8888 -p 5000:5000 gcr.io/google-samples/tf-workshop:v5
```

Then, run the following in the Docker container.

### 9. Configure the Docker container. You’ll need your project ID for this step.

```shell
gcloud config set project <your-project-ID>
PROJECT_ID=$(gcloud config list project --format "value(core.project)")
BUCKET=gs://${PROJECT_ID}-ml
```

```shell
gcloud auth login
```
(and follow the subsequent instructions)

```shell
gcloud beta auth application-default login
```
(and follow the subsequent instructions)

### 10. Copy some data for the 'transfer learning' example to your bucket:

```shell
gsutil cp -r gs://tf-ml-workshop/transfer_learning/hugs_preproc_tfrecords $BUCKET
GCS_PATH=$BUCKET/hugs_preproc_tfrecords
```

## If you need to restart the container later

If you later exit your container and then want to restart it again, you can find the container ID by running the following in your VM:

```shell
docker ps -a
docker start <container_id>
```
Once the workshop container is running again, you can exec back into it like this:

```shell
docker exec -it <container_id> bash
```

## Cleanup

Once you’re done with your VM, you can stop or delete it. If you think you might return to it later, you might prefer to just stop it. (A stopped instance does not incur charges, but all of the resources that are attached to the instance will still be charged). You can do this from the [cloud console](https://console.cloud.google.com), or via command line from the Cloud Shell as follows:

```shell
gcloud compute instances delete --zone us-central1-b mlworkshop
```
Or:

```shell
gcloud compute instances stop --zone us-central1-b mlworkshop
```
Then later:

```shell
gcloud compute instances start --zone us-central1-b mlworkshop
```
Delete the firewall rule as well:

```shell
gcloud compute firewall-rules delete mlworkshop
```
BIN +47.1 KB assets/api_manager.png
Binary file not shown.
BIN +18.4 KB assets/cloudshell.png
Binary file not shown.
Binary file not shown.
@@ -26,7 +26,7 @@ RUN pip install --upgrade --force-reinstall https://storage.googleapis.com/cloud

# RUN python -c "import nltk; nltk.download('punkt')"

RUN curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-132.0.0-linux-x86_64.tar.gz | tar xvz
RUN curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-138.0.0-linux-x86_64.tar.gz | tar xvz
RUN ./google-cloud-sdk/install.sh -q
RUN ./google-cloud-sdk/bin/gcloud components install beta

@@ -36,5 +36,7 @@ ADD download_git_repo.py download_git_repo.py
EXPOSE 6006
# IPython
EXPOSE 8888
# Flask
EXPOSE 5000

CMD ["sh", "-c", "python download_git_repo.py ; /bin/bash"]
@@ -11,6 +11,8 @@ This directory contains the workshop labs.
- [Building custom `Estimator`s for a version of MNIST that uses CNNs](mnist_series/mnist_cnn/README.md).
- [Running distributed MNIST using Cloud ML](mnist_series/cloudml).
- [Transfer learning: using a trained model to 'bootstrap' learning new classifications](transfer_learning/README.md).
+ [Using Cloud ML](transfer_learning/cloudml)
+ [Using a custom Estimator](transfer_learning/TF_Estimator)
- [Building a word2vec model using a Custom Estimator, and exploring the learned embeddings](word2vec/README.md). Introducing [TFRecords](https://www.tensorflow.org/versions/r0.11/api_docs/python/python_io.html#data-io-python-functions).

In addition, there is an [extras](extras/README.md) directory, that contains some older labs not currently used in this workshop, but which may be of interest.
@@ -1,6 +1,18 @@

# Classifying your own images using *transfer learning*

- [The "hugs/not-hugs" image classification task](#the-hugsnot-hugs-image-classification-task)
- [1. Image Preprocessing](#1-image-preprocessing)
- [1.1 Deploy the preprocessing job to Cloud Dataflow](#11-deploy-the-preprocessing-job-to-cloud-dataflow)
- [2. Modeling: Training the classifier](#2-modeling-training-the-classifier)
- [2.1 For the workshop, use pre-generated TFRecords for training](#21-for-the-workshop-use-pre-generated-tfrecords-for-training)
- [2.2 Run the training script](#22-run-the-training-script)
- [2.3 Monitor the training](#23-monitor-the-training)
- [3. Prediction: Using the trained model](#3-prediction-using-the-trained-model)
- [3.1 Prediction from the command line using gcloud](#31-prediction-from-the-command-line-using-gcloud)
- [3.2 Prediction using the Cloud ML API: A prediction web server](#32-prediction-using-the-cloud-ml-api-a-prediction-web-server)
- [Appendix: Running training locally](#appendix-running-training-locally)

The [Google Vision API](https://cloud.google.com/vision/) is great for identifying labels, or categories, for a given
image. However, sometimes you want to further classify your own images, in more specialized categories that the Google
Vision API hasn't been trained on.

0 comments on commit 41f0f72

Please sign in to comment.
You can’t perform that action at this time.