Tutorial

In this tutorial, we will create a simple video classifier model. It will input video metadata and output a category prediction. Here is some info on the dataset:

https://github.com/AIAScience/deep-learning-intro/tree/master/data

Ready to start coding? Great! Clone (or download) the deep-learning-intro repository so you have a local copy. Inside the deep-learning-intro folder, create a file called tutorial.py and paste in the following code for importing the dataset:

import json
import os
import pprint

with open(os.path.join('data', 'videos.json')) as json_file:
    videos = json.load(json_file)

print('We have {} videos'.format(len(videos)))
print("Data for the first video looks like this:")
pprint.pprint(videos[0])

Now try to run the program (python tutorial.py). If you have everything set up correctly, it should print

We have 279 videos
{'relevant_topics': [],
 'tags': ['Funny',
          'smosh',
          'video',
          'Massively Multiplayer Online Role-playing Game (Video Game Genre)',
          'World Of Warcraft (Video Game)',
          'WoW',
          'elf',
          'warlords of draenor',
          'smosh games',
          'RPG',
          'honest trailer',
          'game trailer',
          'Video Game (Industry)',
          'Video Game Culture',
          'Role-playing Game (Game Genre)',
          'anthony',
          'fun',
          'ian',
          'spoof',
          'honest game trailers',
          'orc',
          'comedy',
          'smoshgames',
          'video games',
          'honest',
          'Warcraft (Fictional Universe)',
          'games',
          'MMORPG',
          'parody'],
 'target_category_id': 3,
 'title': 'WORLD OF WARCRAFT (Honest Game Trailers)',
 'topics': []}

As you can see, each video has a number of tags. For simplicity, let's use tags as our only input to the machine learning model we are going to make. First, we need to calculate a set that contains tags we are dealing with. Add the following snippet of python code to your tutorial.py:

tags = set()

for video in videos:
    for tag in video["tags"]:
        tags.add(tag)

num_tags = len(tags)

print("We have {} unique tags".format(num_tags))

This should print something like

We have 3184 unique tags

Since machine learning models work on vectors/matrices/tensors (all collections of numbers), we need to make a vector representation of each video. We can do that with binary vectors, where each position represents one tag, and the number at that position (either 0 or 1) represents the presence of that tag on the video.

tag_to_index = {tag: index for index, tag in enumerate(tags)}


def vectorize_video_input(video):
    input_vector = [0] * num_tags
    for tag in video["tags"]:
        tag_index = tag_to_index.get(tag, None)
        if tag_index is not None:
            input_vector[tag_index] = 1
    return input_vector


print('The first video input in vector form looks like this:')
print(vectorize_video_input(videos[0]))

This will print something like this:

The first video in vector form looks like this:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, ... , 0, 0, 0, 0, 0, 0, 0]

We also need to vectorize the target data. In other words, the ground truth category must be vectorized. We can do that with a one-hot-vector.

with open(os.path.join("data", "categories.json")) as json_file:
    categories = json.load(json_file)

num_categories = len(categories)

print("We have {} categories:".format(num_categories))
pprint.pprint(categories)

category_id_to_index = {
    category["id"]: index for index, category in enumerate(categories)
}
print('Category id to index in target vector:')
print(category_id_to_index)


def vectorize_video_target(video):
    target_vector = [0] * num_categories
    category_index = category_id_to_index.get(video["target_category_id"], None)
    if category_index is not None:
        target_vector[category_index] = 1
    return target_vector


print('The first video target category in vector form looks like this:')
print(vectorize_video_target(videos[0]))

This will print something like this:

We have 3 categories:
[{'id': 1, 'name': 'Cats'},
 {'id': 2, 'name': 'Magic'},
 {'id': 3, 'name': 'Funny Gaming Clips'}]
Category id to index in target vector:
{1: 0, 2: 1, 3: 2}
The first video target category in vector form looks like this:
[0, 0, 1]

Next, let's vectorize all videos and put the vectors in numpy arrays. Numpy arrays are typically what ML models expect.

input_vectors = [vectorize_video_input(video) for video in videos]
target_vectors = [vectorize_video_target(video) for video in videos]

import numpy as np

input_vectors = np.array(input_vectors)
target_vectors = np.array(target_vectors)

Next, split up the dataset into two groups: Training data and validation data. Read about cross-validation here:

https://en.wikipedia.org/wiki/Cross-validation_(statistics)#Holdout_method

training_fraction = 0.8  # use e.g. 80 % of data for training, 20 % for validation
split_index = int(len(input_vectors) * training_fraction)
training_input_vectors = input_vectors[0:split_index]
training_target_vectors = target_vectors[0:split_index]
validation_input_vectors = input_vectors[split_index:]
validation_target_vectors = target_vectors[split_index:]

Set up your neural network (machine learning model):

from keras.layers.core import Dense
from keras.models import Sequential
from keras.optimizers import SGD

num_hidden_nodes = 20
model = Sequential()
model.add(Dense(num_hidden_nodes, input_dim=num_tags, activation="relu"))
model.add(Dense(num_categories, activation="softmax"))

model.compile(
    loss="categorical_crossentropy", optimizer=SGD(momentum=0.9), metrics=["accuracy"]
)

Train the model:

model.fit(training_input_vectors, training_target_vectors, epochs=100)

Finally, evaluate the accuracy of your model on the training set and the validation set:

evaluation_scores = model.evaluate(validation_input_vectors, validation_target_vectors)

for i, metric_name in enumerate(model.metrics_names):
    print("Validation {}: {:.3f}".format(metric_name, evaluation_scores[i]))

Here's an example of how we can get a prediction from the model, and compare it with the ground truth:

print('The last video (in the validation set):')
pprint.pprint(videos[-1])

output_vectors = model.predict(np.array([validation_input_vectors[-1]]))
output_vector = output_vectors[0]

print('Output vector: {}'.format(str(output_vector)))
print('Target vector: {}'.format(str(validation_target_vectors[-1])))

If the model was wrong, why do you think it failed? What can you do to make your model more robust? Discuss with other people.

When you are ready to move on, check out the Bonus assignments