Lecture 8

Review of lecture 7

• optimization: SGD+momentum, Nesterov, RMSProp, Adam
  
• Regularization: Dropout
• other recall: weight, loss

Compare CPU and GPU

balabala

?

• Numpy, problem: can't run on GPU; have to compute own gradients
• Tensorflow, could call function to calculate gradient; could determine run on gpu or cpu
• pytorch

Tensorflow

• define computational graph
• create placeholder
• forward pass, compute prediction and loss, i.e., L2 distance between y and y_predicted
• calculate gradients
• after building graph, enter a session to run the graph
• create numpy arrays to fill the place holders above
• run the graph, get loss
• train the network
• change weights from place holder to variables
• assign and update weighs using learning rate*gradients
• use optimizer, predefined loss, initializer

Tensorboard

Keras

• define model object as a sequence of layers
• define optimizer
• build the model, specify the loss function
• train the model

PyTorch

Caffe

Caffe2