Skip to content

Latest commit

 

History

History
executable file
·
75 lines (41 loc) · 1.88 KB

10_Unspervised_learning_and_Generative_models.md

File metadata and controls

executable file
·
75 lines (41 loc) · 1.88 KB

10. Unspervised learning and Generative models

[TOC]

Unsupervised learning

  • Find underlying structure in data

    • Cluster
    • Reducing the dimensionality
    • Build a better representation (word embedding)
    • Learning likelyhood function
    • Generating new samples
  • For complex data

    • Image data : capture low dimensional semantic representation
    • Text data : find fixed size, dense semantic representation of data

    Latet space might be more efficient

  • Self-supervised learning

    • Build supervision without labels
    • Use text structure to create supervision
      • Word2Vec, Skip-thought vectors, language models
    • Image : spatial context of an object
    • Sound,video : exploit temporal context

    No direct accuracy measure

Autoencoders

Keeping the latend code $z$ low-dimensional forces the network to learn a "smart" compression of the data

Encoder and decoder can have arbitrary architectures (CNNs, RNNs)

  • Sparse/Denoising Autoencoder

    • Adding a sparsity constraint on activations to learn sparse features.

      $||encoder(x)||_1 \sim p, p=0.05$

  • Uses and limitations

    • After pre-training , use the latent code z as input to a classifier
    • Semi-supervised learning
    • Use decoder $D(x)$ as a Generative model
    • Limitations
      • Not good for complex data such as images
  • Reality Check

    • ImageNet pretraining is still much better than unsupervised models

Variational Autoencoders

Genarative Adversarial Networks

Alternate training of a generative network $G$ and a discrimininative network D

  • D tries to find out which example are generated or real
  • g tries to fool D into thinking its generated examples are real