This repository contains exploratory study of recent advancements of Deep Metric Learning in particular Contrastive Representation Learning.
InfoNCE (Oord et al 2019) is based on Noise Contrastive Estimation (Gutmann and Hyvarinen 2012) and aims to maximise Mutual Information of query-document pair (or as put originally, context and signal). NCE itself is a generalization of triplet loss used in FaceNet (Schroff et al 2015) to multiple negative examples.
Some intimate connection between NCE and Generative Adverserial Networks (GANs) have been discussed (Goodfellow 2015, Frosst 2019).
Soft Nearest Neighbour Loss (Frosst 2019) further generalises InfoNCE to allow multiple positive examples.
Visualisation were produced using representation of last layer of untrained ResNet18 and t-SNE.
| Cross Entropy | InfoNCE | Soft NN Loss |
|---|---|---|
 |
 |
 |
| Cross Entropy | InfoNCE | Soft NN Loss |
|---|---|---|
 |
 |
 |
poetry install
- simple top-k Retrieval model
- explore connection of GANs and NCE