Make the FLUTE code public.

PiperOrigin-RevId: 416285164
google-research · Dec 15, 2021 · c67dd2b · c67dd2b
1 parent e771fd0
commit c67dd2b
Show file tree

Hide file tree

Showing 20 changed files with 6,713 additions and 733 deletions.
diff --git a/LICENSE b/LICENSE
@@ -201,3 +201,28 @@
    See the License for the specific language governing permissions and
    limitations under the License.
 
+------------------
+
+Files: meta_dataset/data/sur_decoder.py
+
+MIT License
+
+Copyright (c) 2021 Nikita Dvornik
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/Leaderboard.ipynb b/Leaderboard.ipynb
diff --git a/README.md b/README.md
@@ -10,9 +10,8 @@ for more information and example code snippets.
 # Meta-Dataset
 
 This repository contains accompanying code for the article introducing
-Meta-Dataset, [arxiv.org/abs/1903.03096](https://arxiv.org/abs/1903.03096) and the follow-up paper that proposes the VTAB+MD merged benchmark [arxiv.org/abs/2104.02638](http://arxiv.org/abs/2104.02638). It also contains accompanying code and checkpoints for CrossTransformers,
-[https://arxiv.org/abs/2007.11498](https://arxiv.org/abs/2007.11498), a
-follow-up work which improves performance.
+Meta-Dataset, [arxiv.org/abs/1903.03096](https://arxiv.org/abs/1903.03096) and the follow-up paper that proposes the VTAB+MD merged benchmark [arxiv.org/abs/2104.02638](http://arxiv.org/abs/2104.02638). It also contains accompanying code and checkpoints for the CrossTransformers
+[https://arxiv.org/abs/2007.11498](https://arxiv.org/abs/2007.11498) and FLUTE [https://arxiv.org/abs/2105.07029](https://arxiv.org/abs/2105.07029) follow-up works, which improve performance.
 
 This code is provided here in order to give more details on the implementation
 of the data-providing pipeline, our back-bones and models, as well as the
@@ -90,6 +89,28 @@ their performance should be on-par.  The network structure is the same for all
 three models, and so they can be loaded using either of the CrossTransformer
 config files.
 
+## Learning a Universal Template for Few-shot Dataset Generalization (FLUTE)
+_Eleni Triantafillou, Hugo Larochelle, Richard Zemel, Vincent Dumoulin
+
+Few-shot Learning with a Universal TEmplate (FLUTE) is a model designed for the 
+strong generalization challenge of few-shot learning classes from unseen
+datasets. At the time of publication (ICML 2021), it achieved SOTA on
+Meta-Dataset (train-on-all). It works by leveraging the training datasets to 
+learn a 'universal template' that can be repurposed to solve diverse test tasks, 
+by appropriately 'filling in the blanks' of the template each time, with an 
+appropriate set of FiLM parameters that are learned with gradient descent in 
+each test task.
+
+Configuration files for training FLUTE, as well as the dataset classifier used
+in FLUTE's Blender network can be found in `learn/gin/default/flute.gin` and
+`learn/gin/default/flute_dataset_classifier.gin`, respectively. Configuration
+files for testing different variants of FLUTE can be found in
+`learn/gin/best/flute*` The results reported in the paper were obtained with
+`learn/gin/best/flute.gin`.
+
+The training script for FLUTE is `train_flute.py`. We also have pre-trained
+checkpoints for FLUTE and its Blender network: https://console.cloud.google.com/storage/gresearch/flute
+
 # Leaderboard (in progress)
 
 The tables below were generated by
@@ -142,14 +163,14 @@ fo-Proto-MAML [[1]]        |11.6                       |46.52±1.05&nbsp;(12)
 CNAPs [[2]]                |10.25                      |50.80±1.10&nbsp;(10.5)     |91.70±0.50&nbsp;(8)        |83.70±0.60&nbsp;(7.5)      |73.60±0.90&nbsp;(10)       |59.50±0.70&nbsp;(17)       |74.70±0.80&nbsp;(11)       |50.20±1.10&nbsp;(7.5)      |88.90±0.50&nbsp;(10)       |56.50±1.10&nbsp;(8.5)      |39.40±1.00&nbsp;(12.5)     
 SUR [[6]]                  |7.65                       |56.10±1.10&nbsp;(7)        |93.10±0.50&nbsp;(5.5)      |84.60±0.70&nbsp;(5.5)      |70.60±1.00&nbsp;(11.5)     |71.00±0.80&nbsp;(6.5)      |81.30±0.60&nbsp;(4)        |64.20±1.10&nbsp;(3.5)      |82.80±0.80&nbsp;(14.5)     |53.40±1.00&nbsp;(11.5)     |50.10±1.00&nbsp;(7)        
 SUR-pnf [[6]]              |8.2                        |56.00±1.10&nbsp;(7)        |90.00±0.60&nbsp;(10.5)     |79.70±0.80&nbsp;(10.5)     |75.90±0.90&nbsp;(7.5)      |72.50±0.70&nbsp;(4.5)      |76.70±0.70&nbsp;(8.5)      |49.80±1.10&nbsp;(7.5)      |90.00±0.60&nbsp;(7.5)      |52.20±0.80&nbsp;(11.5)     |50.20±1.10&nbsp;(7)        
-SimpleCNAPS [[8]]          |7.45                       |56.50±1.10&nbsp;(7)        |91.90±0.60&nbsp;(8)        |83.80±0.60&nbsp;(7.5)      |76.10±0.90&nbsp;(7.5)      |70.00±0.80&nbsp;(8.5)      |78.30±0.70&nbsp;(6.5)      |49.10±1.20&nbsp;(7.5)      |91.30±0.60&nbsp;(6)        |59.20±1.00&nbsp;(6)        |42.40±1.10&nbsp;(10)       
-TransductiveCNAPS [[9]]    |6.05                       |**57.90**±1.10&nbsp;(2.5)  |94.30±0.40&nbsp;(3.5)      |84.70±0.50&nbsp;(5.5)      |78.80±0.70&nbsp;(3.5)      |66.20±0.80&nbsp;(12.5)     |77.90±0.60&nbsp;(6.5)      |48.90±1.20&nbsp;(7.5)      |**92.30**±0.40&nbsp;(3)    |59.70±1.10&nbsp;(6)        |42.50±1.10&nbsp;(10)       
-URT [[10]]                 |6.05                       |55.70±1.00&nbsp;(7)        |94.40±0.40&nbsp;(3.5)      |85.80±0.60&nbsp;(4)        |76.30±0.80&nbsp;(7.5)      |71.80±0.70&nbsp;(4.5)      |**82.50**±0.60&nbsp;(2)    |63.50±1.00&nbsp;(3.5)      |88.20±0.60&nbsp;(10)       |51.10±1.10&nbsp;(14)       |52.20±1.10&nbsp;(4.5)      
-URT-pf [[10]]              |7.55                       |55.50±1.10&nbsp;(7)        |90.20±0.60&nbsp;(10.5)     |79.80±0.70&nbsp;(10.5)     |77.50±0.80&nbsp;(5)        |73.50±0.70&nbsp;(3)        |75.80±0.70&nbsp;(10)       |48.10±0.90&nbsp;(10.5)     |**91.90**±0.50&nbsp;(3)    |52.00±1.40&nbsp;(11.5)     |52.10±1.00&nbsp;(4.5)      
-FLUTE [[11]]               |5.9                        |51.80±1.10&nbsp;(10.5)     |93.20±0.50&nbsp;(5.5)      |87.20±0.50&nbsp;(3)        |79.20±0.80&nbsp;(3.5)      |68.80±0.80&nbsp;(10.5)     |79.50±0.70&nbsp;(5)        |58.10±1.10&nbsp;(5)        |**91.60**±0.60&nbsp;(3)    |58.40±1.10&nbsp;(6)        |50.00±1.00&nbsp;(7)        
-URL [[12]]                 |2.15                       |**57.51**±1.08&nbsp;(2.5)  |**94.51**±0.41&nbsp;(1.5)  |88.59±0.46&nbsp;(2)        |**80.54**±0.69&nbsp;(1.5)  |**76.17**±0.67&nbsp;(1.5)  |**81.94**±0.56&nbsp;(2)    |**68.75**±0.95&nbsp;(1.5)  |**92.11**±0.48&nbsp;(3)    |63.34±1.19&nbsp;(3.5)      |54.03±0.96&nbsp;(2.5)      
-ITA [[13]]                 |**1.65**                   |**57.35**±1.05&nbsp;(2.5)  |**94.96**±0.38&nbsp;(1.5)  |**89.33**±0.44&nbsp;(1)    |**81.42**±0.74&nbsp;(1.5)  |**76.74**±0.72&nbsp;(1.5)  |**82.01**±0.57&nbsp;(2)    |**67.40**±0.99&nbsp;(1.5)  |**92.18**±0.52&nbsp;(3)    |**83.55**±0.90&nbsp;(1)    |**55.75**±1.06&nbsp;(1)    
-TriM [[14]]                |6.6                        |**58.60**±1.00&nbsp;(2.5)  |92.00±0.60&nbsp;(8)        |82.80±0.70&nbsp;(9)        |75.30±0.80&nbsp;(7.5)      |71.20±0.80&nbsp;(6.5)      |77.30±0.70&nbsp;(8.5)      |48.50±1.00&nbsp;(10.5)     |90.50±0.50&nbsp;(7.5)      |63.00±1.00&nbsp;(3.5)      |52.80±1.10&nbsp;(2.5)      
+SimpleCNAPS [[7]]          |7.45                       |56.50±1.10&nbsp;(7)        |91.90±0.60&nbsp;(8)        |83.80±0.60&nbsp;(7.5)      |76.10±0.90&nbsp;(7.5)      |70.00±0.80&nbsp;(8.5)      |78.30±0.70&nbsp;(6.5)      |49.10±1.20&nbsp;(7.5)      |91.30±0.60&nbsp;(6)        |59.20±1.00&nbsp;(6)        |42.40±1.10&nbsp;(10)       
+TransductiveCNAPS [[8]]    |6.05                       |**57.90**±1.10&nbsp;(2.5)  |94.30±0.40&nbsp;(3.5)      |84.70±0.50&nbsp;(5.5)      |78.80±0.70&nbsp;(3.5)      |66.20±0.80&nbsp;(12.5)     |77.90±0.60&nbsp;(6.5)      |48.90±1.20&nbsp;(7.5)      |**92.30**±0.40&nbsp;(3)    |59.70±1.10&nbsp;(6)        |42.50±1.10&nbsp;(10)       
+URT [[9]]                  |6.05                       |55.70±1.00&nbsp;(7)        |94.40±0.40&nbsp;(3.5)      |85.80±0.60&nbsp;(4)        |76.30±0.80&nbsp;(7.5)      |71.80±0.70&nbsp;(4.5)      |**82.50**±0.60&nbsp;(2)    |63.50±1.00&nbsp;(3.5)      |88.20±0.60&nbsp;(10)       |51.10±1.10&nbsp;(14)       |52.20±1.10&nbsp;(4.5)      
+URT-pf [[9]]               |7.55                       |55.50±1.10&nbsp;(7)        |90.20±0.60&nbsp;(10.5)     |79.80±0.70&nbsp;(10.5)     |77.50±0.80&nbsp;(5)        |73.50±0.70&nbsp;(3)        |75.80±0.70&nbsp;(10)       |48.10±0.90&nbsp;(10.5)     |**91.90**±0.50&nbsp;(3)    |52.00±1.40&nbsp;(11.5)     |52.10±1.00&nbsp;(4.5)      
+FLUTE [[10]]               |5.9                        |51.80±1.10&nbsp;(10.5)     |93.20±0.50&nbsp;(5.5)      |87.20±0.50&nbsp;(3)        |79.20±0.80&nbsp;(3.5)      |68.80±0.80&nbsp;(10.5)     |79.50±0.70&nbsp;(5)        |58.10±1.10&nbsp;(5)        |**91.60**±0.60&nbsp;(3)    |58.40±1.10&nbsp;(6)        |50.00±1.00&nbsp;(7)        
+URL [[11]]                 |2.15                       |**57.51**±1.08&nbsp;(2.5)  |**94.51**±0.41&nbsp;(1.5)  |88.59±0.46&nbsp;(2)        |**80.54**±0.69&nbsp;(1.5)  |**76.17**±0.67&nbsp;(1.5)  |**81.94**±0.56&nbsp;(2)    |**68.75**±0.95&nbsp;(1.5)  |**92.11**±0.48&nbsp;(3)    |63.34±1.19&nbsp;(3.5)      |54.03±0.96&nbsp;(2.5)      
+ITA [[12]]                 |**1.65**                   |**57.35**±1.05&nbsp;(2.5)  |**94.96**±0.38&nbsp;(1.5)  |**89.33**±0.44&nbsp;(1)    |**81.42**±0.74&nbsp;(1.5)  |**76.74**±0.72&nbsp;(1.5)  |**82.01**±0.57&nbsp;(2)    |**67.40**±0.99&nbsp;(1.5)  |**92.18**±0.52&nbsp;(3)    |**83.55**±0.90&nbsp;(1)    |**55.75**±1.06&nbsp;(1)    
+TriM [[13]]                |6.6                        |**58.60**±1.00&nbsp;(2.5)  |92.00±0.60&nbsp;(8)        |82.80±0.70&nbsp;(9)        |75.30±0.80&nbsp;(7.5)      |71.20±0.80&nbsp;(6.5)      |77.30±0.70&nbsp;(8.5)      |48.50±1.00&nbsp;(10.5)     |90.50±0.50&nbsp;(7.5)      |63.00±1.00&nbsp;(3.5)      |52.80±1.10&nbsp;(2.5)      
 
 ## References
 
@@ -160,13 +181,12 @@ TriM [[14]]                |6.6                        |**58.60**±1.00&nbsp;(2.
 [5]: #5-saikia-et-al-2020
 [6]: #6-dvornik-et-al-2020
 [7]: #7-bateni-et-al-2020a
-[8]: #8-bateni-et-al-2020a
-[9]: #9-bateni-et-al-2020b
-[10]: #10-liu-et-al-2021a
-[11]: #11-triantafillou-et-al-2021
-[12]: #12-li-et-al-2021a
-[13]: #13-li-et-al-2021b
-[14]: #14-liu-et-al-2021b
+[8]: #8-bateni-et-al-2020b
+[9]: #9-liu-et-al-2021a
+[10]: #10-triantafillou-et-al-2021
+[11]: #11-li-et-al-2021a
+[12]: #12-li-et-al-2021b
+[13]: #13-liu-et-al-2021b
 
 ###### \[1\] Triantafillou et al. (2020)
 
@@ -203,39 +223,34 @@ Nikita Dvornik, Cordelia Schmid, Julien Mairal; [_Selecting Relevant Features fr
 Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, Leonid Sigal; [_Improved Few-Shot Visual Classification_](https://openaccess.thecvf.com/content_CVPR_2020/html/Bateni_Improved_Few-Shot_Visual_Classification_CVPR_2020_paper.html); CVPR 2020.
 
 
-###### \[8\] Bateni et al. (2020a)
-
-Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, Leonid Sigal; [_Improved Few-Shot Visual Classification_](https://openaccess.thecvf.com/content_CVPR_2020/html/Bateni_Improved_Few-Shot_Visual_Classification_CVPR_2020_paper.html); CVPR 2020.
-
-
-###### \[9\] Bateni et al. (2020b)
+###### \[8\] Bateni et al. (2020b)
 
 Peyman Bateni, Jarred Barber, Jan-Willem van de Meent, Frank Wood; [_Enhancing Few-Shot Image Classification with Unlabelled Examples_](https://arxiv.org/abs/2006.12245); arXiv 2020.
 
 
-###### \[10\] Liu et al. (2021a)
+###### \[9\] Liu et al. (2021a)
 
 Lu Liu, William Hamilton, Guodong Long, Jing Jiang, Hugo Larochelle; [_Universal Representation Transformer Layer for Few-Shot Image Classification_](https://arxiv.org/abs/2006.11702); ICLR 2021.
 
 
-###### \[11\] Triantafillou et al. (2021)
+###### \[10\] Triantafillou et al. (2021)
 
-Eleni Triantafillou, Hugo Larochelle, Richard Zemel, Vincent Dumoulin; [_Learning a Universal Template for Few-shot Dataset Generalization_](https://arxiv.org/abs/2105.07029); To appear in ICML 2021.
+Eleni Triantafillou, Hugo Larochelle, Richard Zemel, Vincent Dumoulin; [_Learning a Universal Template for Few-shot Dataset Generalization_](https://arxiv.org/abs/2105.07029); ICML 2021.
 
 
-###### \[12\] Li et al. (2021a)
+###### \[11\] Li et al. (2021a)
 
-Wei-Hong Li, Xialei Liu, Hakan Bilen; [_Universal Representation Learning from Multiple Domains for Few-shot Classification_](https://arxiv.org/pdf/2103.13841.pdf); To appear in ICCV 2021.
+Wei-Hong Li, Xialei Liu, Hakan Bilen; [_Universal Representation Learning from Multiple Domains for Few-shot Classification_](https://arxiv.org/pdf/2103.13841.pdf); ICCV 2021.
 
 
-###### \[13\] Li et al. (2021b)
+###### \[12\] Li et al. (2021b)
 
 Wei-Hong Li, Xialei Liu, Hakan Bilen; [_Improving Task Adaptation for Cross-domain Few-shot Learning_](https://arxiv.org/pdf/2107.00358.pdf); arXiv 2021.
 
 
-###### \[14\] Liu et al. (2021b)
+###### \[13\] Liu et al. (2021b)
 
-Yanbin Liu, Juho Lee, Linchao Zhu, Ling Chen, Humphrey Shi, Yi Yang; [_A Multi-Mode Modulator for Multi-Domain Few-Shot Classification_](https://openaccess.thecvf.com/content/ICCV2021/papers/Liu_A_Multi-Mode_Modulator_for_Multi-Domain_Few-Shot_Classification_ICCV_2021_paper.pdf); To appear in ICCV 2021.
+Yanbin Liu, Juho Lee, Linchao Zhu, Ling Chen, Humphrey Shi, Yi Yang; [_A Multi-Mode Modulator for Multi-Domain Few-Shot Classification_](https://openaccess.thecvf.com/content/ICCV2021/papers/Liu_A_Multi-Mode_Modulator_for_Multi-Domain_Few-Shot_Classification_ICCV_2021_paper.pdf); ICCV 2021.
 
 
 <!-- End of content generated by `Leaderboard.ipynb` -->
@@ -266,6 +281,11 @@ Meta-Dataset uses several established datasets, that are available from
 different sources. You can find below a summary of these datasets, as well as
 instructions to download them and convert them into a common format.
 
+In addition to the datasets below, the FLUTE paper reported results on 3 extra
+datasets, following recent work. You can find instructions for downloading and 
+converting those 3 additional datasets (MNIST, CIFAR-10 and CIFAR-100) in the
+CNAPs [repo](https://github.com/cambridge-mlg/cnaps).
+
 For brevity of the command line examples, we assume the following environment
 variables are defined: