Question about new_instance & new_class benchmark

[ElvishElvis]

Hi!
I am trying to implement a scenario where there's 10 group of train/test data pairs, where each train/test was under a progressive transformation of the last group for only one step. There's are 11 same label classes for all of those group, and the number of data in each group are the same as well. I need to learn on each pair of train in a fix order with fix class as well(fix task order fix class order), and evaluate on the test data from all the group after each training. It should be indeed an instance incremental learning. However, when I was trying to implement using ni_benchmark, the print out show

Current Classes: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109]

however, when I use nc_benchmark, the print out show
Current Classes: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
I am quite confused by this as well, seemingly it should be the opposite for them? Where new instance have only 10 class and new class have 10*11 class.

Which implementation should I stick to and what should I refer to? Thank you in advance!

[vlomonaco]

Hi @ElvishElvis!

Your question is indeed very relevant since this may be a confusion point worth clarifying in the documentation in the future. For your specific use-case I'd suggest to use the lower-level "dataset_benchmark" generator that is only partially documented cause it is just a rename of this method. This method accepts a list of AvalancheDatasets where you'll be able to setup eventual "task" labels and are pretty much agnostic to the pytorch dataset content.

Alternatively you can use the nc_benchmark generator as done here for the RotatedMNIST benchmark but keep in mind that the nc_benchmark was designed with a different objective in mind (split a dataset based on the class info) as the ni_benchmark (split a dataset maintaining all the classes in each split). If you have already the data split as you like, this may be an overkill :)

Please also note that ni_benchmark assumes you want to concatenate the datasets before operating the split. This is why classes are re-mapped sequentially as you can see from your output.

Let me know if this solves you issues and clarifies a bit the confusion.

[ElvishElvis]

Thank you so much for your detail clarification! I had used nc_benchmark in my senario aligned with the RotatedMNIST example and it work well now.

[ElvishElvis]

Copying my previsou question here from slack channel for reference

Question:
Hello, I have a question about the using of avalanche. In the avalanche/avalanche/benchmarks/classic/cmnist.py, both permuted MNIST and rotated MNIST are implemented with nc_benchmark, which is new class learning according to the avalanche definition. However, it's clear that both of the task should be instance incremental(since same label class are presented over a shrift distribution of input data). I am quite confused about that, any help would be highly appreciated!

Answer: (by andcos)
Hi @jia Shi, thanks for getting in touch! The key is the parameter one_dataset_per_exp: when True, the input to nc_benchmark is a list of datasets and the generator will consider a dataset for each experience. I agree that the current implementation can be confusing, we will try to make it clearer 🙂