IncrementalClassifier doesn't forget first experience and doesn't learn newers ones

[PerezLucas2903]

I'm experimenting with IncrementalClassifier and while running a basic SplitMNIST run of it, encountered these unexpected results

benchmark = SplitMNIST(n_experiences=5, shuffle=False)

shape = np.array(benchmark.train_stream[0].dataset[0][0].shape)
in_features = shape.prod()

baseline_model = IncrementalClassifier(in_features = in_features, initial_out_features=2, masking=True)
optimizer = SGD(baseline_model.parameters(), lr=0.001, momentum=0.9)
criterion = CrossEntropyLoss()

params = {'batch_size': 512,
          'num_workers': 2}

for experience in benchmark.train_stream:
       print(experience.classes_in_this_experience)
      
  baseline_model.adaptation(experience)
  print(baseline_model)

  n_params = get_n_params(baseline_model)
  
  data_tensor = torch.empty((0, 784), dtype=torch.float32)
  data_label = []

  for data in experience.dataset:
      data_ravel = torch.ravel(data[0])
      data_reshaped = torch.reshape(data_ravel, (-1, 784))
      data_tensor = torch.cat((data_tensor, data_reshaped), 0)
      data_label.append(data[1])

  custom_ds = CustomDataset(data_tensor, data_label)
  train_loader = DataLoader(custom_ds, **params)

  for X, y in train_loader:
      baseline_model.train()
      optimizer.zero_grad()
      output = baseline_model.forward(X)
      loss = criterion(output, y)
      loss.backward()
      optimizer.step()

After training with the code above, I get the test accuracy on each experience:

Experience 0: 0.9852941176470589
Experience 1: 0.00341796875
Experience 2: 0.0020833333333333333
Experience 3: 0.00146484375
Experience 4: 0.0751008064516129
Average: 0.213472213986401

As we can see, learning only occurs on experience 0, and there is no forgetting. What am I missing here?

[AntonioCarta]

I think baseline_model.adaptation(experience) <- this is incorrect. You have to call adaptation for each submodule too.

[PerezLucas2903]

There aren't any submodules. In this experiment, the neural network has only one layer, given by the IncrementalClassifier