Load data in batches/chunks

[caesar753]

Hello,

I have a custom image dataset, it's a bit large (10.000 images) and when I use Avalanche I first load all the data in torch tensors then transform them into avalanche tensors and then create an NC benchmark.
Loading all data at once is computationally heavy and I'd like to know if there is a way to split them into batches/chunks.
This is a part of my code

#Defining transformations
train_transform = transforms.Compose([
    transforms.Resize((224, 224)),
    # transforms.RandomResizedCrop(224),
    # transforms.RandomHorizontalFlip(),
    transforms.ToTensor(), # ToTensor : [0, 255] -> [0, 1]
    transforms.Normalize(mean = [0.485, 0.456, 0.406],
                         std = [0.229, 0.224, 0.225])
])

#defining function that transforms images for testing/validation (resizing), and normalize
test_transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(), # ToTensor : [0, 255] -> [0, 1]
    transforms.Normalize(mean = [0.485, 0.456, 0.406],
                         std = [0.229, 0.224, 0.225])
])

#Setting and checking datasets  
train_data = datasets.ImageFolder("/home/caesar753/data/Train",\
     train_transform)

 test_data = datasets.ImageFolder("/home/caesar753/data/Val", \
     test_transform)

#Defining train and test loaders
train_loader = DataLoader(train_data,
                          batch_size=len(train_data),
#                           batch_size = 50#,
                          shuffle=True,
                          #drop_last=True)
                          )
#dataloader for testing
test_loader = DataLoader(test_data, 
                         batch_size=len(test_data),
#                          batch_size = 50#,
                         shuffle=True
                        )


# #generating an iterator
dataiter = iter(train_loader)
dataiter_test = iter(test_loader)

# #calling next image and label with iterator
images, labels = next(dataiter)
images_test, labels_test = next(dataiter_test)


#Transforming labels and images into PyTorch tensors
tensor_dset_labels = torch.tensor(labels)
tensor_dset_train = torch.tensor(images)
tensor_dset_eval_labels = torch.tensor(labels_test)
tensor_dset_eval = torch.tensor(images_test)

#Transforming PyTorch tensors into Avalanche tensors
maketensor_train = MakeTensor(tensor_dset_train, tensor_dset_labels)
maketensor_eval = MakeTensor(tensor_dset_eval, tensor_dset_eval_labels)


#Calling Avalanche benchmark generator
# pippo = Generator.dataset_benchmark([avalanche_dset_train], [avalanche_dset_eval])

pippo_nc = Generator.nc_benchmark([maketensor_train], [maketensor_eval],\
    #  task_labels = False, n_experiences = 11)
    task_labels = False, n_experiences = 3)

[AntonioCarta]

There are two solutions:

• you don't need to convert your data to tensors. You can just wrap it in an AvalancheDataset (e.g. using make_classification_dataset).
• If you really want tensors, you can use lazy streams. You can use this comment as a starting point.

[caesar753]

Perfect!
I tried the first (create an AvalancheDataset) and it works great
Thanks again!