LambdaDataset?

[kohr-h]

I'm wondering if this is of interest to the torchvision community: a dataset defined with two functions. It would look like this:

wrapped_dset = # some dataset

def data(i):
    return wrapped_dset[i][0]

def target(i):
    return data(i)[:, ::2, ::2]  # downsample on the fly

dset = LambdaDataset(data_func=data, target_func=target, length=len(wrapped_dset))

It could be super-handy to combine datasets, switch out the target like above, etc., without having to write a separate class.

[vfdev-5]

@kohr-h actually, as I understand a Dataset is just an iterable with length, so a dataset corresponding to your LambdaDataset is just

dset = [(data(i), target(i)) for i in range(len(wrapped_dset))]

So, the 👍 of LambdaDataset is that it is already an instance of Dataset.
Could you please provide more use-cases where it can be used ?

[fmassa]

I'm not sure we would want to add it to torchvision though, as it is not really specific to image data.

Also, while it might look that it is sort of generic, it also imposes a restriction that there are only two objects that are returned (which can be tuples or dict, but still).

In the end, I'm not sure that the benefits it brings compensate the extra complexity.

I'm closing this, but please feel free to comment if you disagree.

[kohr-h]

I'm not sure we would want to add it to torchvision though, as it is not really specific to image data.

It's not specific to to images, that's true.

Also, while it might look that it is sort of generic, it also imposes a restriction that there are only two objects that are returned (which can be tuples or dict, but still).

Good point there. It would be better to have just one function draw or so which returns the tuple or whatever you like.

In the end, I'm not sure that the benefits it brings compensate the extra complexity.

I'm thinking mainly of on-the-fly sample generation. The list approach by @vfdev-5 works as well, of course, but you may not want to have all the data lying around at once. A generator fixes that but misses the __len__ attribute. So in plain Python you would write a small custom iterator class, which would be doing the exact same thing as a LambdaDataset, only in __iter__ instead of __getitem__.

In the end, I'm not sure that the benefits it brings compensate the extra complexity.

I guess the benefit would be providing the iterator boilerplate so the user only has to write a function (a bit like the LambdaLR, but that one has much more boilerplate, admittedly).
If you think that's not enough to add code, I'm okay with that.

[fmassa]

I think that a LambdaLR abstracts away a few more things than the LambdaDataset. Still, I've faced serialization issues with LambdaLR, so have mixed feelings with using it.
I'm still not 100% convinced about a LambdaDataset: it doesn't really bring new functionality, and don't save much typing. The same can be said about ListDataset for example, and in the end I think having a small class is generally simpler.

Finally, I think such functionality is outside of the scope of torchvision, so I still hold on my previous view that it might be better to avoid adding this to torchvision.

[kohr-h]

Yeah, I tried to bring forth the best arguments, but they're not really that strong. And I understand the reasons against.

[fmassa]

Thanks for the understanding @kohr-h
But there might be other places that such a functionality might be desired, for example as a third-party package?