New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create a pytorch dataloader with 2 data sources #5220
Comments
Hi @rvandeghen, Could you tell us more on the use case at hand (what data you'd like to read, what framework you're using)? In principle, you can have a single pipeline with multiple readers/external sources that produces multiple outputs and wrap the pipeline in a |
Hi @stiepan, I'm using pytorch, and I would like to have data coming from videos and from images. Currently, I was using them separately, so I have 2 independent pipes with 2 different dataloader, but I would like to merge them in the same dataloader and use them in the same training. If you need more information about the use case, let me know. |
Thanks for the details @rvandeghen.
In the simple case when you need images and videos you could just read them both and return from the same pipeline.
This may be tricky. With the exception of if/else blocks (with conditional execution enabled), DALI requires batch size to stay the same across a single iteration in the pipeline. So, something like reading at once batch of video, batch of imges and merging it into a single batch won't work (as the output batch will be bigger). With the conditionals, you could create an output batch that picks some samples from one batch and others from another batch (as long as the two source batch sizes and the output batch size stay the same), but that's also subject to some restrictions (i.e. the dimensionality and data-type of the the samples in a batch must match). Could you tell us more how you mix the modalities when feeding them into the network? If you switch after a number of iterations, maybe custom Python iterator that wraps two Do you use DALI's |
@stiepan For the images, we have a pipeline something similar to
and its corresponding dataloader
For the videos, our pipeline follows the same idea
and the dataloader
Note that for this simple case, our effective batch size is given as the batch size multiplied by the sequence length, thus the dimension is the same as for the image pipeline. We consider that our model is agnostic to the modality of data. |
Hi @rvandeghen, Thanks for the details of your use case. As I mentioned before, unfortunately, the pipeline (and the provided iterator) are constrained in that the batch for the inputs and outputs must be equal, so currently there's no way to merge two different batches into a single one of bigger size. Maybe even more importantly, if we have two readers in the pipeline, both of them will produce the full batch in each iteration, so there's no way to delay progress of reading images to videos (or the other way round) within single pipeline. Given those constraint, I think the best option at the table is to use the two pipelines separately and implement Python iterator that takes the two pipeline instances in the constructor and implements its own logic of which of the pipelines should be quereied for the next iteration and combining the outputs using pytroch. The closest to what you would like to do that is possible with DALI right now is, I think, reading two full batches of images and videos and mixing them into two batches with the samples shuffled. I am not sure if that's of any help in your case, but just for completness, here's a sample code that does that:
|
Describe the question.
Hi,
I would like to know how I can create 2 pipelines with 2 different sources, and use them in the same dataloader using
DALIGenericIterator
, or something else more appropriated.Thanks
Check for duplicates
The text was updated successfully, but these errors were encountered: