Allow to shutdown persistent workers #71383
Description
🚀 The feature, motivation and pitch
Using persistent workers speeds up my training pipeline by a significant margin, but now I would like to be able to implement hard negative mining every N epochs and thus would like to change the data batches from my dataset.
For this to be doable, I would need to forcefully shutdown the persistent worker processes each time I perform hard negative mining, so that the next epoch will create new workers and use the new negative samples from my data.
Alternatives
Currently I disabled persistent workers and that works, but it is about 1.4 times slower, so I would love to be able to use persistent workers.
Additional context
No response