Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

opt.nThreads leads to possibly wrong data. #34

Closed
Ailon-Island opened this issue Apr 19, 2022 · 1 comment
Closed

opt.nThreads leads to possibly wrong data. #34

Ailon-Island opened this issue Apr 19, 2022 · 1 comment

Comments

@Ailon-Island
Copy link

I set the option --nThreads to 8, but the dataloader seems to not work as expected. I guess that could be duplicate datasets are working with independent attributes in these 8 threads.
I've got all the data_type printed at the beginning of each batch. One type of data is fed successively for nThreads times until it gets increased. Is there any way to fix this without loss of much performance?

------------ Options -------------
Arc_path: arcface_model/arcface_checkpoint.tar
batchSize: 8
beta1: 0.5
checkpoints_dir: ./checkpoints
continue_train: False
data_type: 32
dataroot: ./dataset/CelebA
debug: False
display_freq: 1
display_winsize: 224
feat_num: 3
fineSize: 512
fp16: False
gan_mode: hinge
gpu_ids: [0]
image_size: 224
input_nc: 3
instance_feat: False
isTrain: True
label_feat: False
label_nc: 0
lambda_GP: 1e-05
lambda_feat: 10.0
lambda_id: 10.0
lambda_rec: 10.0
latent_size: 512
loadSize: 1024
load_features: False
load_pretrain: 
local_rank: 0
lr: 0.0002
max_dataset_size: inf
model: pix2pixHD
nThreads: 8
n_blocks_global: 6
n_blocks_local: 3
n_clusters: 10
n_downsample_E: 4
n_downsample_global: 3
n_layers_D: 4
n_local_enhancers: 1
name: CelebA_224
ndf: 64
nef: 16
netG: global
ngf: 64
niter: 10000
niter_decay: 10000
niter_fix_global: 0
no_flip: False
no_ganFeat_loss: False
no_html: False
no_instance: False
no_vgg_loss: False
norm: batch
norm_G: spectralspadesyncbatch3x3
num_D: 2
output_nc: 3
phase: train
pool_size: 0
print_freq: 1
resize_or_crop: scale_width
save_epoch_freq: 10000
save_latest_freq: 10000
semantic_nc: 3
serial_batches: False
tf_log: False
times_G: 1
use_dropout: False
verbose: False
which_epoch: latest
-------------- End ----------------
CustomDatasetDataLoader
dataset [FaceDataSet] was created
#training images = 202129
create web directory ./checkpoints\CelebA_224\web...
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 8, time: 0.955) G_GAN: -0.215 G_GAN_Feat: 23.784 G_ID: 9.646 G_Rec: 11.735 D_GP: 0.001 D_real: 0.808 D_fake: 1.215 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 16, time: 0.092) G_GAN: 0.023 G_GAN_Feat: 16.439 G_ID: 9.752 G_Rec: 7.549 D_GP: 0.000 D_real: 1.191 D_fake: 1.277 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 24, time: 0.081) G_GAN: 0.035 G_GAN_Feat: 14.662 G_ID: 9.737 G_Rec: 6.130 D_GP: 0.000 D_real: 1.055 D_fake: 1.185 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 32, time: 0.082) G_GAN: 0.003 G_GAN_Feat: 12.767 G_ID: 9.161 G_Rec: 5.381 D_GP: 0.000 D_real: 1.105 D_fake: 1.121 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 40, time: 0.081) G_GAN: 0.142 G_GAN_Feat: 12.409 G_ID: 9.250 G_Rec: 5.028 D_GP: 0.000 D_real: 1.126 D_fake: 1.014 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 48, time: 0.084) G_GAN: 0.018 G_GAN_Feat: 11.651 G_ID: 8.848 G_Rec: 4.848 D_GP: 0.000 D_real: 0.995 D_fake: 1.036 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 56, time: 0.086) G_GAN: -0.180 G_GAN_Feat: 10.410 G_ID: 7.619 G_Rec: 4.273 D_GP: 0.000 D_real: 0.858 D_fake: 1.186 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 64, time: 0.081) G_GAN: 0.026 G_GAN_Feat: 11.858 G_ID: 8.109 G_Rec: 5.547 D_GP: 0.000 D_real: 1.032 D_fake: 1.011 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 72, time: 0.082) G_GAN: 0.103 G_GAN_Feat: 10.665 G_ID: 9.722 G_Rec: 4.928 D_GP: 0.000 D_real: 0.956 D_fake: 0.912 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 80, time: 0.086) G_GAN: 0.070 G_GAN_Feat: 10.263 G_ID: 9.511 G_Rec: 4.405 D_GP: 0.000 D_real: 0.933 D_fake: 0.958 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 88, time: 0.085) G_GAN: 0.052 G_GAN_Feat: 9.233 G_ID: 9.924 G_Rec: 3.842 D_GP: 0.000 D_real: 0.872 D_fake: 0.979 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 96, time: 0.082) G_GAN: -0.091 G_GAN_Feat: 10.016 G_ID: 9.510 G_Rec: 4.408 D_GP: 0.000 D_real: 0.833 D_fake: 1.098 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 104, time: 0.085) G_GAN: 0.041 G_GAN_Feat: 9.755 G_ID: 9.953 G_Rec: 4.054 D_GP: 0.000 D_real: 0.950 D_fake: 0.988 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 112, time: 0.086) G_GAN: 0.002 G_GAN_Feat: 9.763 G_ID: 10.223 G_Rec: 4.298 D_GP: 0.000 D_real: 0.786 D_fake: 1.010 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 120, time: 0.080) G_GAN: 0.310 G_GAN_Feat: 9.213 G_ID: 9.937 G_Rec: 3.761 D_GP: 0.000 D_real: 1.077 D_fake: 0.826 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 128, time: 0.080) G_GAN: -0.077 G_GAN_Feat: 11.179 G_ID: 10.065 G_Rec: 4.348 D_GP: 0.000 D_real: 0.824 D_fake: 1.104 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 136, time: 0.082) G_GAN: -0.130 G_GAN_Feat: 10.189 G_ID: 9.776 G_Rec: 4.090 D_GP: 0.000 D_real: 0.703 D_fake: 1.133 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 144, time: 0.079) G_GAN: 0.337 G_GAN_Feat: 13.464 G_ID: 9.856 G_Rec: 7.010 D_GP: 0.000 D_real: 1.178 D_fake: 0.791 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 152, time: 0.082) G_GAN: -0.030 G_GAN_Feat: 10.651 G_ID: 9.827 G_Rec: 4.328 D_GP: 0.000 D_real: 0.848 D_fake: 1.090 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 160, time: 0.080) G_GAN: 0.254 G_GAN_Feat: 7.981 G_ID: 10.843 G_Rec: 3.265 D_GP: 0.000 D_real: 1.095 D_fake: 0.847 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 168, time: 0.083) G_GAN: 0.036 G_GAN_Feat: 9.070 G_ID: 10.492 G_Rec: 4.058 D_GP: 0.000 D_real: 0.949 D_fake: 0.989 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 176, time: 0.084) G_GAN: 0.152 G_GAN_Feat: 8.269 G_ID: 9.705 G_Rec: 3.551 D_GP: 0.000 D_real: 0.938 D_fake: 0.874 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 184, time: 0.087) G_GAN: 0.261 G_GAN_Feat: 7.916 G_ID: 9.935 G_Rec: 3.237 D_GP: 0.000 D_real: 0.908 D_fake: 0.766 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 192, time: 0.081) G_GAN: 0.237 G_GAN_Feat: 9.409 G_ID: 10.117 G_Rec: 4.025 D_GP: 0.000 D_real: 0.885 D_fake: 0.810 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 200, time: 0.081) G_GAN: 0.158 G_GAN_Feat: 8.814 G_ID: 10.127 G_Rec: 3.870 D_GP: 0.000 D_real: 0.777 D_fake: 0.883 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 208, time: 0.087) G_GAN: 0.842 G_GAN_Feat: 8.845 G_ID: 10.243 G_Rec: 3.296 D_GP: 0.001 D_real: 1.372 D_fake: 0.526 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 216, time: 0.080) G_GAN: -0.421 G_GAN_Feat: 9.650 G_ID: 9.546 G_Rec: 3.654 D_GP: 0.001 D_real: 0.506 D_fake: 1.439 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 224, time: 0.080) G_GAN: -0.022 G_GAN_Feat: 8.847 G_ID: 10.050 G_Rec: 3.227 D_GP: 0.001 D_real: 0.724 D_fake: 1.052 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 232, time: 0.082) G_GAN: 0.445 G_GAN_Feat: 9.311 G_ID: 10.803 G_Rec: 4.323 D_GP: 0.001 D_real: 1.091 D_fake: 0.649 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 240, time: 0.080) G_GAN: 0.361 G_GAN_Feat: 8.344 G_ID: 9.757 G_Rec: 3.223 D_GP: 0.001 D_real: 0.910 D_fake: 0.693 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 248, time: 0.083) G_GAN: 0.114 G_GAN_Feat: 8.736 G_ID: 9.966 G_Rec: 4.477 D_GP: 0.001 D_real: 0.693 D_fake: 0.935 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 256, time: 0.082) G_GAN: 0.224 G_GAN_Feat: 8.297 G_ID: 9.602 G_Rec: 3.353 D_GP: 0.001 D_real: 0.794 D_fake: 0.909 
same ID: tensor([4, 4, 4, 4, 4, 4, 4, 4])
(epoch: 1, iters: 264, time: 0.080) G_GAN: 0.028 G_GAN_Feat: 8.113 G_ID: 9.754 G_Rec: 3.604 D_GP: 0.001 D_real: 0.768 D_fake: 1.041 
same ID: tensor([4, 4, 4, 4, 4, 4, 4, 4])
(epoch: 1, iters: 272, time: 0.082) G_GAN: 0.275 G_GAN_Feat: 7.745 G_ID: 10.281 G_Rec: 3.235 D_GP: 0.001 D_real: 0.924 D_fake: 0.809 
same ID: tensor([4, 4, 4, 4, 4, 4, 4, 4])
(epoch: 1, iters: 280, time: 0.082) G_GAN: 0.199 G_GAN_Feat: 9.014 G_ID: 9.414 G_Rec: 5.138 D_GP: 0.001 D_real: 1.058 D_fake: 0.861 
same ID: tensor([4, 4, 4, 4, 4, 4, 4, 4])
(epoch: 1, iters: 288, time: 0.081) G_GAN: 0.195 G_GAN_Feat: 8.024 G_ID: 10.019 G_Rec: 3.524 D_GP: 0.001 D_real: 0.952 D_fake: 0.870 
same ID: tensor([4, 4, 4, 4, 4, 4, 4, 4])
(epoch: 1, iters: 296, time: 0.082) G_GAN: 0.190 G_GAN_Feat: 7.509 G_ID: 10.013 G_Rec: 2.861 D_GP: 0.001 D_real: 0.862 D_fake: 0.838 
same ID: tensor([4, 4, 4, 4, 4, 4, 4, 4])
(epoch: 1, iters: 304, time: 0.082) G_GAN: 0.253 G_GAN_Feat: 8.452 G_ID: 10.450 G_Rec: 3.357 D_GP: 0.001 D_real: 0.854 D_fake: 0.776 
same ID: tensor([4, 4, 4, 4, 4, 4, 4, 4])
(epoch: 1, iters: 312, time: 0.080) G_GAN: 0.571 G_GAN_Feat: 8.652 G_ID: 10.282 G_Rec: 3.527 D_GP: 0.002 D_real: 1.114 D_fake: 0.527 
same ID: tensor([4, 4, 4, 4, 4, 4, 4, 4])
(epoch: 1, iters: 320, time: 0.080) G_GAN: -0.004 G_GAN_Feat: 8.572 G_ID: 9.927 G_Rec: 3.243 D_GP: 0.002 D_real: 0.462 D_fake: 1.041 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 328, time: 0.080) G_GAN: 0.565 G_GAN_Feat: 9.724 G_ID: 7.166 G_Rec: 4.114 D_GP: 0.003 D_real: 1.065 D_fake: 0.671 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 336, time: 0.078) G_GAN: 0.118 G_GAN_Feat: 8.737 G_ID: 6.280 G_Rec: 3.815 D_GP: 0.002 D_real: 0.868 D_fake: 1.031 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 344, time: 0.081) G_GAN: 0.062 G_GAN_Feat: 7.814 G_ID: 5.888 G_Rec: 3.160 D_GP: 0.002 D_real: 0.717 D_fake: 1.016 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 352, time: 0.079) G_GAN: 0.521 G_GAN_Feat: 8.553 G_ID: 5.634 G_Rec: 4.576 D_GP: 0.002 D_real: 1.117 D_fake: 0.566 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 360, time: 0.081) G_GAN: 0.107 G_GAN_Feat: 9.291 G_ID: 6.601 G_Rec: 5.221 D_GP: 0.002 D_real: 0.888 D_fake: 0.912 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 368, time: 0.082) G_GAN: 0.060 G_GAN_Feat: 9.828 G_ID: 5.915 G_Rec: 5.354 D_GP: 0.003 D_real: 0.724 D_fake: 0.975 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 376, time: 0.082) G_GAN: 0.892 G_GAN_Feat: 9.091 G_ID: 5.354 G_Rec: 4.656 D_GP: 0.002 D_real: 1.482 D_fake: 0.421 
same ID: tensor([0, 0, 0, 0, 0, 0, 0, 0])
(epoch: 1, iters: 384, time: 0.081) G_GAN: -0.054 G_GAN_Feat: 6.884 G_ID: 5.002 G_Rec: 2.787 D_GP: 0.003 D_real: 0.627 D_fake: 1.072 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 392, time: 0.079) G_GAN: 0.188 G_GAN_Feat: 8.042 G_ID: 10.007 G_Rec: 3.202 D_GP: 0.004 D_real: 0.744 D_fake: 0.826 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 400, time: 0.081) G_GAN: 0.454 G_GAN_Feat: 7.759 G_ID: 10.022 G_Rec: 3.154 D_GP: 0.004 D_real: 1.167 D_fake: 0.638 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 408, time: 0.081) G_GAN: 0.461 G_GAN_Feat: 7.816 G_ID: 10.175 G_Rec: 3.143 D_GP: 0.003 D_real: 0.967 D_fake: 0.618 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 416, time: 0.081) G_GAN: 0.018 G_GAN_Feat: 8.015 G_ID: 9.605 G_Rec: 2.684 D_GP: 0.004 D_real: 0.704 D_fake: 1.041 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 424, time: 0.098) G_GAN: -0.196 G_GAN_Feat: 7.417 G_ID: 9.965 G_Rec: 2.688 D_GP: 0.004 D_real: 0.761 D_fake: 1.274 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 432, time: 0.088) G_GAN: 0.521 G_GAN_Feat: 9.091 G_ID: 9.760 G_Rec: 5.096 D_GP: 0.003 D_real: 1.240 D_fake: 0.668 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 440, time: 0.087) G_GAN: -0.066 G_GAN_Feat: 7.830 G_ID: 9.920 G_Rec: 3.300 D_GP: 0.002 D_real: 0.889 D_fake: 1.080 
same ID: tensor([1, 1, 1, 1, 1, 1, 1, 1])
(epoch: 1, iters: 448, time: 0.124) G_GAN: 0.099 G_GAN_Feat: 6.673 G_ID: 10.154 G_Rec: 2.599 D_GP: 0.002 D_real: 0.788 D_fake: 0.948 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 456, time: 0.091) G_GAN: 0.049 G_GAN_Feat: 7.493 G_ID: 9.551 G_Rec: 3.624 D_GP: 0.002 D_real: 0.827 D_fake: 0.965 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 464, time: 0.088) G_GAN: 0.286 G_GAN_Feat: 8.065 G_ID: 9.909 G_Rec: 3.528 D_GP: 0.002 D_real: 0.839 D_fake: 0.755 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 472, time: 0.094) G_GAN: 0.415 G_GAN_Feat: 7.930 G_ID: 9.979 G_Rec: 4.074 D_GP: 0.003 D_real: 1.199 D_fake: 0.658 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 480, time: 0.084) G_GAN: 0.360 G_GAN_Feat: 7.778 G_ID: 9.684 G_Rec: 2.843 D_GP: 0.003 D_real: 0.868 D_fake: 0.673 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 488, time: 0.082) G_GAN: 0.247 G_GAN_Feat: 7.325 G_ID: 9.680 G_Rec: 3.086 D_GP: 0.004 D_real: 0.891 D_fake: 0.775 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 496, time: 0.081) G_GAN: 0.197 G_GAN_Feat: 8.729 G_ID: 9.770 G_Rec: 4.807 D_GP: 0.005 D_real: 0.694 D_fake: 0.859 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 504, time: 0.080) G_GAN: 0.332 G_GAN_Feat: 9.149 G_ID: 9.967 G_Rec: 3.969 D_GP: 0.006 D_real: 0.948 D_fake: 0.722 
same ID: tensor([2, 2, 2, 2, 2, 2, 2, 2])
(epoch: 1, iters: 512, time: 0.080) G_GAN: 0.243 G_GAN_Feat: 7.408 G_ID: 9.460 G_Rec: 2.800 D_GP: 0.006 D_real: 0.974 D_fake: 0.839 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 520, time: 0.079) G_GAN: 0.039 G_GAN_Feat: 8.930 G_ID: 9.992 G_Rec: 4.354 D_GP: 0.004 D_real: 0.958 D_fake: 0.989 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 528, time: 0.081) G_GAN: -0.200 G_GAN_Feat: 7.089 G_ID: 9.590 G_Rec: 3.049 D_GP: 0.004 D_real: 0.714 D_fake: 1.220 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 536, time: 0.083) G_GAN: 0.464 G_GAN_Feat: 7.312 G_ID: 10.589 G_Rec: 3.485 D_GP: 0.003 D_real: 1.075 D_fake: 0.588 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
(epoch: 1, iters: 544, time: 0.081) G_GAN: 0.038 G_GAN_Feat: 8.542 G_ID: 9.659 G_Rec: 4.123 D_GP: 0.005 D_real: 0.745 D_fake: 0.990 
same ID: tensor([3, 3, 3, 3, 3, 3, 3, 3])
@Ailon-Island Ailon-Island changed the title opt.nThreads leads to possibly wrong data. opt.nThreads leads to possibly wrong data. Apr 19, 2022
@Ailon-Island Ailon-Island changed the title opt.nThreads leads to possibly wrong data. opt.nThreads leads to possibly wrong data. Apr 19, 2022
@Ailon-Island
Copy link
Author

I've solved it myself. The optional argument worker_init_fn in DataLoader helps. In specific, instantiate the DataLoader with function of setting the correct data_type as worker_init_fn.

class MyDataset(Dataset):
    ...
    def set_data_type(self, worker_id):
       self.data_type = worker_id % 6 # or any other cycle length
...
loader = DataLoader(..., worker_init_fn=dataset.set_data_type)

And the in-worker data_type update step should also be modified to opt.nThreads % 6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Ailon-Island