Loading training set... Image shape: [3, 1024, 1024] Label shape: [0] Constructing networks... Setting up TensorFlow plugin "fused_bias_act.cu": Compiling... Loading... Done. Setting up TensorFlow plugin "upfirdn_2d.cu": Compiling... Loading... Done. Resuming from "/content/drive/MyDrive/ganResults/stylegan2-ada/str-carn-2020-ryg/network-snapshot-000080.pkl" G Params OutputShape WeightShape --- --- --- --- latents_in - (?, 512) - labels_in - (?, 0) - G_mapping/Normalize - (?, 512) - G_mapping/Dense0 262656 (?, 512) (512, 512) G_mapping/Dense1 262656 (?, 512) (512, 512) G_mapping/Broadcast - (?, 18, 512) - dlatent_avg - (512,) - Truncation/Lerp - (?, 18, 512) - G_synthesis/4x4/Const 8192 (?, 512, 4, 4) (1, 512, 4, 4) G_synthesis/4x4/Conv 2622465 (?, 512, 4, 4) (3, 3, 512, 512) G_synthesis/4x4/ToRGB 264195 (?, 3, 4, 4) (1, 1, 512, 3) G_synthesis/8x8/Conv0_up 2622465 (?, 512, 8, 8) (3, 3, 512, 512) G_synthesis/8x8/Conv1 2622465 (?, 512, 8, 8) (3, 3, 512, 512) G_synthesis/8x8/Upsample - (?, 3, 8, 8) - G_synthesis/8x8/ToRGB 264195 (?, 3, 8, 8) (1, 1, 512, 3) G_synthesis/16x16/Conv0_up 2622465 (?, 512, 16, 16) (3, 3, 512, 512) G_synthesis/16x16/Conv1 2622465 (?, 512, 16, 16) (3, 3, 512, 512) G_synthesis/16x16/Upsample - (?, 3, 16, 16) - G_synthesis/16x16/ToRGB 264195 (?, 3, 16, 16) (1, 1, 512, 3) G_synthesis/32x32/Conv0_up 2622465 (?, 512, 32, 32) (3, 3, 512, 512) G_synthesis/32x32/Conv1 2622465 (?, 512, 32, 32) (3, 3, 512, 512) G_synthesis/32x32/Upsample - (?, 3, 32, 32) - G_synthesis/32x32/ToRGB 264195 (?, 3, 32, 32) (1, 1, 512, 3) G_synthesis/64x64/Conv0_up 2622465 (?, 512, 64, 64) (3, 3, 512, 512) G_synthesis/64x64/Conv1 2622465 (?, 512, 64, 64) (3, 3, 512, 512) G_synthesis/64x64/Upsample - (?, 3, 64, 64) - G_synthesis/64x64/ToRGB 264195 (?, 3, 64, 64) (1, 1, 512, 3) G_synthesis/128x128/Conv0_up 1442561 (?, 256, 128, 128) (3, 3, 512, 256) G_synthesis/128x128/Conv1 721409 (?, 256, 128, 128) (3, 3, 256, 256) G_synthesis/128x128/Upsample - (?, 3, 128, 128) - G_synthesis/128x128/ToRGB 132099 (?, 3, 128, 128) (1, 1, 256, 3) G_synthesis/256x256/Conv0_up 426369 (?, 128, 256, 256) (3, 3, 256, 128) G_synthesis/256x256/Conv1 213249 (?, 128, 256, 256) (3, 3, 128, 128) G_synthesis/256x256/Upsample - (?, 3, 256, 256) - G_synthesis/256x256/ToRGB 66051 (?, 3, 256, 256) (1, 1, 128, 3) G_synthesis/512x512/Conv0_up 139457 (?, 64, 512, 512) (3, 3, 128, 64) G_synthesis/512x512/Conv1 69761 (?, 64, 512, 512) (3, 3, 64, 64) G_synthesis/512x512/Upsample - (?, 3, 512, 512) - G_synthesis/512x512/ToRGB 33027 (?, 3, 512, 512) (1, 1, 64, 3) G_synthesis/1024x1024/Conv0_up 51297 (?, 32, 1024, 1024) (3, 3, 64, 32) G_synthesis/1024x1024/Conv1 25665 (?, 32, 1024, 1024) (3, 3, 32, 32) G_synthesis/1024x1024/Upsample - (?, 3, 1024, 1024) - G_synthesis/1024x1024/ToRGB 16515 (?, 3, 1024, 1024) (1, 1, 32, 3) --- --- --- --- Total 28794124 D Params OutputShape WeightShape --- --- --- --- images_in - (?, 3, 1024, 1024) - labels_in - (?, 0) - 1024x1024/FromRGB 128 (?, 32, 1024, 1024) (1, 1, 3, 32) 1024x1024/Conv0 9248 (?, 32, 1024, 1024) (3, 3, 32, 32) 1024x1024/Conv1_down 18496 (?, 64, 512, 512) (3, 3, 32, 64) 1024x1024/Skip 2048 (?, 64, 512, 512) (1, 1, 32, 64) 512x512/Conv0 36928 (?, 64, 512, 512) (3, 3, 64, 64) 512x512/Conv1_down 73856 (?, 128, 256, 256) (3, 3, 64, 128) 512x512/Skip 8192 (?, 128, 256, 256) (1, 1, 64, 128) 256x256/Conv0 147584 (?, 128, 256, 256) (3, 3, 128, 128) 256x256/Conv1_down 295168 (?, 256, 128, 128) (3, 3, 128, 256) 256x256/Skip 32768 (?, 256, 128, 128) (1, 1, 128, 256) 128x128/Conv0 590080 (?, 256, 128, 128) (3, 3, 256, 256) 128x128/Conv1_down 1180160 (?, 512, 64, 64) (3, 3, 256, 512) 128x128/Skip 131072 (?, 512, 64, 64) (1, 1, 256, 512) 64x64/Conv0 2359808 (?, 512, 64, 64) (3, 3, 512, 512) 64x64/Conv1_down 2359808 (?, 512, 32, 32) (3, 3, 512, 512) 64x64/Skip 262144 (?, 512, 32, 32) (1, 1, 512, 512) 32x32/Conv0 2359808 (?, 512, 32, 32) (3, 3, 512, 512) 32x32/Conv1_down 2359808 (?, 512, 16, 16) (3, 3, 512, 512) 32x32/Skip 262144 (?, 512, 16, 16) (1, 1, 512, 512) 16x16/Conv0 2359808 (?, 512, 16, 16) (3, 3, 512, 512) 16x16/Conv1_down 2359808 (?, 512, 8, 8) (3, 3, 512, 512) 16x16/Skip 262144 (?, 512, 8, 8) (1, 1, 512, 512) 8x8/Conv0 2359808 (?, 512, 8, 8) (3, 3, 512, 512) 8x8/Conv1_down 2359808 (?, 512, 4, 4) (3, 3, 512, 512) 8x8/Skip 262144 (?, 512, 4, 4) (1, 1, 512, 512) 4x4/MinibatchStddev - (?, 513, 4, 4) - 4x4/Conv 2364416 (?, 512, 4, 4) (3, 3, 513, 512) 4x4/Dense0 4194816 (?, 512) (8192, 512) Output 513 (?, 1) (512, 1) --- --- --- --- Total 29012513 Exporting sample images... Replicating networks across 1 GPUs... Initializing augmentations... Setting up optimizers... Constructing training graph... Finalizing training ops... Initializing metrics... Training for 25000 kimg... tick 0 kimg 0.0 time 4m 20s sec/tick 84.4 sec/kimg 5272.93 maintenance 176.0 gpumem 10.1 augment 0.000 Evaluating metrics... Downloading https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada/pretrained/metrics/inception_v3_features.pkl ... done Calculating real image statistics for fid50k_full... network-snapshot-000000 time 14m 29s fid50k_full 117.5238 tick 1 kimg 4.0 time 30m 52s sec/tick 690.3 sec/kimg 172.57 maintenance 900.9 gpumem 10.1 augment 0.015 tick 2 kimg 8.0 time 42m 24s sec/tick 692.1 sec/kimg 173.03 maintenance 0.0 gpumem 10.1 augment 0.030 tick 3 kimg 12.0 time 53m 57s sec/tick 693.2 sec/kimg 173.29 maintenance 0.0 gpumem 10.1 augment 0.045 tick 4 kimg 16.0 time 1h 05m 31s sec/tick 694.1 sec/kimg 173.52 maintenance 0.0 gpumem 10.1 augment 0.056 tick 5 kimg 20.0 time 1h 17m 06s sec/tick 694.9 sec/kimg 173.71 maintenance 0.0 gpumem 10.1 augment 0.066 tick 6 kimg 24.0 time 1h 28m 42s sec/tick 695.8 sec/kimg 173.96 maintenance 0.0 gpumem 10.1 augment 0.073 tick 7 kimg 28.0 time 1h 40m 18s sec/tick 696.3 sec/kimg 174.08 maintenance 0.0 gpumem 10.1 augment 0.080 tick 8 kimg 32.0 time 1h 51m 55s sec/tick 697.2 sec/kimg 174.31 maintenance 0.0 gpumem 10.1 augment 0.089 tick 9 kimg 36.0 time 2h 03m 33s sec/tick 697.6 sec/kimg 174.39 maintenance 0.0 gpumem 10.1 augment 0.098 tick 10 kimg 40.0 time 2h 15m 11s sec/tick 698.7 sec/kimg 174.66 maintenance 0.0 gpumem 10.1 augment 0.105 Evaluating metrics... network-snapshot-000040 time 12m 39s fid50k_full 100.5016 tick 11 kimg 44.0 time 2h 39m 45s sec/tick 698.6 sec/kimg 174.65 maintenance 774.5 gpumem 10.1 augment 0.111 tick 12 kimg 48.0 time 2h 51m 23s sec/tick 698.7 sec/kimg 174.67 maintenance 0.0 gpumem 10.1 augment 0.117 tick 13 kimg 52.0 time 3h 03m 02s sec/tick 698.5 sec/kimg 174.64 maintenance 0.0 gpumem 10.1 augment 0.122 tick 14 kimg 56.0 time 3h 14m 42s sec/tick 700.6 sec/kimg 175.16 maintenance 0.0 gpumem 10.1 augment 0.126 tick 15 kimg 60.0 time 3h 26m 23s sec/tick 700.3 sec/kimg 175.09 maintenance 0.1 gpumem 10.1 augment 0.128 tick 16 kimg 64.0 time 3h 38m 04s sec/tick 701.3 sec/kimg 175.33 maintenance 0.0 gpumem 10.1 augment 0.131 tick 17 kimg 68.0 time 3h 49m 46s sec/tick 701.6 sec/kimg 175.39 maintenance 0.0 gpumem 10.1 augment 0.132 tick 18 kimg 72.0 time 4h 01m 28s sec/tick 701.8 sec/kimg 175.45 maintenance 0.0 gpumem 10.1 augment 0.132 tick 19 kimg 76.0 time 4h 13m 08s sec/tick 700.6 sec/kimg 175.16 maintenance 0.0 gpumem 10.1 augment 0.138 tick 20 kimg 80.0 time 4h 24m 48s sec/tick 699.4 sec/kimg 174.84 maintenance 0.0 gpumem 10.1 augment 0.140 Evaluating metrics... network-snapshot-000080 time 12m 43s fid50k_full 87.8921 tick 21 kimg 84.0 time 4h 49m 37s sec/tick 698.7 sec/kimg 174.67 maintenance 790.3 gpumem 10.1 augment 0.145 tick 22 kimg 88.0 time 5h 01m 16s sec/tick 699.4 sec/kimg 174.86 maintenance 0.0 gpumem 10.1 augment 0.146 tick 23 kimg 92.0 time 5h 12m 55s sec/tick 698.4 sec/kimg 174.61 maintenance 0.0 gpumem 10.1 augment 0.148 tick 24 kimg 96.0 time 5h 24m 34s sec/tick 699.4 sec/kimg 174.85 maintenance 0.0 gpumem 10.1 augment 0.146 tick 25 kimg 100.0 time 5h 36m 13s sec/tick 698.8 sec/kimg 174.71 maintenance 0.0 gpumem 10.1 augment 0.148 tick 26 kimg 104.0 time 5h 47m 51s sec/tick 698.4 sec/kimg 174.61 maintenance 0.0 gpumem 10.1 augment 0.147 tick 27 kimg 108.0 time 5h 59m 30s sec/tick 698.9 sec/kimg 174.71 maintenance 0.0 gpumem 10.1 augment 0.151 tick 28 kimg 112.0 time 6h 11m 09s sec/tick 699.2 sec/kimg 174.80 maintenance 0.0 gpumem 10.1 augment 0.155 tick 29 kimg 116.0 time 6h 22m 48s sec/tick 698.9 sec/kimg 174.73 maintenance 0.0 gpumem 10.1 augment 0.157 tick 30 kimg 120.0 time 6h 34m 28s sec/tick 699.6 sec/kimg 174.90 maintenance 0.0 gpumem 10.1 augment 0.157 Evaluating metrics... network-snapshot-000120 time 12m 35s fid50k_full 76.9912 tick 31 kimg 124.0 time 6h 58m 58s sec/tick 699.5 sec/kimg 174.89 maintenance 770.4 gpumem 10.1 augment 0.158 tick 32 kimg 128.0 time 7h 10m 40s sec/tick 701.7 sec/kimg 175.42 maintenance 0.0 gpumem 10.1 augment 0.161 tick 33 kimg 132.0 time 7h 22m 21s sec/tick 701.4 sec/kimg 175.36 maintenance 0.0 gpumem 10.1 augment 0.165 tick 34 kimg 136.0 time 7h 34m 03s sec/tick 701.7 sec/kimg 175.42 maintenance 0.0 gpumem 10.1 augment 0.166 tick 35 kimg 140.0 time 7h 45m 44s sec/tick 701.7 sec/kimg 175.42 maintenance 0.0 gpumem 10.1 augment 0.166 tick 36 kimg 144.0 time 7h 57m 27s sec/tick 702.6 sec/kimg 175.64 maintenance 0.0 gpumem 10.1 augment 0.168 tick 37 kimg 148.0 time 8h 09m 08s sec/tick 701.1 sec/kimg 175.28 maintenance 0.0 gpumem 10.1 augment 0.171 tick 38 kimg 152.0 time 8h 20m 51s sec/tick 702.8 sec/kimg 175.69 maintenance 0.0 gpumem 10.1 augment 0.170 tick 39 kimg 156.0 time 8h 32m 32s sec/tick 701.1 sec/kimg 175.28 maintenance 0.0 gpumem 10.1 augment 0.174 tick 40 kimg 160.0 time 8h 44m 13s sec/tick 701.4 sec/kimg 175.35 maintenance 0.0 gpumem 10.1 augment 0.174 Evaluating metrics... network-snapshot-000160 time 12m 36s fid50k_full 70.5429 tick 41 kimg 164.0 time 9h 08m 47s sec/tick 702.4 sec/kimg 175.61 maintenance 771.6 gpumem 10.1 augment 0.178 tick 42 kimg 168.0 time 9h 20m 29s sec/tick 701.5 sec/kimg 175.37 maintenance 0.0 gpumem 10.1 augment 0.180 tick 43 kimg 172.0 time 9h 32m 09s sec/tick 700.1 sec/kimg 175.03 maintenance 0.0 gpumem 10.1 augment 0.184 tick 44 kimg 176.0 time 9h 43m 50s sec/tick 700.6 sec/kimg 175.16 maintenance 0.0 gpumem 10.1 augment 0.184 tick 45 kimg 180.0 time 9h 55m 30s sec/tick 700.3 sec/kimg 175.07 maintenance 0.0 gpumem 10.1 augment 0.183 tick 46 kimg 184.0 time 10h 07m 11s sec/tick 701.4 sec/kimg 175.36 maintenance 0.0 gpumem 10.1 augment 0.189 tick 47 kimg 188.0 time 10h 18m 53s sec/tick 701.7 sec/kimg 175.42 maintenance 0.0 gpumem 10.1 augment 0.188 tick 48 kimg 192.0 time 10h 30m 35s sec/tick 702.1 sec/kimg 175.53 maintenance 0.0 gpumem 10.1 augment 0.186 tick 49 kimg 196.0 time 10h 42m 18s sec/tick 702.4 sec/kimg 175.61 maintenance 0.0 gpumem 10.1 augment 0.184 tick 50 kimg 200.0 time 10h 54m 00s sec/tick 702.3 sec/kimg 175.58 maintenance 0.0 gpumem 10.1 augment 0.188 Evaluating metrics... network-snapshot-000200 time 12m 36s fid50k_full 66.2340 tick 51 kimg 204.0 time 11h 18m 34s sec/tick 701.6 sec/kimg 175.40 maintenance 772.3 gpumem 10.1 augment 0.187 tick 52 kimg 208.0 time 11h 30m 17s sec/tick 702.8 sec/kimg 175.71 maintenance 0.0 gpumem 10.1 augment 0.193 tick 53 kimg 212.0 time 11h 41m 59s sec/tick 702.2 sec/kimg 175.54 maintenance 0.0 gpumem 10.1 augment 0.200 tick 54 kimg 216.0 time 11h 53m 43s sec/tick 704.3 sec/kimg 176.06 maintenance 0.0 gpumem 10.1 augment 0.198 tick 55 kimg 220.0 time 12h 05m 27s sec/tick 703.4 sec/kimg 175.85 maintenance 0.0 gpumem 10.1 augment 0.201 tick 56 kimg 224.0 time 12h 17m 10s sec/tick 703.7 sec/kimg 175.91 maintenance 0.0 gpumem 10.1 augment 0.203 tick 57 kimg 228.0 time 12h 28m 53s sec/tick 702.6 sec/kimg 175.64 maintenance 0.0 gpumem 10.1 augment 0.205 tick 58 kimg 232.0 time 12h 40m 35s sec/tick 702.3 sec/kimg 175.57 maintenance 0.0 gpumem 10.1 augment 0.207 tick 59 kimg 236.0 time 12h 52m 17s sec/tick 702.1 sec/kimg 175.52 maintenance 0.0 gpumem 10.1 augment 0.211 tick 60 kimg 240.0 time 13h 04m 02s sec/tick 704.3 sec/kimg 176.08 maintenance 0.0 gpumem 10.1 augment 0.212 Evaluating metrics... network-snapshot-000240 time 12m 35s fid50k_full 63.1304 tick 61 kimg 244.0 time 13h 28m 37s sec/tick 704.3 sec/kimg 176.07 maintenance 771.2 gpumem 10.1 augment 0.214 tick 62 kimg 248.0 time 13h 40m 21s sec/tick 704.3 sec/kimg 176.08 maintenance 0.0 gpumem 10.1 augment 0.217 tick 63 kimg 252.0 time 13h 52m 05s sec/tick 703.6 sec/kimg 175.91 maintenance 0.0 gpumem 10.1 augment 0.218 tick 64 kimg 256.0 time 14h 03m 49s sec/tick 703.5 sec/kimg 175.87 maintenance 0.0 gpumem 10.1 augment 0.222 tick 65 kimg 260.0 time 14h 15m 32s sec/tick 702.9 sec/kimg 175.72 maintenance 0.1 gpumem 10.1 augment 0.222 tick 66 kimg 264.0 time 14h 27m 15s sec/tick 703.9 sec/kimg 175.96 maintenance 0.0 gpumem 10.1 augment 0.225 tick 67 kimg 268.0 time 14h 38m 59s sec/tick 703.1 sec/kimg 175.78 maintenance 0.0 gpumem 10.1 augment 0.227 tick 68 kimg 272.0 time 14h 50m 42s sec/tick 703.8 sec/kimg 175.95 maintenance 0.0 gpumem 10.1 augment 0.229 tick 69 kimg 276.0 time 15h 02m 26s sec/tick 703.5 sec/kimg 175.87 maintenance 0.0 gpumem 10.1 augment 0.231 tick 70 kimg 280.0 time 15h 14m 09s sec/tick 703.4 sec/kimg 175.86 maintenance 0.0 gpumem 10.1 augment 0.232 Evaluating metrics... network-snapshot-000280 time 12m 36s fid50k_full 64.3989 tick 71 kimg 284.0 time 15h 38m 56s sec/tick 702.7 sec/kimg 175.67 maintenance 784.3 gpumem 10.1 augment 0.234 tick 72 kimg 288.0 time 15h 50m 40s sec/tick 703.6 sec/kimg 175.90 maintenance 0.0 gpumem 10.1 augment 0.235 tick 73 kimg 292.0 time 16h 02m 23s sec/tick 703.4 sec/kimg 175.85 maintenance 0.0 gpumem 10.1 augment 0.238 tick 74 kimg 296.0 time 16h 14m 08s sec/tick 704.6 sec/kimg 176.14 maintenance 0.0 gpumem 10.1 augment 0.239 tick 75 kimg 300.0 time 16h 25m 52s sec/tick 704.5 sec/kimg 176.13 maintenance 0.0 gpumem 10.1 augment 0.242 tick 76 kimg 304.0 time 16h 37m 37s sec/tick 704.2 sec/kimg 176.05 maintenance 0.0 gpumem 10.1 augment 0.249 tick 77 kimg 308.0 time 16h 49m 21s sec/tick 704.7 sec/kimg 176.18 maintenance 0.0 gpumem 10.1 augment 0.254 tick 78 kimg 312.0 time 17h 01m 06s sec/tick 704.8 sec/kimg 176.19 maintenance 0.0 gpumem 10.1 augment 0.256 tick 79 kimg 316.0 time 17h 12m 50s sec/tick 704.2 sec/kimg 176.06 maintenance 0.0 gpumem 10.1 augment 0.254 tick 80 kimg 320.0 time 17h 24m 35s sec/tick 704.5 sec/kimg 176.13 maintenance 0.0 gpumem 10.1 augment 0.253 Evaluating metrics... network-snapshot-000320 time 12m 34s fid50k_full 59.3415 tick 81 kimg 324.0 time 17h 49m 09s sec/tick 704.5 sec/kimg 176.13 maintenance 769.4 gpumem 10.1 augment 0.251 tick 82 kimg 328.0 time 18h 00m 54s sec/tick 704.7 sec/kimg 176.18 maintenance 0.0 gpumem 10.1 augment 0.251 tick 83 kimg 332.0 time 18h 12m 38s sec/tick 704.8 sec/kimg 176.21 maintenance 0.0 gpumem 10.1 augment 0.250 tick 84 kimg 336.0 time 18h 24m 24s sec/tick 705.4 sec/kimg 176.36 maintenance 0.0 gpumem 10.1 augment 0.256 tick 85 kimg 340.0 time 18h 36m 09s sec/tick 705.1 sec/kimg 176.26 maintenance 0.0 gpumem 10.1 augment 0.255 tick 86 kimg 344.0 time 18h 47m 55s sec/tick 706.0 sec/kimg 176.49 maintenance 0.0 gpumem 10.1 augment 0.253 tick 87 kimg 348.0 time 18h 59m 40s sec/tick 705.4 sec/kimg 176.34 maintenance 0.0 gpumem 10.1 augment 0.257 tick 88 kimg 352.0 time 19h 11m 27s sec/tick 706.2 sec/kimg 176.55 maintenance 0.0 gpumem 10.1 augment 0.258 tick 89 kimg 356.0 time 19h 23m 12s sec/tick 705.1 sec/kimg 176.27 maintenance 0.0 gpumem 10.1 augment 0.257 tick 90 kimg 360.0 time 19h 34m 58s sec/tick 706.0 sec/kimg 176.51 maintenance 0.0 gpumem 10.1 augment 0.257 Evaluating metrics... network-snapshot-000360 time 12m 33s fid50k_full 55.5202