Interactive merge takes >30 seconds/frame on 3090

[Zeesy]

Expected behavior

I am using the interactive merge tool with SAEHD on an RTX 3090. Expected behaviour from previous merges on my system would take <2s to make a change to the frame or override config to next frame.

Actual behavior

When trained at <50K and <100K iterations, merging frames occurs faster. After 150K, each change to merge settings (eg. mask_mode, erode_mask_modifier, sharpen_mode, etc.) takes >30s. Override config to next frame/previous frame also takes >30s.

Steps to reproduce

1. Train model using SAEHD with sot-m color_transfer_mode=1. Settings:

==     Current iteration: 150000             ==
==                                           ==
==-------------- Model Options --------------==
==                                           ==
==            resolution: 128                ==
==             face_type: f                  ==
==     models_opt_on_gpu: True               ==
==                 archi: df-ud              ==
==               ae_dims: 256                ==
==                e_dims: 64                 ==
==                d_dims: 64                 ==
==           d_mask_dims: 22                 ==
==       masked_training: True               ==
==       eyes_mouth_prio: True               ==
==           uniform_yaw: True               ==
==             adabelief: True               ==
==            lr_dropout: n                  ==
==           random_warp: True               ==
==       true_face_power: 0.0                ==
==      face_style_power: 0.0                ==
==        bg_style_power: 0.0                ==
==               ct_mode: none               ==
==              clipgrad: False              ==
==              pretrain: False              ==
==       autobackup_hour: 0                  ==
== write_preview_history: False              ==
==           target_iter: 150000             ==
==       random_src_flip: False              ==
==       random_dst_flip: True               ==
==            batch_size: 8                  ==
==             gan_power: 0.0                ==
==        gan_patch_size: 16                 ==
==              gan_dims: 16                 ==
==                                           ==
==--------------- Running On ----------------==
==                                           ==
==          Device index: 0                  ==
==                  Name: GeForce RTX 3090   ==
==                  VRAM: 21.31GB            ==
==                                           ==
===============================================

2. Merge SAEHD on GPU using interactive merge with 8 workers (based on 8 cores for Ryzen 7 3800). Example settings:

MergerConfig 00131.jpg:
Mode: overlay
mask_mode: learned-prd*learned-dst
erode_mask_modifier: 18
blur_mask_modifier: 123
motion_blur_power: 0
output_face_scale: 0
color_transfer_mode: sot-m
sharpen_mode : box
blursharpen_amount : 1
super_resolution_power: 1
image_denoise_power: 6
bicubic_degrade_power: 7
color_degrade_power: 6
================
Merging:  13%|########3                                                         | 131/1037 [1:11:45<8:16:18, 32.87s/it]

Above example is for a completed frame.

Other relevant information

Windows 10 v10.0.19043 Build 19043
Deepfacelab build 05_21_2021

[iperov]

sot-m is cpu intensive method
with super_resolution_power it can take too long
set super_resolution_power to 0