How to add Light Depth model to IW3_desktop?

[Jokeru18]

I'm getting good results with Distilled Any S while trying to play BF6 on RTX 3090. Gameplay is smooth at times but there are many instances where it stutters a lot so was thinking if I can try a light model.
Googling I found that LIght Depth model might be able to run faster.
With help of Grok I tried to add it to Iw3 and seems to point me in the right direction but was never able to make it show up in GUI.
First I tried the dumb approach, just renaming the file to some other one that was already integrated but it was giving errors.

https://github.com/suxrobgm/lightdepth

[nagadomi]

Using gemini-cli, code investigation, planning, implementation, and even fixing runtime errors can all be done automatically. You may need to handle application testing and provide detailed feedback when it does not work.

Also, regarding performance, please check whether you are using the following settings.

• Screenshot=wc_cuda
• torch.compile https://github.com/nagadomi/nunif/blob/master/windows_package/docs/torch_compile.md

[Jokeru18]

Thank you very much for the reply. You are a wizard ! This tool is fantastic.
Torch compile makes a huge difference, about 40-50% improvement.
Now I'm not sure I need a lighter model :), I will try anyhow as it doesn't hurt to learn something new.

[Jokeru18]

Was able to integrate it with Gemini-cli but reached my limit when trying to fix this error. LightDepth work with CPU but not with GPU
I continued with the 2.5 light model but that is useless....

[nagadomi]

It is probably related to FP16 (autocast). If it works on CPU, turning off the FP16 option may allow it to work for now.

For investigating errors, open a prompt using nunif-prompt.bat, then run python -m iw3.gui to launch the GUI.
Detailed error information will be shown in the console.
If you paste the error information into gemini-cli, it should be able to understand the issue.

A better approach would be to have gemini-cli run iw3.cli command directly, but this may be difficult due to environment variable issues (That is set in setenv.bat).
For example, you can test it with a command like the following.
python -m iw3.cli -i input_image.png -o output_image.png --depth-model Any_S
Replace Any_S with the LightDepth model name.

[nagadomi]

The above comment confused iw3 and iw3.desktop.
The GUI command for iw3.desktop is python -m iw3.desktop.gui.
The CLI can also be started with python -m iw3.desktop.cli --screenshot wc_cuda --local-viewer --depth-model Any_S.

[Jokeru18]

Thanks. Indeed related to FP16, Copilot_Cli fixed it and was able to test the model.

IT's doesn't look great. Perfornace without any game running was 25% higher compared to DistillAnyS but lower with the game running. Also there was something wrong with the depth and object stability. After few passes copilot managed to make it look ok but was still very unstable and performing worse.

PS: I tried to work with Grok_cli as well but I think they block it for free licente because as soon as I run it my quota goes to 0
So happy you helped me discovered the existence of this AI_Cli , didn't realize something like this was available to the public.

[nagadomi]

Glad it worked. AI tools that can do that are really impressive.
lightdepth appears to be a small project created for a class report. The evaluation method for DAv2 (relative depth, disparity output) may need to be reviewed. An AI that can read code, such as gemini-cli, may be able to evaluate those aspects properly.

[Jokeru18]

After numerous tries I manager to successfully integrate the ResNet18 LIght Depth into IW3-Desktop.
On first few tries the depth was all messed up. Floating objects, incorrect placement, blurry image and compressed depth.

The performance is about 50% better compared to Distill_Any_S which is already quite efficient. It is able to hit crazy high FPS with RTX 3090
I can now play smoothly BF6 in 3D.

I add the files here if anyone interested and some instructions made by QWEN-cli (without the trained model checkpoint which you can download yourself) https://github.com/suxrobgm/lightdepth

nunif.zip
README.md

PS: I will probably tweak a bit to give it more depth out of the box it it's still good enough in this state and very temporally stable.

[Jokeru18]

But you use it for realtime conversion? Best I was getting was 5 fps. Maybe if you have more info on the codec or some specific on the integration Incould take another look

[KolaKater]

No I nevery use realtime conversion. in reality, your LightDepth for iw3 is really slow with my computer, especially the result is really bad.
this is my backward warp (orinallly from Nagadomi) of the FlashDepth:

def backward_warp_mlbw(img_bgr: np.ndarray,
                       depth_np: np.ndarray,
                       stereo_strength):

    h, w = depth_np.shape

    img = torch.from_numpy(
        cv2.cvtColor(img_bgr, cv2.COLOR_BGR2RGB)
    ).float().permute(2, 0, 1).unsqueeze(0).to(DEVICE) / 255.0

    depth = torch.from_numpy(depth_np).float().unsqueeze(0).unsqueeze(0).to(DEVICE)

    if isinstance(stereo_strength, str) and stereo_strength.lower() == "auto":
        stereo_strength = 1.0

    base_divergence = float(stereo_strength) * 1.5

    toe = float(TOE_IN_STRENGTH) if "TOE_IN_STRENGTH" in globals() else 0.0
    div_scale = 1.0 + toe
    divergence = base_divergence * div_scale
    convergence = 0.5 - 0.25 * toe
    convergence = max(0.1, min(0.9, convergence))

    stereo_model = load_stereo_model(divergence)

    left_t = apply_divergence_nn_delta_weight(
        stereo_model, img, depth,
        divergence=divergence,
        convergence=convergence,
        steps=1,
        shift=-1,
        return_mask=False
    )

    right_t = apply_divergence_nn_delta_weight(
        stereo_model, img, depth,
        divergence=divergence,
        convergence=convergence,
        steps=1,
        shift=1,
        return_mask=False
    )

    left = (left_t[0].permute(1, 2, 0).cpu().numpy() * 255).astype(np.uint8)
    right = (right_t[0].permute(1, 2, 0).cpu().numpy() * 255).astype(np.uint8)

    left = cv2.cvtColor(left, cv2.COLOR_RGB2BGR)
    right = cv2.cvtColor(right, cv2.COLOR_RGB2BGR)

    return left, right

[laiyangli001]

The LightDepth author has deleted the LightDepth.pth from google drive,can you upload to google drive or here

[Jokeru18]

Yes I will upload my own version in the next days but first i need to isolate the addiotional code as there are some changes. I have a 1st version but I'm not that happy with the range so I keep working on the training. Is still better than the original lightdepth checkpoint

[KolaKater]

Yes I will upload my own version in the next days but first i need to isolate the addiotional code as there are some changes. I have a 1st version but I'm not that happy with the range so I keep working on the training. Is still better than the original lightdepth checkpoint

please don't forget to work on FlashDepth für iw3, if you need some more codes I will give you if I can help. Thank you

[KolaKater]

actually I test the speed only with the timer and the quality with my own eyes. In my opinion, Distill and FlashDepth have the best quality and speed, FlashDepth is quicker than Distill. and The borders of object from FlashDepth looks better than Distill. If Nagadomi would use FlashDepth for iw3, it would be very nice, I would be even quicker and with better quality with the the iw3 inpaint

[KolaKater]

here is my Depth inference for FlashDepth:

@torch.no_grad()
def run_depth_flashdepth(model, img_bgr: np.ndarray) -> np.ndarray:
    # print(">>> run_depth_flashdepth CALLED")
    h0, w0 = img_bgr.shape[:2]

    max_size = 1280
    scale = min(max_size / max(h0, w0), 1.0)

    if scale < 1.0:
        proc_w = int(w0 * scale)
        proc_h = int(h0 * scale)
        bgr_proc = cv2.resize(img_bgr, (proc_w, proc_h), interpolation=cv2.INTER_AREA)
    else:
        bgr_proc = img_bgr.copy()
        proc_h, proc_w = h0, w0

    h = (proc_h + 13) // 14 * 14
    w = (proc_w + 13) // 14 * 14

    bgr_resized = cv2.resize(bgr_proc, (w, h))
    rgb = cv2.cvtColor(bgr_resized, cv2.COLOR_BGR2RGB)

    frame = torch.from_numpy(rgb).float().permute(2, 0, 1).unsqueeze(0).to(DEVICE) / 255.0
    B, C, H, W = frame.shape
    ph, pw = H // model.patch_size, W // model.patch_size

    with torch.cuda.amp.autocast(dtype=torch.float16):
        feat = model.get_dpt_features(frame, input_shape=(B, C, H, W))
        depth = model.final_head(feat, ph, pw)

    depth_np = depth[0].float().cpu().numpy()

    depth_np = cv2.resize(depth_np, (proc_w, proc_h), interpolation=cv2.INTER_CUBIC)
    if scale < 1.0:
        depth_np = cv2.resize(depth_np, (w0, h0), interpolation=cv2.INTER_CUBIC)

    depth_np -= depth_np.min()
    depth_np /= max(depth_np.max(), 1e-6)

    return depth_np.astype(np.float32)

[KolaKater]

Just I checked, your computer is about 2 times quicker than mine, if the FlashDepth models work in the right way on iw3, they would be the quickest models with top quality for iw3

[Jokeru18]

Would be great, the quality of Lightdepth is quite bad, that's why Ibstarted training my own model but runing into some limitation of the codecs. Already have a working model with decent 3D but I want to try to find the moddleground btw depth and temporal stability. My model on Flasdepth achieves now 60-70 % more fps conpared to DistillS but the depth model is not very detailed it focuses on big objects not bump texture ....

[Jokeru18]

This week I am too busy but next week I will give it a go

[KolaKater]

My test results and conclusions

1. Speed comparison on RTX 3070

   FlashDepth L is significantly faster than Distill L.

   FlashDepth M performs at roughly the same speed as Distill M.

   FlashDepth S is faster than Distill S.

2. Quality comparison
   a. Object shape preservation

FlashDepth generally preserves object shapes better after conversion, while Distill tends to deform shapes more often.
b. Depth characteristics

Distill usually produces stronger, more aggressive depth.

FlashDepth produces depth that feels more natural, comfortable, and accurate overall.

c. Model‑specific observations (FlashDepth)

FlashDepth S offers an excellent balance: very good depth with very high speed.

FlashDepth M is also fast, but sometimes produces less accurate depth for mid‑range objects.

FlashDepth L and FlashDepth S are the strongest models in terms of quality, with S being much faster.

d. Model‑specific observations (Distill)

Distill L and Distill S frequently produce incorrect depth.

Distill B is, in my opinion, the best Distill model:

    very fast (only slightly slower than Distill S)

    more consistent and reliable depth than the other Distill variants.

P.S.:
I have also tested many images with Depth Anything V3, and in my opinion Distill B, S and FlashDepth L, S deliver far better results. Anyone can compare them side‑by‑side and see the difference for themselves.

[IkariDevGIT]

Could you compare VDA_Stream_L and VDA_Stream_B too?

[KolaKater]

Could you compare VDA_Stream_L and VDA_Stream_B too?

In my Eyes, VDA and Distill have the similar speed, but Distill have better quality. So I don't really use VDA, it's name sounds good for videos, but in really from my test the quality is not good. I would really suggest to use distill B, FlashDepth S on iw3, for better quality you can use FlashDepth L. FlashDepth is developed for Videos, but from my Tests, it is also very good for images

[IkariDevGIT]

Could you compare VDA_Stream_L and VDA_Stream_B too?

In my Eyes, VDA and Distill have the similar speed, but Distill have better quality. So I don't really use VDA, it's name sounds good for videos, but in really from my test the quality is not good. I would really suggest to use distill B, FlashDepth S on iw3, for better quality you can use FlashDepth L. FlashDepth is developed for Videos, but from my Tests, it is also very good for images

FlashDepth is in iw3 now? Ill test that then.

[KolaKater]

Could you compare VDA_Stream_L and VDA_Stream_B too?

In my Eyes, VDA and Distill have the similar speed, but Distill have better quality. So I don't really use VDA, it's name sounds good for videos, but in really from my test the quality is not good. I would really suggest to use distill B, FlashDepth S on iw3, for better quality you can use FlashDepth L. FlashDepth is developed for Videos, but from my Tests, it is also very good for images

FlashDepth is in iw3 now? Ill test that then.

No. I could not make FlashDepth work on iw3. From my understanding, if Nagadomi would support it officially, FlashDepth will run even quicker, because of the way how iw3 works.

[Jokeru18]

This is the best I was able to achieve with forcing resolution limit and skiping Mamba.

It's about half of what DistillNy S gets.