[D3D9] Halo Custom Edition: Flickering textures when using Open Sauce

[TheGreatMcPain]

When using DXVK along side Open Sauce, which is a mod for Halo Custom Edition that is required to play mods like SPV3 and Project Takedown, there is some major texture flicker that noticeably affects Cortana.

Video Example

The cutscenes in the video are from the beginning of "The Piller of Autumn", and the beginning of "Two Betrayals".

Apitraces are from the cutscene in "Two Betrayals".

Software information

Download links:

Halo Custom Edition "Requires a Halo PC CD key."
1.0.10 Patch for Custom Edition
Halo CE Refined "Campaign I used in the video"
Universal UI "Allows me to access all campaign missions at the start"
An Open Sauce installer called AmaiSosu "Requires dotnet461 to run"

Installing:

1. winetricks -q msxml4 mfc42 dotnet461 d3dcompiler_4{2,3,7} d3dx9
2. Run setup for Halo Custom Edition
3. Run 1.0.10 patch
4. Extract the '.map' files from Halo CE Refined into the 'maps' directory inside Halo Custom Edition.
5. Replace the 'ui.map' in the 'maps' directory with Universal UI's 'UI.map'.
6. Install Open Sauce into the Halo Custom Edition directory using AmaiSosu.
7. Set dinput8 to (native, builtin)
8. Run haloce.exe with -vidmode height,width,refreshrate

System information

• GPU: GTX 1050
• Driver: NVIDIA Vulkan Beta 440.66.07
• Wine version: wine-staging 5.4
• DXVK version: 1.6 and v1.6-8-g5750a7d8+

Apitrace file(s)

• wined3d-haloce.trace.tar.xz
• dxvk-haloce.trace.tar.xz

Log files

• haloce_d3d9.log

[misyltoad]

Unfortunately, apitrace breaks when recording this trace. There's an option you can enable that fixes it, I'll try and find out what it is tomorrow for you 👍

[TheGreatMcPain]

I was wondering why halo's apitaces weren't showing the thumbnails properly.

I tried recording them on windows with and without OpenSauce, and they all came out the same way.

[misyltoad]

The option is in Halo CE I know that much. Something render target option in the per-adapter graphics options.

[TheGreatMcPain]

I found "DisableAlphaRenderTargets" and "DisableRenderTargets" in config.txt.

I'll try those out tomorrow.

[misyltoad]

Yeah it's one of those iirc -- I think the latter.

[TheGreatMcPain]

Okay I set DisableRenderTargets, and got a working apitrace via wined3d. I tried to get one with dxvk enabled, but it ended up broken.

working-wined3d-haloce.trace.tar.xz

[misyltoad]

That trace seems to render fine on my end, not entirely sure what I am looking for though.

[TheGreatMcPain]

The flickering shows up if i run d3dretrace.exe in a wine prefix with dxvk enabled, I wonder if it's possible to get a trace with dxvk enabled.

[TheGreatMcPain]

Okay, I tried to record a trace with dxvk installed on the prefix, and the problem I'm seeing is not showing up in the trace.

Here's the new trace anyway: new-dxvk-haloce.trace.tar.xz

EDIT: Not sure if these are of any help, but I made some validation layer logs.
With OpenSauce disabled: haloce-layers-no-os.log
With OpenSauce enabled: haloce-layers-with-os.log

EDIT 2: I have also made some renderdoc captures just in case. 4 of them have OpenSauce enabled and the other 4 have OpenSauce disabled.
haloce-dxvk-renderdoc.tar.xz
Removed since I found a solution, and I'd like to save space on my GDrive.

[TheGreatMcPain]

Well, I found the solution, and of course it had to be the most simple solution.
Based on Halo 2's similar graphical glitches I found that
adding "d3d9.invariantPosition = True" to dxvk.conf fixed it.

[TheGreatMcPain]

@Joshua-Ashton I'm going to go ahead and close this issue since I found a solution.

[misyltoad]

I'll add this as a default

[TheGreatMcPain]

I read the 1.7 changelog which mentioned that this issue has been fixed, so I retested with the newest commit without using d3d9.invariantPosition = True inside of dxvk.conf, and found that this issue still persists.