Tekken 7 - Severe bottleneck in certain stages

[Waistless]

It seems like certain stages in this game bottleneck at 100% GPU usage, even at low settings.

The most troubled stages are Brimstone and Fire, where at most camera angles sidestepping around the framerate drops to ~50 (with a few angles climbing back to 60), and in Arctic Snowfall, where the framerate stays at around 35fps, despite being easily capable of 60 at equivalent settings in D3D11.

I have also encountered performance issues in the mishima building and forgotten realm. This performance level is consistent in different linux distros (Kubuntu and Arch) and Windows, ruling out a proton/platform specific issue.

Screenshots with DXVK hud=full attached.

1st screenshot: Twilight Conflict stage working well at 60fps.

2nd screenshot: Brimstone and Fire

3rd screenshot: Arctic Snowfall

Going from a mix of high/medium settings to all low settings: performance in Arctic snowfall barely increases up to 41fps. None of the dxvk.conf options listed in the wiki had a noticeable impact.

Note: Graphics pipelines appears to incrementally increase across different maps, loading the Arctic snowfall map first will result in less than 1000 pipelines and will not have any effect on performance.

TekkenGame-Win64-Shipping_d3d11.log
TekkenGame-Win64-Shipping_dxgi.log

System information

• CPU: Ryzen 5 1600X @ 4GHz
• RAM: 16GB
• GPU: Nvidia GTX 960 4GB
• Driver: 417.35 (Windows) | 415.27 (Linux)
• DXVK version: 0.96

The current workaround is to use potato versions of these maps https://www.tapatalk.com/groups/sfxt_mods/tekken-7-stage-modding-t1476.html Although ideally if these issues can be fixed at the very least for arctic snowfall it would likely improve performance across the board.

edit: should mention when stress testing, always pick 2 bears, seems to put a strain on the engine. I did not pick bears in this instance to show the performance problems as being purely stage related.
edit2: reuploaded log files with debug flag enabled

[doitsujin]

Is it possible to record an apitrace? I don't have the game and screenshots alone aren't exactly helpful.

One thing to note is that this kind of performance loss is somewhat expected on especially older Nvidia cards, this isn't the only game that runs poorly. This is not something that can be fixed in DXVK.

[Tk-Glitch]

@Waistless I have provided a trace to Doitsujin with both Arctic Snowfall and Brimstone and Fire, just in case.

[Waistless]

Thanks @Tk-Glitch , had generated one of arctic snowfall but had to compress and upload. Would by any chance a vktrace whilst using dxvk be useful? If I can figure out how/if its possible to get working.

[doitsujin]

vktrace is sadly not in a usable state, no.

[Waistless]

Unusable as in you can't use what it outputs? Well if so then that's a shame, I'll leave this here anyway just in case or if one day it becomes usable: https://drive.google.com/open?id=14dhr9yBBGbI4wfM6vBn4rT_zC8LMJrXo

(Generated under DXVK + Windows, VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_vktrace)

[doitsujin]

Does the game have an option to disable Tessellation? The trace @Tk-Glitch shared shows that the game uses it in a rather inefficient way, which might very well be the cause of your performance issues.

Other than that, there does not seem to be anything that runs particularly slowly.

[BlazeKl]

the game doesn't have an option for tessellation, the graphics menu is not very specific on what it is actually enabling, is @Tk-Glitch 's trace using high settings? maybe the low preset deactivates tessellation

[Tk-Glitch]

Yeah, I have traced with maximum settings. Though everyone having issues with the game's performance with DXVK so far couldn't fix it with low settings. I can upload a low settings trace if needed @doitsujin

[doitsujin]

@Waistless ping? Can you try setting the maximum tessellation factor to 8 as described on the wiki?

[Waistless]

@doitsujin Apologies for the slow response.

No noticeable difference with that option enabled, even in other circumstances for example two bears in a non modded stage on high textures. (excluding the laggy stages).

I did some more testing, depending on the settings used and on the laggy stages, can end up with this sort of CPU usage behavior:

(note SMT threads disabled through affinity to make the graph clear, has no impact on performance).

Those dips on CPU#6 are a result of a slight change in camera angle depending on how you walk around the stage. Those levels are consistent, its either at 40% usage or fully bottlenecked, with no middle ground. During the lower cpu usage the FPS will sit at 60, at maxxed out it will sit at somewhere below 60, depending on settings. (Also noticed the same behavior on linux). GPU usage itself remains fairly constant. This occurs regardless of VRAM usage sitting at around 2GB used out of 4GB, and other stages appear to use more or similar amounts without the same problem occurring.

I am able to almost fine tune this behavior with all low settings on arctic snowfall using the rendering scale. At 50 (so 50% of 1080p), there is no CPU bottleneck at any camera angle. At 55% this behavior shows up, and at 60% there are only small camera angles where the bottleneck goes away. At 65% the CPU remains bottlenecked and beyond this is where the huge dip in performance comes from.

This also occurs if you leave rendering scale at 50%, all low settings with either Effects on Ultra, or Post Processing at Ultra. Ultra textures has less impact (except for bears), Ultra Shadows have very little impact. These values are different for brimstone and fire, but the end result is the same.

Appreciate the effort btw, so happy that this project exists!

edit: tested dxvk.numCompilerThreads = 6 and 12, no noticeable effect.

[doitsujin]

This is extremely weird, the Rendering Scale option should have absolutely no effect on CPU performance at all since it only resizes the render targets. It just doesn't do anything else.

Can you monitor PCIe bus utilization in the Nvidia settings (on Linux)?

[Waistless]

I monitored this is in Linux(nv settings) and Windows (msi afterburner), same behavior

PCI Generation Gen3
Max PCIE Link Width x16
Max PCIE Link Speed 8.0 GT/S

In the menu the utilization will stay at 1%
When loading it behaves as expected, some spiking to load resources. On linux I saw a couple of 20% spikes, on windows msi afterburner it only shows going as high as maybe 5%.
In game: on Arctic Snowfall stage I cannot get bus utilization above 0%, no matter the settings if I put render scale to 50% for 60 fps, or 100% render scale on all low or all high settings for its slow capped out behavior.

In other stages when I put settings to High and 100% render scale, it can still do 60fps and bus utilization stays at around 1%.

edit: Tested with Native D3D11, arctic snowfall all high settings. 60fps with identical pcie bus behavior, so probably not much to look into there. Also tested FB usage, no real difference compared to native D3D11.

[Waistless]

@doitsujin BREAKTHROUGH! I suspected as much, there were certain objects on the stage causing problems.

Remember those potato stages I linked? Turns out its really easy to remove things from stages, and the potato mod of Arctic Snowfall doesn't actually remove a whole lot of things. So I only took all out of the files in the "Developer" subfolder (hexedit rename to neveloper, 24 dev objects renamed out) while leaving everything else in, paked up a new mod, and now the stage works like a breeze, 60fps at high settings.

Regarding the issue itself, its possible these "developer" objects aren't meant to ever be loaded, hence why I can't see any visual change from removing them. For whatever reason in native DX11 these files might be ignored/not loaded, but they give DXVK a massive headache.

edit: I did some stress testing on Native D3D11, to test the effects of removing dev objects on this stage. I used 2 Bears on Arctic Snowfall, with and without the developers objects removed.

Native D3D11: No mod, 200% RenderScale Ultra: 45-47 FPS
Native D3D11: Dev Objects removed, 200% RenderScale Ultra: 48-51 FPS

So it seems like Native D3D11 does load these objects, albeit at a minimal cost.

edit2: Removing dev objects from other stages such as brimstone and fire hasn't fixed their performance issues. At least its a start though, can chip away at their other objects until I work out what is causing the problem, whether it be too many objects loaded or certain objects tanking performance.

[Waistless]

I narrowed the major cause of the FPS drop and CPU usage in the Arctic Snowfall stage to a small developer uasset called "SnowA_FIXED" which itself links to a larger dev uasset "SnowMaster_LO". Hex editing out their texture asset links in a mod doesn't fix the lag, so I assume its related to one of the core assets themselves.
Just hex editing the link to "SnowA_FIXED" and the stage goes from 35-40fps low settings to 60fps playable on medium settings. Taking out the rest of the dev files and the stage goes to 60fps high settings playable as stated earlier.

Unfortunately, Tekken 7 uses a modified version of Unreal Engine 4.13.2, so I was unable to view the assets in the respective tool (and hence why modders aren't directly using UE for their mods). This makes linking the files rather pointless.

Another more extreme example of the CPU usage spikes I found though, was on infinite azure. When the opponent hits the ground and enough water splash effects occur, game drops slightly below 60 and I get CPU usage that looks like this:

That happens on medium settings, on low settings the effects aren't enough to tip dxvk over the edge and cause high cpu usage. It's almost like it ends up running out of some specific resource and when that happens, has to fallback to the CPU pushing everything through.

[doitsujin]

No such thing as a "CPU fallback" for anything in 3d rendering.

Anyway, thanks for looking into this, not being an UE4 developer I'm just not really sure what to make of it, and why it would cause this obscene CPU utilization. This almost sounds like it's hitting some GPU<>CPU synchronization, which does cause 100% utilization on one thread for a while, but I haven't found any hints to that actually happening in the apitrace that TKG provided.

[juicefreak]

@Waistless

Not all "developer assets" are invisible. I tried the mod where I overwrite all instances of "/Developers/" with junk near the beginning of /TekkenGame/Content/Maps/stg03_geom.umap first, and that seemed to remove all the carts from the trains moving in the background and left just the train engine there. In fact one of the referenced paths is "/Game/Developers/sasakit/T7_SM_ENV_s3railctn03" and another one is "/Game/Developers/sasakit/T7_SM_ENV_s3railtank".

Disabling SnowA_FIXED only seems to not cause any visual differences at a quick glance, but I wonder if there's still something missing there now that just isn't easy to notice.

I've seen other games where assets have been organized into directories by the names of artists working on the game. That most likely is what's happened here too, instead of these being development leftovers.

Still, nice to know that I don't really have to use the mod which removes the background from Arctic Snowfall pretty much completely to play this game on a GTX770 without rebooting to Windows! I'll have to see if there's a single invisible or almost invisible asset reference that can be disabled to fix Brimstone too, unless you happen to find it first... (though the already made "potato mod" for that stage isn't that bad, it just simplifies the background a bit instead of removing it completely like PotatoArcticSnowfall_P.pak does)

[juicefreak]

I just updated the DXVK DLLs manually from 0.95 to 1.0 in my Proton x64 library directory. This improved the framerate on these stages on my GTX770 a lot!

In a 1600x900 window with High preset settings but AA disabled (the settings I usually play with), framerate in unmodified Arctic stage rose to about 50-55fps from the 30s, so it is almost playable without the mod now. Brimstone is now a solid 60fps without the mod, at least without bears on screen.

In fact, Brimstone now runs at 60fps in 1920x1080 resolution with the same settings, Arctic dips to about 44fps then, though. Seems Brimstone can even tolerate raising the settings a little bit further than that, like enabling AA...