Skyrim: Special Edition + ENB freezes before title screen

[elloellie]

Software information

Skyrim Special Edition being played via WineSteam on Lutris. I have installed Skyrim Script Extender 64 and as such launch the game via skse64_loader.exe. This requires a special version of wine which is detailed below. When launched with just SKSE, things run absolutely perfectly. Same goes for SKSE with any combination of typical mods.

The problem arises when trying to use ENB, a graphics mod that essentially replaces Skyrim special edition's native post-processing. ENB is more similar to SKSE64 than other mods, as it is installed to ~/Steam/steamapps/common/Skyrim Special Edition instead of ~/Steam/steamapps/common/Skyrim Special Edition/Data. Installation entails dragging "d3d11.dll" and "d3dcompiler_46e.dll" into this Skyrim Special Edition folder, which auto generates an enblocal.ini and an enbseries.ini when launched. Other files are included in the downloaded .zip, but they are only placeholders for ENB preset creators to edit and aren't needed for the ENB to run.

The game will launch, however, it will freeze on this screen:

I have let it set for at least an hour and nothing will happen. This happens before the Skyrim main menu appears. I have tried launching the game without SKSE and without any mods enabled. This screen, on my heavily modded Windows set-up, would take a minute, max, to load through.

Given this is such a niche scenario, I don't expect anything to come from this! I just wanted to bring the issue to your attention and see if it was an easy fix :) Also sorry if you knew all this about ENBs and all that, I didn't want to leave you hanging with no info! If you do need more info on Skyrim modding to understand this problem I'd be happy to help. Thanks for such an amazing program!

System information

• GPU: Geforce GTX 1070
• Driver: 415.27 (Proprietary)
• Wine version: 3.21 (on Lutris, it's called tkg-fo4-3.21-x86_64. It includes a patch which can be found here to make script extenders used in Bethesda game modding work properly)
• DXVK version: 0.95

Apitrace file(s)

SkyrimSE.trace
I'm not entirely sure I did this right, so if it's broken lemme know!

Log files

SkyrimSE_dxgi.log
SkyrimSE_d3d11.log

[doitsujin]

The trace file is unfortunately kind of useless. Do you have access to a Windows machine? If so, please try to trace it there, if that is at all possible with ENB.

Other than that, does ENB do any logging? Would be interesting if it's complaining about anything. I'm not even sure if it is possible for it to work on wine, although with SKSE working I don't really see why not.

[K0bin]

I don't know how ENB works but maybe d3dcompiler_46e.dll is a small wrapper around the normal d3dcompiler dll. Have you tried installing that first (winetricks)?

[elloellie]

Sadly, ENB doesn't seem to log anything. I'll turn on SKSE's logs, but I don't imagine they'd include anything about ENB. I'm downloading Skyrim SE onto my Windows 10 laptop as I type this, but it only has Intel integrated graphics, so I'm not sure if that'll impact the trace. Either way, I'll upload the new trace later today.

As for winetweaks, I've tried installing just about every d3d file I could from winetricks; none seem to work, even after configuring them in winecfg.

[elloellie]

Alright, here's the other trace! I used the .dlls linked in the DKVK wiki under the Apitrace section of Common Issues to trace it. Since both ENB and Apitrace had a d3d11.dll, I renamed Apitrace's to traced3d11.dll. If that messed with the results, lemme know and I'll be happy to figure out another way to run it!

[doitsujin]

Basically, if a trace file is smaller than at least a few dozen Megabytes, the file is pretty much guaranteed to be useless. You're not necessarily doing anything wrong, it may just be impossible to trace ENB due to the way it is injected into the process.

Similarly, it's still possible that the game fails to run due to reasons that are unrelated to DXVK. Without any kind of logging it's impossible to tell what might be going on. Can you post the full wine console output?

Since both ENB and Apitrace had a d3d11.dll, I renamed Apitrace's to traced3d11.dll

Yeah, that's never going to work.

[elloellie]

Apologies for being so late to reply, but I've been packing all day! And yeah, I didn't really expect it to work but I figured I should try something at least. Anyways, here's that wine console output. I set the environment variable to WINEDEBUG=+all and VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_standard_validation wine /media/ellie/Games/Lutris/WineSteam/steamapps/common/SkyrimSpecialEdition/skse64_loader.exe for this, so it's pretty big, but if I did it wrong again lemme know and I'll take another go at it.

[ssorgatem]

Are you using a winelib build?

As I understand it, you'd need to replace wine's d3d11.dll with DXVK's (with a winelib build), then override it as (native, builtin), so that the game will pick up ENB's d3d11.dll (which is native), which in turn will call on the built-in d3d11.dll (which is why you need to replace wine's own with DXVK).

[ssorgatem]

Also, maybe you ould first try to get it to work without DXVK?

[qsniyg]

This also happens for Fallout 4 ENB using DXVK 1.1.1. The error being logged is that D3DX11CreateShaderResourceViewFromFileA is a stub.

[doitsujin]

winetricks d3dx11_43 should solve that problem. D3DX11 is not part of D3D11.

[qsniyg]

Thanks for replying. It still doesn't work with the winetricks command. I've attached the log here: f4selog.log

If I set dxgi and d3d11 to builtin instead of native (bypassing DXVK), it doesn't crash.

[doitsujin]

Lots of "hlsl parsing failed". winetricks d3dcompiler_43 or winetricks d3dcompiler_47 should fix that.

[qsniyg]

That allowed it to move further, but unfortunately it still crashed again. fallout4.log

In comparison, this is the log when the ENB dlls are removed (and the menu loads properly):
fallout4_success.log

[doitsujin]

Unfortunately there's no hint at all as to what this might be; I assume it fails to inject itself into DXVK. In that case we're pretty much screwed.

[OrdinaryMagician]

Getting rid of the d3dcompiler_46e.dll file fixes the crashing for me. ENB is able to work, partially at least, since it needs to be able to disassemble game shaders for some things to work, like time of day detection or lighting modifications, but the functions it tries to call for it are just stubs. There are hundreds of lines like this one:
0031:fixme:d3dcompiler:D3DDisassemble data 000000004E117C20, size 3488, flags 0, comments 0000000000000000, disassembly 000000000022F368 stub!

[hedmo]

hi all

i am on this road to and like to assist.i am running sse from steam/proton and everything is working out of the box but enb.i can run enb with OpenGL (PROTON_USE_WINED3D=1) out of the box but with low fps.with Dxvk (PROTON_USE_WINED3D=0) i need to remove d3dcompiler_46e.dll as @OrdinaryMagician wrote or it will crash.

my logs:
steam-489830.zip
SkyrimSE_d3d11.log
SkyrimSE_dxgi.log

regards hedmo

[doitsujin]

Can anyone of you record an apitrace with a working ENB setup (i.e. with wined3d) or is that impossible?

I cannot debug this problem without knowing exactly what causes it to crash.

[doitsujin]

nvm, I found out what's causing the crash - ENB tries to create a broken pixel shader that trips over dxvk. Technically this is an ENB bug, I'll try to work around this.

[doitsujin]

Crash should be fixed, but this will not work in game, it just locks up during the loading screen for some reason.

Edit: Looks like ENB generates horrendously broken shaders. Someone please tell Boris to fix that bloody mess.

[doitsujin]

With latest master I can load successfully into the game now.

[hedmo]

@doitsujin .I have manage to start my wine install with your update . no d3dcompiler_46e.dll removal needed but as you say - horrendously broken shaders. i can not manage to port it to my proton .it crash
like before .do you have any advice. let me know if i am doing it wrong;
hedmo@mya51 ~/Downloads/dxvk-master $ export WINEPREFIX=/home/hedmo/share/Steam/steamapps/compatdata/489830/pfx hedmo@mya51 ~/Downloads/dxvk-master $ ./setup_dxvk.sh installwine: configuration in 'H:\share\Steam\steamapps\compatdata\489830\pfx' has been updated. hedmo@mya51 ~/Downloads/dxvk-master $
and how did you get the gui up?

thank you for this ....
regards hedmo

[doitsujin]

Works fine on Proton for me without additional configuration.

Also, make sure your DXVK build has 2d6a339 included, this fixes more broken ENB shaders.

[hedmo]

@doitsujin where and how do i put them there?

[doitsujin]

put what?

If you're asking about DXVK, copy the files to ~/.steam/steam/steamapps/common/Proton 4.11/dist/lib64/wine/dxvk, not the prefix.

[hedmo]

@doitsujin do you mean the dll.so files i have in the x32/x64 dirs .in ~/.steam/steam/steamapps/common/Proton 4.11/dist/lib64/wine/dxvk-i have dll files

regards hedmo

[doitsujin]

If you have .dll.so files then you are using a winelib build of dxvk for some reason; I'd recommend you rebuild dxvk with mingw instead (which is the default anyway).

[OrdinaryMagician]

Honestly, I've got to congratulate you for putting up with the horrendous mess of breakage that is ENB.

[hedmo]

@OrdinaryMagician .yes i know. i have had it all the time.i have been running ENB in a month now but with opengl before dxvk .i have figured out what is wrong with the gui to.but i have not found why i have broken graphics with dxvk with my nvidia card.

[rockerbacon]

@doitsujin Sorry to bump this old issue but it seems like newer versions are broken again in Skyrim Special Edition.

v0.347 works, v0.435 only works after d3dcompiler_46e.dll is removed. No d3dcompiler and d3d11 winetricks combination I tried worked. Sadly, Boris has the bad habit of deleting releases but I opened a thread on Reddit to see if people can send me older versions to test.

I'm running Valve's Proton 5-09 which supposedly uses a fairly up to date version of DXVK. I also tried replacing the libraries with the ones included here and had the same results.

Let me know if you require any additional information, I will help with anything I can.

[hedmo]

@rockerbacon i still play SSE and using Valve's Proton 5-09. I am stuck with an old ENB because of broken shaders . I have older versions of ENB if you like to have but i will test the latest ENB to day and post back.

regards

[rockerbacon]

@rockerbacon i still play SSE and using Valve's Proton 5-09. I am stuck with an old ENB because of broken shaders . I have older versions of ENB if you like to have but i will test the latest ENB to day and post back.

regards

I managed to get my hands on v0.386 and v0.390 and neither worked. If you have anything different, please do send me (email in my profile).

Wine logs didn't have anything useful, it just gets stuck without any errors or useful warnings.

[hedmo]

i can confirm that v0.435 gives a black screen but this time the game starts and it did not do that
at first when this issue was opened . v0.347 works as @rockerbacon say.

EDIT: i can confirm that v0.435 and down to v0.409 gives a black screen with nvidia-drivers-450.57 but all versions of ENB works with xf86-video-amdgpu . v0.409 did work with older nvidia-drivers but not now .

[ajventer]

This seems to be, at least in part, dependent on your skyrim configs - there are clearly some configurations (produced by BethINI I think) that cause ENB to break.
With a newly created set produced by the skyrimlauncher - it worked again.

[hedmo]

@ajventer

i cannot agree with you.i made a new prefix and downloaded the latest enb...it is the same black screen at the bethesda intro.i did even try GEs proton as there is a patch for skse64 there but it throws me out after the skse is loaded.still stuck with v0.347....

[ajventer]

@ajventer

i cannot agree with you.i made a new prefix and downloaded the latest enb...it is the same black screen at the bethesda intro.i did even try GEs proton as there is a patch for skse64 there but it throws me out after the skse is loaded.still stuck with v0.347....

Yeah, I think I was wrong - further testing has been... muddled. Using Proton-GE I could reliably get 0.347 to work for a while - until I couldn't get it to launch the game at all - I have no idea why.
Right now, using wine 5.20 (TKG build) + winesteam - I have the game working but absolutely no version of ENB works - as soon as SKSE64 is done loading it crashes.
I've tried every version I've got (Back to 0.341) - same results.

But here is the output at crash time - in case it's helpful:
Steam_SetMinidumpSteamID: Caching Steam ID: 76561198118458408 [API loaded no]
06ec:fixme:shell:CustomDestinationList_BeginList 0x8b6ef90 (0xc18fb9c {92ca9dcd-5622-4bba-a805-5e9f541bd8c9} 0xc18fba8): stub
info: Game: SkyrimSE.exe
info: DXVK: v1.7.2
info: Found config file: dxvk.conf
info: Effective configuration:
info: dxgi.syncInterval = 1
info: Built-in extension providers:
info: Win32 WSI
info: OpenVR
Unable to read VR Path Registry from C:\users\metalpoetza\Local Settings\Application Data\openvr\openvrpaths.vrpath
warn: OpenVR: Failed to initialize OpenVR
info: Enabled instance extensions:
info: VK_KHR_get_surface_capabilities2
info: VK_KHR_surface
info: VK_KHR_win32_surface
info: GeForce GTX 1050 Ti:
info: Driver: 440.100.0
info: Vulkan: 1.1.119
info: Memory Heap[0]:
info: Size: 4096 MiB
info: Flags: 0x1
info: Memory Type[7]: Property Flags = 0x1
info: Memory Type[8]: Property Flags = 0x1
info: Memory Heap[1]:
info: Size: 11974 MiB
info: Flags: 0x0
info: Memory Type[0]: Property Flags = 0x0
info: Memory Type[1]: Property Flags = 0x0
info: Memory Type[2]: Property Flags = 0x0
info: Memory Type[3]: Property Flags = 0x0
info: Memory Type[4]: Property Flags = 0x0
info: Memory Type[5]: Property Flags = 0x0
info: Memory Type[6]: Property Flags = 0x0
info: Memory Type[9]: Property Flags = 0x6
info: Memory Type[10]: Property Flags = 0xe
info: D3D11CoreCreateDevice: Probing D3D_FEATURE_LEVEL_11_0
info: D3D11CoreCreateDevice: Using feature level D3D_FEATURE_LEVEL_11_0
info: Device properties:
info: Device name: : GeForce GTX 1050 Ti
info: Driver version : 440.100.0
info: Enabled device extensions:
info: VK_EXT_depth_clip_enable
info: VK_EXT_full_screen_exclusive
info: VK_EXT_host_query_reset
info: VK_EXT_shader_demote_to_helper_invocation
info: VK_EXT_shader_viewport_index_layer
info: VK_EXT_transform_feedback
info: VK_EXT_vertex_attribute_divisor
info: VK_KHR_create_renderpass2
info: VK_KHR_depth_stencil_resolve
info: VK_KHR_draw_indirect_count
info: VK_KHR_driver_properties
info: VK_KHR_image_format_list
info: VK_KHR_sampler_mirror_clamp_to_edge
info: VK_KHR_swapchain
info: Device features:
info: robustBufferAccess : 1
info: fullDrawIndexUint32 : 1
info: imageCubeArray : 1
info: independentBlend : 1
info: geometryShader : 1
info: tessellationShader : 1
info: sampleRateShading : 1
info: dualSrcBlend : 1
info: logicOp : 1
info: multiDrawIndirect : 1
info: drawIndirectFirstInstance : 1
info: depthClamp : 1
info: depthBiasClamp : 1
info: fillModeNonSolid : 1
info: depthBounds : 1
info: multiViewport : 1
info: samplerAnisotropy : 1
info: textureCompressionBC : 1
info: occlusionQueryPrecise : 1
info: pipelineStatisticsQuery : 1
info: vertexPipelineStoresAndAtomics : 0
info: fragmentStoresAndAtomics : 1
info: shaderImageGatherExtended : 1
info: shaderStorageImageExtendedFormats : 1
info: shaderStorageImageReadWithoutFormat : 1
info: shaderStorageImageWriteWithoutFormat : 1
info: shaderClipDistance : 1
info: shaderCullDistance : 1
info: shaderFloat64 : 1
info: shaderInt64 : 1
info: variableMultisampleRate : 1
info: VK_EXT_conditional_rendering
info: conditionalRendering : 1
info: VK_EXT_4444_formats
info: formatA4R4G4B4 : 0
info: formatA4B4G4R4 : 0
info: VK_EXT_custom_border_color
info: customBorderColors : 0
info: customBorderColorWithoutFormat : 0
info: VK_EXT_depth_clip_enable
info: depthClipEnable : 1
info: VK_EXT_extended_dynamic_state
info: extendedDynamicState : 0
info: VK_EXT_host_query_reset
info: hostQueryReset : 1
info: VK_EXT_memory_priority
info: memoryPriority : 0
info: VK_EXT_robustness2
info: robustBufferAccess2 : 0
info: robustImageAccess2 : 0
info: nullDescriptor : 0
info: VK_EXT_shader_demote_to_helper_invocation
info: shaderDemoteToHelperInvocation : 1
info: VK_EXT_transform_feedback
info: transformFeedback : 1
info: geometryStreams : 1
info: VK_EXT_vertex_attribute_divisor
info: vertexAttributeInstanceRateDivisor : 1
info: vertexAttributeInstanceRateZeroDivisor : 1
info: Queue families:
info: Graphics : 0
info: Transfer : 1
info: DXVK: Read 1150 valid state cache entries
info: DXVK: Using 7 compiler threads
warn: winevulkan detected, disabling exclusive fullscreen support
info: Presenter: Actual swap chain properties:
info: Format: VK_FORMAT_B8G8R8A8_UNORM
info: Present mode: VK_PRESENT_MODE_IMMEDIATE_KHR
info: Buffer size: 1920x1080
info: Image count: 3
info: Exclusive FS: 1
0684:err:seh:NtRaiseException Unhandled exception code c0000005 flags 0 addr (nil)
0758:fixme:shell:CustomDestinationList_BeginList 0x8a42fd0 (0xbf2fb9c {92ca9dcd-5622-4bba-a805-5e9f541bd8c9} 0xbf2fba8): stub
0770:fixme:shell:CustomDestinationList_BeginList 0x8a43030 (0xc03fb9c {92ca9dcd-5622-4bba-a805-5e9f541bd8c9} 0xc03fba8): stub
02fc:fixme:file:ReplaceFileW Ignoring flags 2

I'm guessing that unhandled exception near the end is the moment where it dies.

[ajventer]

And the run after that, it suddenly worked - and with the latest ENB even. Though it then hit the black screen. I'll test it again with 0.347 but it seems whatever is causing the exception doesn't happen consistently.

[ajventer]

And now I have it loaded with 0.347 - after two more changes:

1. I built the latest wine-tkg-git with ALL the proton patches enabled (one of those could be making the difference)
2. I explicitely disabled BOTH FSYNC and ESYNC using env vars. I know that SSE guides encourage turning them off - and I figured it can't hurt.
   Thing is - when I did it, the CPU usage during load went WAY down and was neatly distibuted across all cores rather than being 100% on one core like before. So it's entirely possible that 0.347 works but was failing due to core limitations with ESYNC kicking in - while it could threat better without it and avoid the problem.

Anyway, the older version is now working for me and I'm sticking to this as long as that remains true.

[hedmo]

ATM.i have compiled proton with the skse64 patch and downloaded skse64_2_00_19.the game starts now with
v0.347 but ATM i have corrupted save to deal with or just roll back to a working game save 20h back :(.

[ajspandigital]

I'm still on 2_00_17 - would love to move up to 19 because there's a mod I would love ot use that requires it (Enemy Revolution).

Can you link me to the patch in question ? I'd love to tell my wine-tkg-git to use it (should be able to slip it into the community patches).

[hedmo]

@ajspandigital

good luck.i had a running SSE before 2_00_19!

skse64_proton.txt

[ajspandigital]

Thanks, I've now built a package with that patch, but I kept a backup copy of my previous wine package and I'll also backup my prefix before I try it.

[hedmo]

Thanks, I've now built a package with that patch, but I kept a backup copy of my previous wine package and I'll also backup my prefix before I try it.

Backup your Skyrim to before you update skse

[hedmo]

after many hours of testing,crashing and corrupted stuff i have manage to update skse64 to 2_00_19. For me
2_00_19 did not like racemenu-alpha wich had the old skyrim dll file in it and it corrupted my Data folder and FNIS.
I had to use my old backup and update my mods again.i had some fragile mods like XPMSSE and alpha mods to deal with. but now all works and using ATM Proton-5.12-GE-1,ENB and Reshade with all my 80 mods again.

< BUT WE STILL HAVE TO BE ON v0.347. >

if we use nvidia-drivers.......

[ajspandigital]

I'm working well on 2_00_19 as well, and I have racemenu installed (could be a different version).
All my plugins are working except .Net Script Framework (which is probably unrelated - I don't know if it can work under wine at all yet).
But yeah, ENB only works up to v0.347

[hedmo]

yes the latest racemenu worked for me to. for me above v0.347 just throws me out. how about you @ajspandigital ?

[ajspandigital]

Yep, exact same issue for me.

[ecbfry]

All my plugins are working except .Net Script Framework (which is probably unrelated - I don't know if it can work under wine at all yet).

I have .net script framework working under proton, with a number of mods that depend on it also working fine. I think what made the difference was installing dotnet48 using protontricks... but I should really keep a better track of the changes I make.

[ajventer]

Hah, I had mistakenly believed I had already done that, forgot that dotnet is broken under wine 5.20, but based on your post I backed up my prefix and then used an older wine version to run winetricks and install it.

Right now the only .net plugin I'm using with it SSE-Bugfixes. But the .net script framework log says it's working fine.

Thanks.

[hedmo]

skyrim VR has got this to now with v0.410 . just a black screen and no more :(.

[CreativeGrey]

@hedmo
I don't suppose there's a chance you remember how you got your GUI working do you, running V.0347 and it's driving me crazy trying to get the GUI working.

[hedmo]

@hedmo
I don't suppose there's a chance you remember how you got your GUI working do you, running V.0347 and it's driving me crazy trying to get the GUI working.

yes i do . i have a special one made by Boris.it removes the double click problem for us but it does
not help.... the only way to make the GUI pop up is to start it before the main menu shows up. when the sound with the bethesda logo comes at the very first start . hit the keys like a crazy person at the start and off you go.do your stuff and never use it more as you have to have it open all the time because you wont be able to start it after the logo.

[CreativeGrey]

@hedmo
Yikes, all right, thanks.

[foresto]

According to some notes I made a while back, ENB's Shift + Enter hotkeys worked for me when I rolled back to the lutris-4.21 build of Wine. I believe this also fixed ReShade GUI hotkeys.

Unfortunately, that older wine build has other problems that break my Mod Organizer 2 installation, but perhaps this information could help someone narrow down the underlying problem.

[hedmo]

@foresto the GUIs works if you hit the key at boot logo but you cannot close them after.

[foresto]

the GUIs works if you hit the key at boot logo but you cannot close them after.

I understand that. When I used that earlier version of Wine, the GUIs and other hotkeys worked completely, even after the boot logo.