Not working after creators update

[weltmeyer]

Hi,
the directX overlay is not working after the windows 10 creators update, even the demo with notepad is not showing anything on the app

[weltmeyer]

Some ugly working version:

in DirectXOverlayWindow.cs:

private bool CreateWindow() {
            /*
            Handle = Native.CreateWindowEx(
                WindowConstants.WindowExStyleDx,
                WindowConstants.DesktopClass,
                "",
                WindowConstants.WindowStyleDx,
                X,
                Y,
                Width,
                Height,
                IntPtr.Zero,
                IntPtr.Zero,
                IntPtr.Zero,
                IntPtr.Zero);

            if (Handle == IntPtr.Zero) {
                return false;
            }
            */
            var f = new OverlayForm();
            f.Show();
            this.Handle = f.Handle;

OverlayFormClass;

private class OverlayForm:Form
        {
            [DllImport("user32.dll")]
            public static extern int SetWindowLong(IntPtr hWnd, int nIndex, int dwNewLong);

            [DllImport("user32.dll")]
            static extern bool SetLayeredWindowAttributes(IntPtr hwnd, uint crKey, byte bAlpha, uint dwFlags);

            [DllImport("user32.dll", SetLastError = true)]
            public static extern int GetWindowLong(IntPtr hWnd, int nIndex);

            [DllImport("user32.dll")]
            [return: MarshalAs(UnmanagedType.Bool)]
            public static extern bool SetWindowPos(IntPtr hWnd, IntPtr hWndInsertAfter, int X, int Y, int cx, int cy, uint uFlags);

            [DllImport("dwmapi.dll")]
            public static extern void DwmExtendFrameIntoClientArea(IntPtr hWnd, ref int[] pMargins);
            //Styles
            public const UInt32 SWP_NOSIZE = 0x0001;
            public const UInt32 SWP_NOMOVE = 0x0002;
            public const UInt32 TOPMOST_FLAGS = SWP_NOMOVE | SWP_NOSIZE;
            public static IntPtr HWND_TOPMOST = new IntPtr(-1);
            public OverlayForm()
            {
                
                int initialStyle = GetWindowLong(this.Handle, -20);
                SetWindowLong(this.Handle, -20, initialStyle | 0x80000 | 0x20);
                SetWindowPos(this.Handle, HWND_TOPMOST, 0, 0, 0, 0, TOPMOST_FLAGS);
                OnResize(null);

                InitializeComponent();
            }

            private void InitializeComponent()
            {
                this.SuspendLayout();
                // 
                // Form1
                // 
                this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.None;
                this.BackColor = System.Drawing.Color.Black;
                this.ClientSize = new System.Drawing.Size(284, 262);
                this.DoubleBuffered = true;
                this.ForeColor = System.Drawing.SystemColors.ControlText;
                this.FormBorderStyle = System.Windows.Forms.FormBorderStyle.None;
                this.Name = "Form1";
                this.Text = "Form1";
                this.TopMost = true;
                this.TransparencyKey = System.Drawing.Color.Black;
                this.Load += new System.EventHandler(this.Form1_Load);
                this.ResumeLayout(false);
            }

            private void Form1_Load(object sender, EventArgs e)
            {
                this.DoubleBuffered = true;
                this.Width = 1920;// set your own size
                this.Height = 1080;
                this.Location = new System.Drawing.Point(0, 0);
                this.SetStyle(ControlStyles.OptimizedDoubleBuffer |// this reduce the flicker
                    ControlStyles.AllPaintingInWmPaint |
                    ControlStyles.DoubleBuffer |
                    ControlStyles.UserPaint |
                    ControlStyles.Opaque |
                    ControlStyles.ResizeRedraw |
                    ControlStyles.SupportsTransparentBackColor, true);
                this.TopMost = true;
                this.Visible = true;
            }
            protected override void OnPaint(PaintEventArgs e)// create the whole form
            {
                int[] marg = new int[] { 0, 0, Width, Height };
                DwmExtendFrameIntoClientArea(this.Handle, ref marg);
            }
        }

[lolp1]

Any idea the cause?

[weltmeyer]

Yeah it seems like it is with the Color.Transparent...

If i set the backcolor of my form1 to Color.Transparent and the
this.TransparencyKey = System.Drawing.Color.Transparent;

then it is the same way invisible :(

With my solution, i am loosing the transparent/alpha channel as it is calculated against black instead of the pixels of the overlayed app.

Seems like this guy has a similar problem: http://stackoverflow.com/questions/43615946/c-d3d9-alpha-blending-with-creators-update

But i am not deep enough in my knowledge with DX to get a solution out from his posts.

[lolp1]

Interesting, I ask because recently we have had a bug in another overlay where for some users it produces a black screen, for others, it works fine. Did the example you posted above actually work? The "Some ugly working version:"?

[weltmeyer]

Yes, thats what i am using currently.. just the alpha channel from brushes is not working correctly anymore.

[inorton]

Guess what? I just rebooted after the creators update and have hit this :)

[inorton]

@weltmeyer your little hack above made no difference for me. I'm overlaying a DirectX11 app.

[inorton]

Ok.. now that is a little odd.. it works now after a reboot

[inorton]

And stopped working again (not even rebuilt app)..

[lolp1]

@inorton Are you running any other overlays such as discord or some graphics settings software?

[inorton]

Hi @lolp1 turns out my "breakages" were actually my-fault. The hack is working for me (at least as well as the non-hack before the creators update).

I am however having quite frequent crashes when switching windows (except of course when the debugger is attached) I'll open a bug when I have a concrete way of reproducing it

[lukechadwick]

The SharpDx overlay isn't working for me on the fall creators update, yatodev also acknowledged it's because of that update, can't find a solution yet.

[lolp1]

@I-M-I The answer is posted by @weltmeyer seems to work for me.

@inorton Could you show me (or skype me if you wish to not share code publicly at JacobKemple@outlook.com) your render loop or give some more context about your application? Because crashing while switching windows often could be an issue directly related to Process.NET

[inorton]

Sure, @lolp1 https://github.com/inorton/EDMCOverlay/blob/master/EDMCOverlay/EDMCOverlay/OverlayController.cs#L128

[psytrx]

I was curious about this and tried an old overlay I was using. To my surprise, it still seemed to work. Maybe this helps..?
The most impactful difference probably is:

I am using SetWindowLong and SetLayeredWindowAttributes instead of UpdateLayeredWindow:

WinApi.SetWindowLong(Handle, GWL_EXSTYLE, (IntPtr) (WinApi.GetWindowLong(Handle, GWL_EXSTYLE) | WS_EX_LAYERED | WS_EX_TRANSPARENT));
WinApi.SetLayeredWindowAttributes(Handle, 0, 255, LWA_ALPHA);

[inorton]

@inorton Ensure you use the dispatcher for the main UI thread for any operations.

@fjdumont

I was curious about this and tried an old overlay I was using. To my surprise, it still seemed to work. Maybe this helps..?

When you say the most impact, do you mean one works and one doesn't?

[lolp1]

@inorton I got confused in the conversation and edited your reply.. instead of just make a new one. So here it was

"@inorton Ensure you use the dispatcher for the main UI thread for any operations."

and

@fjdumont

I was curious about this and tried an old overlay I was using. To my surprise, it still seemed to work. Maybe this helps..?
When you say the most impact, do you mean one works and one doesn't?

[njbrown09]

how do i apply this fix?

[ghost]

Never mind I figured it out, but it is very glitchy. It flashes really fast, freezes and crashes

[lolp1]

@njbrown09
Refer to @weltmeyer code in the second reply. Change the relevant code in Overlay.NET to his.

@hashcoin-project
It should not be flashing at all or freezing or crashing.. please share your code in which this happens via gisthub or github project if it exist, or email it to me if you wish it to be private at jacobkemple@outlook.com just make sure to tag me here saying you did it in an email if you did.

[bestplay9384]

Is there any chance this will be resolved? I just tried code from second comment here, compiled new Overlay.NET dll and included it into DEMO - its really buggy, what is more its making new window ;/

[njbrown09]

yeah, the fix really dosent work

[njbrown09]

@lolp1 i applied the fix, it flickers and crashes a LOT. I cant go more than 30 seconds without it glitching out

[njbrown09]

whats weird is that it also dosent work on my windows 7 machine. My home windows 10 machine it works on, but my friends windows 10 (latest update) machine it dosent work on, but it also dosent work on my windows 7 computer

[njbrown09]

anyone able to fix this? or at least point me to a way to externally draw an overlay with directx

[inorton]

@njbrown09 I have it working reliably with a alternative method but it does flicker a bit https://github.com/inorton/EDMCOverlay

[Sewer56]

Adding in to this conversation, I on my laptop with Intel graphics and another user of a mod loader I've been working on have hit this after not the Creators but the Fall Creators Update (1709).

The code for the external overlay is very similar to this project and relies of setting window extended flags as WS_EX_LAYERED and WS_EX_TRANSPARENT followed by the use of SetLayeredWindowAttributes to LWA_ALPHA.

My Radeon GPU powered desktop machine works fine though under the same OS.

Trying to dig in, in my case here it appears that WS_EX_LAYERED and WS_EX_TRANSPARENT are not working correctly in tandem, the window isn't clickthrough and there is no transparency (although the overlay still works).

Considered and tried using DwmEnableBlurBehindWindow to see if that'd make at least a change to give transparency although seems that's a no. After many experimental hacks to try getting it to work as-is, I believe that WS_EX_LAYERED is broken and at fault.

[inorton]

Try as I might I could not get alpha blending to work, so Overlay.NET is setup to use black as a mask colour (any black pixels are transparent).

[lolp1]

Will look into it shortly.

[Sewer56]

I should also perhaps mention.

I've also tested things with Insider Preview build 17035 (wasn't getting any newer builds) and now having WS_EX_LAYERED as an extended window style also causes my own overlay to fail to stay topmost of other windows (TopMost is set via changing the WinForm property in C#, not natively). This is of course in addition to lack of transparency and hit testing (although it shouldn't with WS_EX_TRANSPARENT when WS_EX_LAYERED is set).

It sounds like they're working on something in the back end and have been for a while, perhaps they haven't started or done the necessary testing and pushed a few too many changes to production. Might it be something in the works to do with Fluent Design breaking things?

I've also not tested replacing the .NET Window/Windows Form with a salf made native window created via P/Invoke (I.e. declare WNDCLASSEX, register class with RegisterClassEx and create window with CreateWindow), however it'd be weird if this only affected windows created by/managed by the .NET Framework.

Away from home and my main PC with an SSD so it's a pain to test as each new build takes a while to install and as Microsoft doesn't always have a download for the latest build in the fast ring, I have to download an older one then update to latest build via Windows Update. My low-mid end laptop takes almost if not two hours to update build.

Tested 4-5 builds so far, things start breaking with 1709 Fall Creators Update, although likely it started earlier on some Insider Build up till then which we probably wouldn't find on the web anymore. On the same hardware, everything is fine on my end both in 1703 and anything prior, including any OS version down to Windows 7. I'm not rolling Virtual Machines but actually testing on real hardware.

That aside, there's been a relatively huge build that released yesterday, haven't had the chance to give that a go.

[Lanboost]

Ive managed to get it to work with a little trickery. At least for notepad.

Problems detected:
Native.cs
public const string DesktopClass = "#32769";

is illegal, use this instead: (https://docs.microsoft.com/en-us/windows/desktop/winmsg/about-window-classes#elements-of-a-window-class) (a.k.a Its a system only class)
public const string DesktopClass = "Static";

Because we are using a layered window (0x80000 //WS_EX_LAYERED) we need to call
Native.SetLayeredWindowAttributes(Handle,0,0, 0x00000001);
according to https://docs.microsoft.com/en-us/windows/desktop/winmsg/window-features

To create a layered window, specify the WS_EX_LAYERED extended window style when calling the CreateWindowEx function, or call the SetWindowLong function to set WS_EX_LAYERED after the window has been created. After the CreateWindowEx call, the layered window will not become visible until the SetLayeredWindowAttributes or UpdateLayeredWindow function has been called for this window.

Did a lot of other stuff with my code to, but think those two steps are the ones that matter.

Over and out
/Lan

[Sewer56]

I would not have thought that out of all threads; this one would have come back.

I actually managed to get D2D working over a WPF overlay since that time; and actually built a few overlays into a Mod Loader/Hacking Library of mine (Reloaded-Mod-Loader/Reloaded-Overlay) but this sounds very interesting nonetheless.

Since I think my laptop still exhibits this - I might try installing Win10 on it and checking this out over a quick dual boot; but if you did get this right - the key would specifically be public const string DesktopClass = "Static";. I actually tried Native.SetLayeredWindowAttributes(Handle,0,0, 0x00000001); all the way back then; albeit the code I was working with was not Overlay.NET but my own (albeit it was conceptually the same).

Kind regards ^-^

[psytrx]

I currently have a fully working, non-flickering overlay working in my solutions. However, I did not use Overlay.NET, but copied and adjusted pieces of the code in this library for my use cases.
@lolp1 if you are willing to compare our solutions in order to find differences/potential fixes, let me know. I think we can figure something out.

[inorton]

I too went with a different implementation over at https://github.com/inorton/EDMCOverlay feel free to borrow bits of code according to the license :)

[lolp1]

@fjdumont sure, I would like to update the project anyways.

@inorton thanks will look.

[xaviermonin]

Hi,

I'm working on the problem. I have write 2 simple programs in C++ and C# that does exactly same think. They draw a red alpha semi transparent line.
The both works.

What I know now :

• We can't use SetLayeredWindowAttributes and UpdateLayeredWindow in same time.
• CreateWindowEx with WS_EX_LAYERED doesn't work. We must use SetWindowLongPtr with this param just after the window creation.
• They have a problem with the DirectX code.

I keep you informed of my progress.
I attached my VS solution.
OverLay.zip

[psytrx]

Hey, sorry for the late response. This is really simple, but seems to work for me without any problems:

Windows.SetWindowLong(Handle, GWL_EXSTYLE, (IntPtr) (Windows.GetWindowLong(Handle, GWL_EXSTYLE) | WS_EX_LAYERED | WS_EX_TRANSPARENT));
Windows.SetLayeredWindowAttributes(Handle, 0, 255, LWA_ALPHA);

However, I was not able to reposition the D3D device for bordered windows properly, so I'm currently reinstantiating the device.
Hope that helps.

[xaviermonin]

However, I was not able to reposition the D3D device for bordered windows properly, so I'm currently reinstantiating the device.

Yes, cause SetLayeredWindowAttributes and UpdateLayeredWindow can't be used in same time. Msdn say Note that once SetLayeredWindowAttributes has been called for a layered window, subsequent UpdateLayeredWindow calls will fail until the layering style bit is cleared and set again. Link: https://docs.microsoft.com/en-us/windows/desktop/api/winuser/nf-winuser-setlayeredwindowattributes#remarks

In my tests I dont use SetLayeredWindowAttributes.

[xaviermonin]

I found a lot of problems with UpdateLayeredWindow :

• On initialization, the rendering (BeginPaint, EndPaint) must be done in first, then UpdateLayeredWindow then ShowWindow(Handle, SW_HIDE).
• If ShowWindow(Handle, SW_HIDE) is called before UpdateLayeredWindow, UpdateLayeredWindow fails (error 87: Bad parameters).
• Call UpdateLayeredWindow prevents rendering update (the previous rendering is displayed) but the window move correctly.

[lolp1]

@xaviermonin fixed it.