Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIN32_EXIT_CODE : 1077 (0x435) #1

Open
Pengen opened this issue Nov 12, 2020 · 57 comments
Open

WIN32_EXIT_CODE : 1077 (0x435) #1

Pengen opened this issue Nov 12, 2020 · 57 comments

Comments

@Pengen
Copy link

Pengen commented Nov 12, 2020

Hi @gsuberland ,

I've tried two times to get LBFO working following your blog post step by step. Microsoft Load Balancing/Failover Provider is missing from the Network Service options and both times I end up with this sc query:

SERVICE_NAME: mslbfoprovider
TYPE : 1 KERNEL_DRIVER
STATE : 1 STOPPED
WIN32_EXIT_CODE : 1077 (0x435)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

During extract and install everything looks normal.

Any advise?

LBFO.txt

@gsuberland
Copy link
Owner

Did you try the "Have disk..." option from the network services window?

The error code is "No attempts to start the service have been made since the last boot", and I'm not sure why that would be. You could try starting the service manually with sc start mslbfoprovider.

@Pengen
Copy link
Author

Pengen commented Nov 12, 2020

Thanks for your reply!
No I didn't try the "Have disk..." option, what path should I point it to?
I did try "sc start mslbfoprovider" but that only gave me another exit code "WIN32_EXIT_CODE : 31 (0x1f)"
The machine I'm trying to enable LBFO on is hosting a couple of Hyper-V VMs and after trying install LBFO they all fail and the Hyper-V manager crashes when trying to access the virtual switches settings. Maybe Hyper-V's virtual NICs is the culprit.

@gsuberland
Copy link
Owner

Point the "Have disk..." at the INF/SYS files that you extracted. Those are for the LBFO driver.

Error 31 is a device failure, which is a generic "device failure" error. Doesn't mean much other than "it didn't work", unfortunately.

@Pengen
Copy link
Author

Pengen commented Nov 13, 2020

Ok thank you will try it tomorrow!
Since my VMs failed and the Hyper-V manager went heywire after installation but creating a restore point and restoring from it fixed everything, maybe you could add that to your blog post.

@Teravus
Copy link

Teravus commented Mar 27, 2021

I can also confirm also that Hyper-V Manager and the Hyper V Management Service on a physical machine with this hack is upset with this sometimes. The issue is fairly similar to what happens sometimes if you used a 3rd party VPN solution.

In the past, the test computer had a Cisco VPN Anywhere client that became unsupported in Windows 10.
At some point, Microsoft issued a Windows update that removed the Cisco VPN software if it was installed. Presumably, because it was breaking many people's network stack.

  • You 'may' be unable to modify virtual switch configuration after this with an error message, 'unexpected error: Provider is not capable of the attempted operation'.
  • You 'may' also experience a super long delay restoring a virtual machine from a checkpoint or doing any kind of checkpoint operations on virtual machines.
  • You 'may' also experience a super long delay viewing the virtual machine hardware screen.
  • You 'may' also experience a super long delay connecting to the Virtual machine from the console viewer in Hyper-V Manager.
    • Testing a machine that had this issue.. these delays were between 30 minutes and 80 minutes from the operation initiation to when the operation 'woke up'. These operations, normally, take seconds.

Your chance of experiencing this issue seems to go up if you have Windows VPN profiles Configured (L2TP, maybe others).

@maxim-kiselevich
Copy link

I tried to apply this guide on Windows 10 with UEFI loading. The driver does not appear in the properties of the network card. After sorting out the problem, I found a solution and slightly altered the patch installation file. I tried both patches on a virtual machine, from server 2019 and 2022. Both got up correctly and made it possible to create a network interface. I am attaching my files.
install_lfbo_on_windows_10.zip

@Nodens-
Copy link

Nodens- commented Oct 9, 2021

@gsuberland Since I have to upgrade my workstations to Win11 and Intel has just deprecated Intel ANS, there's currently no other solution, going forward, to do teaming with Intel NICs. I took a look at it and managed to make it work but I know why people may be having problems loading the Network Service Provider. In Win11, it will just refuse to load it and with "Have Disk" it will refuse again with an error code that pertains to unsigned.
I checked @maxim-kiselevich 's package as well and I see that no one of you is pulling the catalog file with the signatures for the .inf file and that's where the problem is.
Using 2022 Server iso (I worked with that directly, didn't check 2019 as 2022 is newer and would possibly have fixes) install.wim, the .inf file's signature is validated with Microsoft-Windows-ServerCore-Drivers-merged-Package~31bf3856ad364e35~amd64~~10.0.20348.1.cat so as long as you copy that one as well, all validations pass. The Provider is verified properly and shows up as signed everywhere. Also it's a good idea to edit the registry file for the service and change start type to 2 as with 3 my Team was failing on reboot. Win11 would not load it on demand for some reason I have no idea if this happens with Win10 builds as well as I never tested on it. Last but no least, if you install the service provider via "Have Disk" instead of placing the files with all signatures and importing the registry keys and then reboot, The provider in the registry will be set to a oemxx.inf inf file and use that instead. I am not sure if this has any side-effects, as these .inf files are assumed to be external, but the values can be changed manually in the registry to point back to the proper instance of the .inf.

Time allowing I will be looking into getting the GUI working properly as well.

PS Since I read about it in the blog, you shouldn't worry about sfc and dism as this is not part of the store. Neither tool will do anything to drivers and files that are not part of the component store.

@cryptz2k
Copy link

Nodens, can you explain how you identified that catalog file? I copied that one as well (i am also using windows 11 and a server 2022 source) In the browse dialogue it indicates the driver is digitally signed correctly, but once i select the file i still get 0xE000022F despite the proceeding screen saying its signed. I noted you chose a server-core package. the rest of my driver files came from the regular server install (non-core) that may not be relevant but since i am having a hard time figuring out how you determined the inf was signed (if i check the inf with sigcheck they all appear unsigned) i am not sure how to validate.

@cryptz2k
Copy link

So a small update, by booting the machine and disabling driver signing enforcement everything worked. maxim-kiselevich's readme states to do this straight away. I guess i am a little confused, all the efforts in copying cat files around, i thought those were efforts to keep driver singing enforement enabled. As mentioned above my MsLbfoProvider.sys file passes a sigcheck. The add service dialog initially says the driver is signed when i select the loadbalancing driver from the list, however here are the logs first with driver signing enforcement, and then later without. The inf file does not seemed to be signed, i take noden's comments to indicate he thought it was.

[SetupCopyOEMInf - c:\windows\system32\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf]
Section start 2021/10/12 08:11:57.943
cmd: C:\Windows\system32\DllHost.exe /Processid:{7007ACD1-3202-11D1-AAD2-00805FC1270E}
inf: Copy style: 0x00000000
sto: {Setup Import Driver Package: c:\windows\system32\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf} 08:11:57.943
inf: Provider: Microsoft
inf: Class GUID: {4D36E974-E325-11CE-BFC1-08002BE10318}
inf: Driver Version: 06/21/2006,10.0.20348.1
sto: {Copy Driver Package: c:\windows\system32\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf} 08:11:57.943
sto: Driver Package = c:\windows\system32\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf
sto: Flags = 0x00000007
sto: Destination = C:\Users\testuser\AppData\Local\Temp{b16e9191-ff79-474e-af63-51bafb8339b6}
sto: Exporting driver package files to 'C:\Users\testuser\AppData\Local\Temp{b16e9191-ff79-474e-af63-51bafb8339b6}'.
flq: {FILE_QUEUE_COMMIT} 08:11:57.959
flq: Copying 'c:\windows\system32\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf' to 'C:\Users\testuser\AppData\Local\Temp{b16e9191-ff79-474e-af63-51bafb8339b6}\mslbfoprovider.inf'.
flq: {FILE_QUEUE_COMMIT - exit(0x00000000)} 08:11:57.959
sto: {Copy Driver Package: exit(0x00000000)} 08:11:57.959
ump: Import flags: 0x00000000
pol: {Driver package policy check} 08:11:57.990
pol: {Driver package policy check - exit(0x00000000)} 08:11:58.006
sto: {Stage Driver Package: C:\Users\testuser\AppData\Local\Temp{b16e9191-ff79-474e-af63-51bafb8339b6}\mslbfoprovider.inf} 08:11:58.006
inf: {Query Configurability: C:\Users\testuser\AppData\Local\Temp{b16e9191-ff79-474e-af63-51bafb8339b6}\mslbfoprovider.inf} 08:11:58.006
inf: Driver package is fully isolated.
inf: Driver package 'mslbfoprovider.inf' is configurable.
inf: {Query Configurability: exit(0x00000000)} 08:11:58.006
flq: {FILE_QUEUE_COMMIT} 08:11:58.006
flq: Copying 'C:\Users\testuser\AppData\Local\Temp{b16e9191-ff79-474e-af63-51bafb8339b6}\mslbfoprovider.inf' to 'C:\Windows\System32\DriverStore\Temp{5ead8ff7-bd1b-e04a-b10d-330f3cbbb5bd}\mslbfoprovider.inf'.
flq: {FILE_QUEUE_COMMIT - exit(0x00000000)} 08:11:58.006
sto: {DRIVERSTORE IMPORT VALIDATE} 08:11:58.006
!!! sig: Driver package does not contain a catalog file, and Code Integrity is enforced.
!!! sig: Driver package failed signature validation. Error = 0xE000022F
sto: {DRIVERSTORE IMPORT VALIDATE: exit(0xe000022f)} 08:11:58.006
!!! sig: Driver package failed signature verification. Error = 0xE000022F
!!! sto: Failed to import driver package into Driver Store. Error = 0xE000022F
sto: {Stage Driver Package: exit(0xe000022f)} 08:11:58.006
sto: {Setup Import Driver Package - exit (0xe000022f)} 08:11:58.006
!!! inf: Failed to import driver package into driver store
!!! inf: Error 0xe000022f: The third-party INF does not contain digital signature information.
<<< Section end 2021/10/12 08:11:58.021
<<< [Exit status: FAILURE(0xe000022f)]

and without driver signing enforcement:

[SetupCopyOEMInf - c:\extracted\fromalpha\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf]
Section start 2021/10/12 08:33:39.029
cmd: C:\Windows\system32\DllHost.exe /Processid:{7007ACD1-3202-11D1-AAD2-00805FC1270E}
inf: Copy style: 0x00000000
sto: {Setup Import Driver Package: c:\extracted\fromalpha\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf} 08:33:39.029
inf: Provider: Microsoft
inf: Class GUID: {4D36E974-E325-11CE-BFC1-08002BE10318}
inf: Driver Version: 06/21/2006,10.0.20348.1
sto: {Copy Driver Package: c:\extracted\fromalpha\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf} 08:33:39.029
sto: Driver Package = c:\extracted\fromalpha\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf
sto: Flags = 0x00000007
sto: Destination = C:\Users\testuser\AppData\Local\Temp{bef3e9ea-6306-694b-be67-4d4adc3bb366}
sto: Copying driver package files to 'C:\Users\testuser\AppData\Local\Temp{bef3e9ea-6306-694b-be67-4d4adc3bb366}'.
flq: {FILE_QUEUE_COMMIT} 08:33:39.029
flq: Copying 'c:\extracted\fromalpha\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf' to 'C:\Users\testuser\AppData\Local\Temp{bef3e9ea-6306-694b-be67-4d4adc3bb366}\mslbfoprovider.inf'.
flq: {FILE_QUEUE_COMMIT - exit(0x00000000)} 08:33:39.044
sto: {Copy Driver Package: exit(0x00000000)} 08:33:39.044
ump: Import flags: 0x00000000
pol: {Driver package policy check} 08:33:39.091
pol: {Driver package policy check - exit(0x00000000)} 08:33:39.091
sto: {Stage Driver Package: C:\Users\testuser\AppData\Local\Temp{bef3e9ea-6306-694b-be67-4d4adc3bb366}\mslbfoprovider.inf} 08:33:39.091
inf: {Query Configurability: C:\Users\testuser\AppData\Local\Temp{bef3e9ea-6306-694b-be67-4d4adc3bb366}\mslbfoprovider.inf} 08:33:39.091
inf: Driver package is fully isolated.
inf: Driver package 'mslbfoprovider.inf' is configurable.
inf: {Query Configurability: exit(0x00000000)} 08:33:39.091
flq: {FILE_QUEUE_COMMIT} 08:33:39.091
flq: Copying 'C:\Users\testuser\AppData\Local\Temp{bef3e9ea-6306-694b-be67-4d4adc3bb366}\mslbfoprovider.inf' to 'C:\Windows\System32\DriverStore\Temp{126ab33f-082a-2e49-a324-042d5da6406c}\mslbfoprovider.inf'.
flq: {FILE_QUEUE_COMMIT - exit(0x00000000)} 08:33:39.091
sto: {DRIVERSTORE IMPORT VALIDATE} 08:33:39.091
! sig: Driver package does not contain a catalog file, but user wants to install anyway.
sto: {DRIVERSTORE IMPORT VALIDATE: exit(0x00000000)} 08:33:51.423
sig: Signer Score = 0x80000000 (Unsigned)
sto: {Core Driver Package Import: mslbfoprovider.inf_amd64_f9d27a6b05ef21aa} 08:33:51.423
sto: {DRIVERSTORE IMPORT BEGIN} 08:33:51.423
bak: Create system restore point:
bak: Description = Device Driver Package Install: Microsoft Network Service
bak: Time = 125ms
bak: Status = 0x00000000 (SUCCESS)
sto: {DRIVERSTORE IMPORT BEGIN: exit(0x00000000)} 08:33:51.563
cpy: {Copy Directory: C:\Windows\System32\DriverStore\Temp{126ab33f-082a-2e49-a324-042d5da6406c}} 08:33:51.563
cpy: Target Path = C:\Windows\System32\DriverStore\FileRepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa
cpy: {Copy Directory: exit(0x00000000)} 08:33:51.563
idb: {Register Driver Package: C:\Windows\System32\DriverStore\FileRepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf} 08:33:51.563
idb: Created driver package object 'mslbfoprovider.inf_amd64_f9d27a6b05ef21aa' in DRIVERS database node.
idb: Created driver INF file object 'oem7.inf' in DRIVERS database node.
idb: Registered driver package 'mslbfoprovider.inf_amd64_f9d27a6b05ef21aa' with 'oem7.inf'.
idb: {Register Driver Package: exit(0x00000000)} 08:33:51.563
idb: {Publish Driver Package: C:\Windows\System32\DriverStore\FileRepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf} 08:33:51.563
idb: Activating driver package 'mslbfoprovider.inf_amd64_f9d27a6b05ef21aa'.
cpy: Published 'mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf' to 'oem7.inf'.
idb: Indexed 2 device IDs for 'mslbfoprovider.inf_amd64_f9d27a6b05ef21aa'.
sto: Flushed driver database node 'DRIVERS'. Time = 15 ms
sto: Flushed driver database node 'SYSTEM'. Time = 0 ms
idb: {Publish Driver Package: exit(0x00000000)} 08:33:51.579
sto: {DRIVERSTORE IMPORT END} 08:33:51.579
dvi: Flushed all driver package files to disk. Time = 0 ms
bak: Commit system restore point:
bak: Description = Device Driver Package Install: Microsoft Network Service
bak: Time = 0ms
bak: Status = 0x00000000 (SUCCESS)
sto: {DRIVERSTORE IMPORT END: exit(0x00000000)} 08:33:51.595
sto: {Core Driver Package Import: exit(0x00000000)} 08:33:51.595
sto: {Stage Driver Package: exit(0x00000000)} 08:33:51.595
! dvi: Unable to find devices that match INF - (00000490)!
sto: {Setup Import Driver Package - exit (0x00000000)} 08:33:51.610
inf: Driver Store Path: C:\Windows\System32\DriverStore\FileRepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf
inf: Published Inf Path: C:\Windows\INF\oem7.inf
<<< Section end 2021/10/12 08:33:51.610
<<< [Exit status: SUCCESS]

@Nodens-
Copy link

Nodens- commented Oct 12, 2021

@cryptz2k My method for finding it was loading the Server 2022 Standard iso in a VMware Workstation VM and using sigcheck. Although I am making a script to identify both required catalog files without a VM.

You can use sigcheck -accepteula -i -f catalog_file mslbfoprovider.inf to verify you have the proper catalog file. The file I listed is from the 4th image in the Server 2022 Standard iso's install.wim. As long as you copy that file correctly and import the registry entries and, most importantly, reboot, the Service provider will be listed on Add Service on it's own, without needing "Have Disk". The reason it's not listed while we have already imported the provider registry keys is simply because the catalog for the .inf is missing. AFTER rebooting, you sigcheck should report that the inf is signed without using -f to point it to the catalog file. At that point you're fine. Add the network provider and start the service (I advise to change the start type before importing the .reg to 2. Then just create the team in powershell.

@cryptz2k
Copy link

My inf file indicates its not signed, is that typical if I don't have the right catalog file specified in the sigcheck command? I would have expected it to be invalid but still signed etc. If I pick the correct one will the file suddenly show as signed? Its working for me now, but I disabled driver integrity checking through the f8 boot process. I don't really mind for my home machine here, but I get the sense you are having luck without that step.

@Teravus
Copy link

Teravus commented Oct 13, 2021

There is a tutorial about packaging (creating a cat.. etc) and signing drivers somewhere on the internet. Back when I tested it, that's what I did.. so driver signing was on for me and it was OK. This was on a physical box.

@Nodens-
Copy link

Nodens- commented Oct 13, 2021

@cryptz2k If you run sigcheck without the -f switch it will try to verify with catalog files in the system environment. "Unsigned" just means it can't verify the signature. So -f switch will show you if you have the proper file. If you have the proper file and it's copied in the proper directory AND you have rebooted (I can't stress how important that this is because without a reboot windows will not "know" about the new catalog file since we're not installing properly, we're making it so it's already installed), sigcheck without -f will show the file as signed as well.
If sigcheck without -f shows inf as unsigned it means 1 or multiple of these 3 things:
a) You have the wrong catalog file (can verify with -f switch)
b) You have not copied it to the proper directory or there's wrong permissions (This should not happen if you use the directory structure from the install.wim AND you use psexec -s to copy it as system as the batch file on this repo does)
c) You have not rebooted after copying.

Disabling Driver signature enforcement is a BIG NO-NO. It means that kernel mode drivers can be loaded with impunity. You're just asking for rootkits, RATs and possibly ransomware.

@cryptz2k
Copy link

Not sure what to make of it then, see below as a test I ran it without the -f flag. it says its signed, but during install it seems to encounter something else that is considers unsigned. I do see the inf has significantly less info but i am assuming yours is the same?

C:\test> sigcheck mslbfoprovider.sys

Sigcheck v2.80 - File version and signature viewer
Copyright (C) 2004-2020 Mark Russinovich
Sysinternals - www.sysinternals.com

C:\test\mslbfoprovider.sys:
Verified: Signed
Signing date: 8:23 PM 8/13/2021
Publisher: Microsoft Windows
Company: Microsoft Corporation
Description: Microsoft Load Balancing/Failover Provider
Product: Microsoft« Windows« Operating System
Prod version: 10.0.20348.1
File version: 10.0.20348.1 (WinBuild.160101.0800)
MachineType: 64-bit

C:\test> sigcheck mslbfoprovider.inf

Sigcheck v2.80 - File version and signature viewer
Copyright (C) 2004-2020 Mark Russinovich
Sysinternals - www.sysinternals.com

C:\test\mslbfoprovider.inf:
Verified: Signed
Signing date: 5:00 AM 9/16/2021
Publisher: Microsoft Windows
Company: n/a
Description: n/a
Product: n/a
Prod version: n/a
File version: n/a
MachineType: n/a

C:\test> sigcheck -i mslbfoprovider.inf
C:\test\mslbfoprovider.inf:
Verified: Signed
Signing date: 5:00 AM 9/16/2021
Signing date: 5:00 AM 9/16/2021
Catalog: C:\Windows\system32\CatRoot{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Microsoft-Windows-ServerCore-Drivers-merged-Package31bf3856ad364e35amd64~~10.0.20348.261.cat
Signers:
Microsoft Windows
Cert Status: Valid

@Nodens-
Copy link

Nodens- commented Oct 13, 2021

Alright this is correct. Start the service, sc start mslbfoprovider. It should start. Then use the powershell commands to make the team.

@Nodens-
Copy link

Nodens- commented Oct 13, 2021

The provider itself will not show up when you do properties on a NIC. Once you build the team it should be automatically added to the member NICs.

@cryptz2k
Copy link

cryptz2k commented Oct 13, 2021

Im fully operational, but only with driver signing enforcement disabled, if i enable it i get the error i posted yesterday: !! sig: Driver package does not contain a catalog file, and Code Integrity is enforced.
!!! sig: Driver package failed signature validation. Error = 0xE000022F

its as if there is a third file in play. the only thing i can think of, and its a stretch, was i initially attempted to install the intel teaming software and thats when i found out it didnt work. Maybe theres something lingering from that but i sort of doubt it. I would have expected all of the intel stuff to be signed anyways

@Nodens-
Copy link

Nodens- commented Oct 14, 2021

You used the Have Disk method and that is your problem. If you look at the registry you'll see it's not using mslbfoprovider.inf but oemxx.inf. That's the problem, that you did it with DSE disabled and Have Disk method. That oemxx.inf file that was created and used in the registry is a copy of mslbfoprovider.inf BUT the inf does not specify a catalog for verification because it's not an oem driver inf but part of the server component store. The inf catalog file you ripped from the install.wim will never verify an oemxx.inf. It will only verify the original mslbfoprovider.inf. I warned about this in my original comment.
So what you need to do is: Delete the provider registry key, reimport the proper one and reboot. Do not try to Have Disk or anything of the sort. Since you have the team created it should just work out of the box. In the miniscule chance that it doesn't just recreate the team. There will be no signature error as long as the registry is pointing to the proper inf.

@Egosumumbravir
Copy link

Egosumumbravir commented Jan 11, 2022

Can confirm this is still working fine on Win10 21h2 using sources from Server 2019.

A few notes - the driver will NOT load with regular boot - Driver Signature Enforcement simply won't allow it.

The workaround for that is to left-shift reboot, startup menu, "7" to disable DSE, then load the driver and click yes on the scary red warning.
Then reboot again and DSE re-enables but the unsigned INF will work fine (this is also useful for hacking Intel drivers on "unsupported" LAN chips).

Will have to try Nodens suggestion up there next time around.

It's also totally possible to port the GUI across and use it to make/config teams.
It's just a major PITA chasing down the files and signatures. Took me an entire afternoon but the process is pretty much the same as digging out the command-line files originally.

@imivz
Copy link

imivz commented Mar 30, 2022

It seems to still work as of the end of march execpt i get an error when trying to create a team There are no teamable NetAdapters on the system matching TeamMembers parameter Does anyone know how to fix this or if im missing some files or anything?

@SergeDubovsky
Copy link

Have anyone managed to run in Win11 without 0xE000022F?

For some reason I am not getting the LBFO in the list of installable services. With the "Have Disk" it throws the 0xE000022F
I had some differences the catalog file name is Package015 not the Package017. The INF is signed in ServerCore-Drivers-merged-Package (I though that was the cause, I was picking up LBFO from wim "1" not "4", but it's still act the same).

Both .INF and .SYS return the sigcheck as "Signed"

@AgentDonut
Copy link

Is this still working for 21H2?

I was able to install and make a team nic but it has no internet

@Pengen
Copy link
Author

Pengen commented Aug 20, 2022

I started this thread a while back and tried just about everything. I’m sure it’s possible but I just went ahead and bought another NIC. Unsupported drivers and hacks will fail sooner or later as updates are rolled out. I do appreciate all the effort but at this point I’m not sure it’s worth the work of all the dedicated people who try/tried to make LBFO work again.

@cryptz2k
Copy link

cryptz2k commented Aug 20, 2022 via email

@cernautan
Copy link

@gsuberland Since I have to upgrade my workstations to Win11 and Intel has just deprecated Intel ANS, there's currently no other solution, going forward, to do teaming with Intel NICs. I took a look at it and managed to make it work but I know why people may be having problems loading the Network Service Provider. In Win11, it will just refuse to load it and with "Have Disk" it will refuse again with an error code that pertains to unsigned. I checked @maxim-kiselevich 's package as well and I see that no one of you is pulling the catalog file with the signatures for the .inf file and that's where the problem is. Using 2022 Server iso (I worked with that directly, didn't check 2019 as 2022 is newer and would possibly have fixes) install.wim, the .inf file's signature is validated with Microsoft-Windows-ServerCore-Drivers-merged-Package~31bf3856ad364e35~amd64~~10.0.20348.1.cat so as long as you copy that one as well, all validations pass. The Provider is verified properly and shows up as signed everywhere. Also it's a good idea to edit the registry file for the service and change start type to 2 as with 3 my Team was failing on reboot. Win11 would not load it on demand for some reason I have no idea if this happens with Win10 builds as well as I never tested on it. Last but no least, if you install the service provider via "Have Disk" instead of placing the files with all signatures and importing the registry keys and then reboot, The provider in the registry will be set to a oemxx.inf inf file and use that instead. I am not sure if this has any side-effects, as these .inf files are assumed to be external, but the values can be changed manually in the registry to point back to the proper instance of the .inf.

Time allowing I will be looking into getting the GUI working properly as well.

PS Since I read about it in the blog, you shouldn't worry about sfc and dism as this is not part of the store. Neither tool will do anything to drivers and files that are not part of the component store.

I am running Windows 10, Version 10.0.19044 Build 19044.

I followed your advice @Nodens- but worked on the files from Windows Server 2019 (not 2022). I extracted the cat files and then in a batch script I checked mslbfoprovider.inf with sigcheck against every catalog and found that the inf checks OK for: Microsoft-Windows-ServerCore-Drivers-net-Package31bf3856ad364e35amd64~~10.0.17763.1.cat.

Next I updated @gsuberland 's extract.bat and install.bat to also include this additional catalog.

When I try to install the Service from Properties of some network interface, I still do not see the driver listed, but, if I use the "Have Disk" option and use the path to the installed inf file (C:\Windows\System32\DriverStore\FileRepository\mslbfoprovider.inf_amd64_9afb7ecb68781bac), then I can see the "Microsoft Load Balancing/Failover Provider" in the list and YES as you mentioned, it sais that "This driver is digitally signed.".

But, when I click on OK, I still get the "Could not add the requested feature. The error is: 0xE000022F".

Do you think this version of the driver might be a bit too old and I should try and get the one from Windows Server 2022 even if I am running Windows 10?

Thanks

@cernautan
Copy link

@Nodens- I tried with drivers from windows server 2022 datacenter evaluation and I can confirm it is the same behavior as I described in my previous reply.

Here is the setup.bat adjusted to include the catalog for the inf as well:

mkdir extracted

set SevenZipPath="C:\Program Files\7-zip\7z.exe"
set InstallWim=install.wim
set ImageIndex=4
set CatalogGUID={F750E6C3-38EE-11D1-85E5-00C04FC295EE}

set CatalogName=Microsoft-Windows-Server-Features-Package015~31bf3856ad364e35~amd64~~10.0.20348.558.cat
set NetCatalogName=Microsoft-Windows-ServerCore-Drivers-merged-Package~31bf3856ad364e35~amd64~~10.0.20348.587.cat

set PathsToExtract=%ImageIndex%\Windows\System32\CatRoot\%CatalogGUID%\%CatalogName%
set PathsToExtract=%PathsToExtract% %ImageIndex%\Windows\System32\CatRoot\%CatalogGUID%\%NetCatalogName%
set PathsToExtract=%PathsToExtract% %ImageIndex%\Windows\System32\drivers\mslbfoprovider.sys
set PathsToExtract=%PathsToExtract% %ImageIndex%\Windows\System32\drivers\en-US\mslbfoprovider.sys.mui
set PathsToExtract=%PathsToExtract% %ImageIndex%\Windows\System32\DriverStore\en-US\MsLbfoProvider.inf_loc
set PathsToExtract=%PathsToExtract% %ImageIndex%\Windows\System32\DriverStore\FileRepository\mslbfoprovider.inf_amd64_*

%SevenZipPath% x -aoa -o.\extracted %InstallWim% %PathsToExtract%

sigcheck -accepteula -i -f .\extracted\%ImageIndex%\Windows\System32\CatRoot\%CatalogGUID%\%CatalogName% .\extracted\%ImageIndex%\Windows\System32\drivers\mslbfoprovider.sys
sigcheck -accepteula -i -f .\extracted\%ImageIndex%\Windows\System32\CatRoot\%CatalogGUID%\%NetCatalogName% .\extracted\%ImageIndex%\Windows\System32\driverstore\filerepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf

@Nodens-
Copy link

Nodens- commented Sep 17, 2022

Ok, so I recently had to install this again and I could not figure out why it worked for me in the first place. After a couple of hours of reverse engineering and brushing up on driver development resources I finally have the proper solution and understand completely why this didn't work for a lot of people (although I still don't understand how it worked for me initially..I must have hit some kind of Windows bug). This is going to be a long post.

First of all. If you have already tried to use this and specially if you tried the "Have disk" method you need to do some cleaning. If someone reading this did not try to do this before, then skip this cleaning part and proceed to the installation part below.
--Cleaning:

  1. Delete the 6 (or 5 if you did not copy the second catalog file) that were installed with install.bat

  2. Go to C:\Windows\INF and use a search in files tool (Notepad++ works great for this) to find all oemx.inf files that contain mslbfoprovider in them. Those are the files that got installed every time you tried the "Have disk" method. Ideally it should only be one, but there could be more depending on what you tried. NOTE down their names and delete them (along with their associated .PNF file if it exists. Eg: oem11.inf and oem11.PNF)

  3. Run psexec -s -i regedit.exe to get into the registry editor as the SYSTEM user. Select HKLM and go to File and select Load Hive. Go to C:\Windows\system32\config select DRIVERS hive file and open it. When asked what to mount it as type DRIVERS.
    Go now under HKLM\DRIVERS\DriverDatabase and search under the 4 sub keys there for any instances of the oemxx.inf files that you noted down and deleted on cleaning step 2. Delete all the keys pertaining to them, close regedit and reboot.

Ok now we're at a clean state and it's time to install. I use the files from the June 2022 update of the Windows Server 2022 Datacenter Standard Edition (not Evaluation). If you're are using a different iso you need to install it in a VM to find the proper files and adjust the below accordingly.

--Installation:

  1. Clone this repo if you have not already and extract install.wim from your iso to the directory you got from cloning the repo.

  2. Edit extract.bat so that it looks like this:

mkdir extracted

set SevenZipPath="C:\Program Files\7-zip\7z.exe"
set InstallWim=install.wim
set ImageIndex=4
set CatalogGUID={F750E6C3-38EE-11D1-85E5-00C04FC295EE}
set CatalogName=Microsoft-Windows-Server-Features-Package015~31bf3856ad364e35~amd64~~10.0.20348.740.cat
set CatalogName2=Microsoft-Windows-ServerCore-Drivers-merged-Package~31bf3856ad364e35~amd64~~10.0.20348.681.cat

set PathsToExtract=%ImageIndex%\Windows\System32\CatRoot\%CatalogGUID%\%CatalogName%
set PathsToExtract=%PathsToExtract% %ImageIndex%\Windows\System32\CatRoot\%CatalogGUID%\%CatalogName2%
set PathsToExtract=%PathsToExtract% %ImageIndex%\Windows\System32\drivers\mslbfoprovider.sys
set PathsToExtract=%PathsToExtract% %ImageIndex%\Windows\System32\drivers\en-US\mslbfoprovider.sys.mui
set PathsToExtract=%PathsToExtract% %ImageIndex%\Windows\System32\DriverStore\en-US\MsLbfoProvider.inf_loc
set PathsToExtract=%PathsToExtract% %ImageIndex%\Windows\System32\DriverStore\FileRepository\mslbfoprovider.inf_amd64_*

%SevenZipPath% x -aoa -o.\extracted %InstallWim% %PathsToExtract%

sigcheck64 -accepteula -i -f .\extracted\%ImageIndex%\Windows\System32\CatRoot\%CatalogGUID%\%CatalogName% .\extracted\%ImageIndex%\Windows\System32\drivers\mslbfoprovider.sys
sigcheck64 -accepteula -i -f .\extracted\%ImageIndex%\Windows\System32\CatRoot\%CatalogGUID%\%CatalogName2% .\extracted\%ImageIndex%\Windows\System32\DriverStore\FileRepository\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa\mslbfoprovider.inf

Note: I use sigcheck64, if you use x86 version replace sigcheck64 with sigcheck.

  1. Edit install.bat so that it looks like this:
set ImageIndex=4

xcopy /H /Y /E .\extracted\%ImageIndex% C:\
reg import mslbfo_service.reg
reg import mslbfo_network_provider.reg
reg import mslbfo_eventlog.reg
reg import Drivers-DeviceIds.reg
reg import Drivers-DriverInfFiles.reg
reg import Drivers-DriverPackages.reg
sigcheck64 -accepteula -i c:\windows\system32\drivers\mslbfoprovider.sys
  1. Create these 3 .reg files:
    Drivers-DeviceIds.reg:
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\DRIVERS\DriverDatabase\DeviceIds\ms_lbfo]
"mslbfoprovider.inf"=hex:01,ff,00,00

Drivers-DriverInfFiles.reg:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\DRIVERS\DriverDatabase\DriverInfFiles\mslbfoprovider.inf]
@=hex(7):6d,00,73,00,6c,00,62,00,66,00,6f,00,70,00,72,00,6f,00,76,00,69,00,64,\
  00,65,00,72,00,2e,00,69,00,6e,00,66,00,5f,00,61,00,6d,00,64,00,36,00,34,00,\
  5f,00,66,00,39,00,64,00,32,00,37,00,61,00,36,00,62,00,30,00,35,00,65,00,66,\
  00,32,00,31,00,61,00,61,00,00,00,00,00
"Active"="mslbfoprovider.inf_amd64_f9d27a6b05ef21aa"

Drivers-DriverPackages.reg:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\DRIVERS\DriverDatabase\DriverPackages\mslbfoprovider.inf_amd64_f9d27a6b05ef21aa]
"Version"=hex:ff,ff,09,00,00,00,00,00,74,e9,36,4d,25,e3,ce,11,bf,c1,08,00,2b,\
  e1,03,18,00,80,8c,a3,c5,94,c6,01,01,00,7c,4f,00,00,0a,00,00,00,00,00,00,00,\
  00,00
"Provider"="Microsoft"
"SignerScore"=dword:0d000003
"FileSize"=hex(b):aa,0c,00,00,00,00,00,00
@="mslbfoprovider.inf"
  1. Edit mslbfo_service.reg and set Start"=dword:00000003 to either 1 (automatic start for kernel mode drivers only) or 2 automatic. Because 3 is manual and on reboot your NIC team will not go up. 1 is better in this case as it gets up sooner.

  2. Now run psexec -s -i regedit.exe. Once the registry editor loads, select HKLM then File and Load hive. Go to C:\Windows\system32\config select DRIVERS hive file and open it. When asked what to mount it as type DRIVERS.
    (We are doing this in order to load the hive that we need for our 3 .reg files to import. A normal Administrator user does not have access to that hive and SYSTEM user does not automatically load the hive. The hive will stay mounted for the 'SYSTEM' user until reboot.)

  3. Exit the registry editor and run psexec -s -i cmd.exe. At the new command prompt running under SYSTEM user, go to the directory with our files and run extract.bat. Once it's done, run install.bat.

  4. Reboot.

  5. Don't try to load the Network Service or anything of the sort. Go to powershell and create your teams directly. The service is now available on the system and new-NetLBFOTeam applet will add it where it needs to add it on its own.

  6. If when the Team NIC goes up, one or more NICs are not properly added to the Team, it means that while you where testing/trying to make this work, these member NICs got corrupted registry entries. In which case you need to delete the Team via powershell, go to Network & Internet Settings, Advanced Network Settings and do Network Reset. Reboot and recreate the Team. (You will have to reconfigure the NICs, obviously.)

That's it.

PS Sorry for being so descriptive but I wanted to leave no room for error.

@cryptz2k
Copy link

cryptz2k commented Sep 28, 2022

@Nodens-

I have updated this comment a few times =). Just wanted to say that after an upgrade to windows 11 22h2 today teaming broke. The nics were in an odd state because they had the Microsoft Multiplexor Driver associated to them but that was not able to be unchecked. I reset networking but the issue remained. I went through the registry and removed the entries for Ethernet and Ethernet 2 connections as well as uninstalled the physical Nics. After a reboot I had 2 normal nics again, both running ipv4/6 etc.

I ran through your additional cleanup steps.

i downloaded the july 2022 server 2022 iso. After a first run through the mslbfoprovider.sys file was stated as being unsigned. the file is from 5/8/2021 and it is version 10.0.20348.1. Truthfully I had provided it with the wrong cat file.

I checked a working/fully patched 2022 server. It has the same .sys file and version etc. Sigcheck told me that file was signed by Microsoft-Windows-Server-Features-Package01531bf3856ad364e35amd64~~10.0.20348.887.cat

I checked index 4 catroot folder in the .wim and that file does not exist (Microsoft-Windows-Server-Features-Package01531bf3856ad364e35amd64~~10.0.20348.887.cat).

I copied that (Microsoft-Windows-Server-Features-Package01531bf3856ad364e35amd64~~10.0.20348.887.cat) off the working system to my machine and everything is working fine.

everything appears to be working now. I went back and learned that if I used Microsoft-Windows-Server-Features-Package01531bf3856ad364e35amd64~~10.0.20348.825.cat that exists on the wim and also works. I am not sure why the installed version references a different file,

@Nodens-
Copy link

Nodens- commented Oct 15, 2022

@cryptz2k Without being able to examine the installation and image, I can only guess. Perhaps the installation is not using the 4th image from .wim? Or perhaps updates (either applied after installation or those pulled automatically during installation) updated the catalog file.

As a side note, I had to install a few more workstations this way with 22H2 (fresh installations) and some of them required doing the Have Disk method (without trying to add the service to a NIC--just do have disk once to trigger the .inf install section) AFTER following my instructions above (again the service will not show up, it'll just take in with no output but that's normal as the driver is already "preinstalled" with my method), or the service would not start.
This leads me to believe that the Install section of the inf is not being triggered properly on all systems by my original method and while just doing Have Disk afterwards fixes it, I will investigate when I find some time to see how to trigger this directly (probably a call in setupapi using rundll32.exe) so I can make a proper install script for the entire thing that automatically locates the proper catalog files as well regardless of the image used as source. It is getting a bit tiresome do it manually when I have to set up or reinstall workstations.

EDIT: And I forgot, yes, upgrading windows build will break the network because windows basically installs an entirely new image and migrates user installed drivers, applications and data. Since this driver is not installed as a user installed driver (oemxx.inf) but a system one that is not part of the component store it is impossible to be kept during build upgrades. We can't install it as user driver because we can't modify the .inf to include proper catalog deifinition (it would break signature) and while we can add it to the component store, windows installs an entirely new image resetting the component store to the new system image, so there's really nothing that can be done about it and you will always require reinstallation. Breaking the team before a windows build upgrade will save you the reset of the NICs though.

There is only one thing that can be done that is only doable for very few people. If you have an EV code signing certificate you can edit the .inf and resign. That way you can make it a proper user installed driver which will stick during build upgrades. And while I do have such a certificate and will eventually go that route, I can not obviously distribute the files for legal reasons. (I will see if I can make a completely new .inf from scratch though -- that should allow me to distribute just that, signed by me. Perhaps along with a Qt GUI app to control teams).
Notice you need an EV code signing certificate. Not any code signing certificate will do as an EV is the only one that allows you to sign kernel mode drivers for x64 since Win8.

@Egosumumbravir
Copy link

@Nodens- I don't think you need to do this every time you install a system.
In my experience doing this hack with Windows 10 Workstation, the driver/service AND the GUI components will survive a sysprep just fine. A major system update will break it, but a sysprep not so much.

@Nodens-
Copy link

Nodens- commented Oct 20, 2022

@Egosumumbravir I know it survives sysprep but I'm not re-imaging these systems. They are relocated in the building and I just install it to hook them to the switches.
Also what do you mean the GUI components? There are no GUI components available in client OS..

@Egosumumbravir
Copy link

Egosumumbravir commented Oct 20, 2022

Ok, fair enough. I've stopped bothering to build a separate non-LBFO version anymore so any machine I've installed in the last year or so has it by default now. Very handy at times.

Strictly, there's no LBFO support in the client OS either until you put it there. Exporting the Server 2019 GUI components works fine ... and they'll survive a sysprep too!

EDIT: ohh nice. Have just applied the 22h2 update to my Win10 WKS testbed and LBFO is still functioning fine. I was expecting the major update to break it (could have sworn 21h2 did?). That's going to simplify rolling the new image :)

@Nodens-
Copy link

Nodens- commented Oct 20, 2022

It is VERY strange that 22H2 didn't break it for you. Are you running with DSE off? Or you somehow managed to installed it as user installed driver without editing the .inf?
Save me some time in researching, what do you pull out of the Server image for the .msc to work? I always assumed it had COM object dependencies, so was planning on writing my own GUI instead of spending time figuring them out (I literally loathe the COM model heh).

@Egosumumbravir
Copy link

Yes, I was very surprised that it worked. So I updated another machine - no reset there either. I do recall 21h2 breaking it, but that Microsoft changed the update delivery system around this time.

DSE: no, I load the driver after a left-shift reboot - which allows for temporary DSE suspension and easy restoration by simply, rebooting afterwards. I've been using this tweak to load modified Intel network drivers since Server 2012.

The dev machine is offline ATM, but I'll be getting it out for a new image rolling shortly. Will get you a file list then.

@SaturnusDJ
Copy link

@gsuberland
Thanks for this great idea and the instructions.

@Nodens-
Thanks for the excellent update. #1 (comment)

Here the 20Gbit/s is not reached with 2x 10 in the team, however there is a slight improvement over single link. 1.1GB/s versus 1.35GB/s. This setup is HPE 561FLR-T in two computers, no switch in between. TeamingMode static, LoadBalancingAlgorithm dynamic. More important the team adapter settings. All offloads enabled, VM enabled even when not using it. Lower speeds here with other settings. More tuning to come. Will update this post only if performance goes up.

@Egosumumbravir
Copy link

@Nodens-

Save me some time in researching, what do you pull out of the Server image for the .msc to work?

Sorry, life has been one spot fire after another for too many months. Directory listing of my GUI included archive

  • %WINDIR%\Microsoft.NET\assembly\GAC_MSIL\Microsoft.Management.Infrastructure\v4.0_1.0.0.0__31bf3856ad364e35\Microsoft.Management.Infrastructure.dll
  • %WINDIR%\Microsoft.NET\assembly\GAC_MSIL\Microsoft.Management.Infrastructure.CimCmdlets\v4.0_1.0.0.0__31bf3856ad364e35\Microsoft.Management.Infrastructure.CimCmdlets.dll
  • %WINDIR%\Microsoft.NET\assembly\GAC_MSIL\Microsoft.Management.Infrastructure.CimCmdlets.Resources\v4.0_1.0.0.0_en_31bf3856ad364e35\Microsoft.Management.Infrastructure.CimCmdlets.Resources.dll
  • %WINDIR%\Microsoft.NET\assembly\GAC_MSIL\Microsoft.Management.Infrastructure.Resources\v4.0_1.0.0.0_en_31bf3856ad364e35\Microsoft.Management.Infrastructure.Resources.dll
  • %WINDIR%\Microsoft.NET\assembly\GAC_MSIL\Microsoft.Management.UI\v4.0_10.0.0.0__31bf3856ad364e35\Microsoft.Management.UI.dll
  • %WINDIR%\Microsoft.NET\assembly\GAC_MSIL\Microsoft.Management.UI.resources\v4.0_10.0.0.0_en_31bf3856ad364e35\Microsoft.Management.UI.resources.dll
  • %WINDIR%\System32\LbfoAdmin.exe
  • %WINDIR%\System32\LbfoAdmin.exe.mui
  • %WINDIR%\System32\LbfoAdminLib.dll
  • %WINDIR%\System32\MuxInst.dll
  • %WINDIR%\System32\CatRoot{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Microsoft-Windows-PowerShell-ServerCore-WOW64-Package31bf3856ad364e35amd64en-US10.0.17763.1.cat
  • %WINDIR%\System32\CatRoot{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Microsoft-Windows-PowerShell-ServerCore-WOW64-Package31bf3856ad364e35amd64~~10.0.17763.1.cat
  • %WINDIR%\System32\CatRoot{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Microsoft-Windows-Server-Features-Package01731bf3856ad364e35amd64~~10.0.17763.1.cat
  • %WINDIR%\System32\CatRoot{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Microsoft-Windows-Server-Gui-Mgmt-Package-net31bf3856ad364e35amd64en-US10.0.17763.1.cat
  • %WINDIR%\System32\CatRoot{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Microsoft-Windows-Server-Gui-Mgmt-Package-net31bf3856ad364e35amd64~~10.0.17763.1.cat
  • %WINDIR%\System32\CatRoot{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Microsoft-Windows-ServerManager-UX-ARW-Plugins-Package31bf3856ad364e35amd64en-US10.0.17763.1.cat
  • %WINDIR%\System32\CatRoot{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Microsoft-Windows-ServerManager-UX-ARW-Plugins-Package31bf3856ad364e35amd64~~10.0.17763.1.cat
  • %WINDIR%\System32\drivers\MsLbfoProvider.sys
  • %WINDIR%\System32\drivers\en-US\MsLbfoProvider.sys.mui
  • %WINDIR%\System32\DriverStore\en-US\MsLbfoProvider.inf_loc
  • %WINDIR%\System32\DriverStore\FileRepository\mslbfoprovider.inf_amd64_9afb7ecb68781bac\mslbfoprovider.inf
  • %WINDIR%\System32\en\LbfoAdminLib.resources.dll
  • %WINDIR%\System32\en-US\LbfoAdmin.exe.mui
  • %WINDIR%\System32\en-US\MuxInst.dll.mui
  • %WINDIR%\WinSxS\amd64_microsoft-managemen..mentation.resources_31bf3856ad364e35_10.0.17763.1_en-us_07f8f728ddd7acb8\MuxInst.dll.mui
  • %WINDIR%\WinSxS\amd64_microsoft-management-ui-instrumentation_31bf3856ad364e35_10.0.17763.1_none_23327f1522b9d151\MuxInst.dll
  • %WINDIR%\WinSxS\amd64_microsoft-windows-n..bfo-admin.resources_31bf3856ad364e35_10.0.17763.1_en-us_f01250abd376c5a1\LbfoAdmin.exe.mui
  • %WINDIR%\WinSxS\amd64_microsoft-windows-n..bfo-admin.resources_31bf3856ad364e35_10.0.17763.1_en-us_f01250abd376c5a1\LbfoAdminLib.resources.dll
  • %WINDIR%\WinSxS\amd64_microsoft-windows-ndis-lbfo-admin_31bf3856ad364e35_10.0.17763.1_none_e14352357d05df34\LbfoAdmin.exe
  • %WINDIR%\WinSxS\amd64_microsoft-windows-ndis-lbfo-admin_31bf3856ad364e35_10.0.17763.1_none_e14352357d05df34\LbfoAdminLib.dll
  • %WINDIR%\WinSxS\amd64_microsoft-windows-ndis-lbfo_31bf3856ad364e35_10.0.17763.1_none_0229ea8a6efa41f4\MsLbfoProvider.sys

Sadly for my Win10 hacks here, it's time to roll a Win11 image for general deployment.
I'm sitting here this morning debating about ripping the files out of Server 2022 as above, or investigating more deeply the New-NetSwitchTeam command which is still functional in my new 22h2 Win11 test VM.

@Egosumumbravir
Copy link

or investigating more deeply the New-NetSwitchTeam command which is still functional in my new 22h2 Win11 test VM.

Hmmm, New-NetSwitchTeam doesn't seem to work at all for load distribution. Three 1gbps lines plugged into the same switch, which has one of it's 10gbps ports to the server. NVMe to NVMe = 112MBps windows copies. Yeah nah. File ripping here I come.

@Nodens-
Copy link

Nodens- commented May 3, 2023

@SaturnusDJ There is currently a kernel issue in Windows Client, unfixed for months (see here: https://techcommunity.microsoft.com/t5/storage-at-microsoft/slower-smb-read-performance-for-large-files-in-22h2/ba-p/3643332/emcs_t/S2h8ZW1haWx8dG9waWNfc3Vic2NyaXB0aW9ufExGTlIzMTY4WTA0VldPfDM3Nzg1NTZ8U1VCU0NSSVBUSU9OU3xoSw#comments). It is listed in known issues on every KB update note but downplayed as to its severity (it also affects local transfers). RSS seems to not be working at all and there's some inherent problem with the buffering on the kernel level. That said, I'd suggest using LACP instead of static if your switch supports it as static if prone to a lot of issues like flapping and switching loops.

@Egosumumbravir thanks a lot :) Regarding New-NetSwitchTeam, it is unfortunately designed to work with Hyper-V switches and you'd only see any load-balancing benefits if you're running Hyper-V VMs (think VMQ).
We're basically stuck with this solution, as long as we can make it work, until someone comes up with something else. But it seems that everyone is pushing switch independent teaming for DC/VM usage alone. We're the edge cases running home labs with teaming on clients, that no one is interested in. I'm tempted to write my own open source, provider independent implementation as I'm not sure how long we'll be able to pull this off and the only alternative is using linux on my workstations as well, with WINE and VMware Workstation (which may not be such a bad idea in the long run).

@SaturnusDJ
Copy link

SaturnusDJ commented May 5, 2023

@Nodens-
Thanks for the information. That's worth keeping an eye on.

Since my last post I peaked at ~1.7GB/s (~13.6Gbit/s). Changing adapter settings gave a significant improvement, however still far from what it should be. I have direct connection between two computers. No switch involved, their price is too high.

Here are the adapter settings (for HPE 561FLR-T, in my case):

Property: Value:
Enable PME Disabled
Encapsulation Overhead 0
Flow Control Rx & Tx Enabled
Interrupt Moderation Disabled
Interrupt Moderation Rate Adaptive
IPv4 Checksum Offload Rx & Tx Enabled
Jumbo Packet 9014 Bytes
Large Send Offload V2 (IPv4) Disabled
Large Send Offload V2 (IPv6) Disabled
Locally Administered Address Not Present
Log Link State Event Disabled
Maximum Number of RSS Queues 8 Queues
Packet Priority & VLAN Packet Priority & VLAN Disabled
Receive Buffers 2048
Receive Side Scaling Disabled
Speed & Duplex 10Gbps Full Duplex
TCP Checksum Offload (IPv4) Rx & Tx Enabled
TCP Checksum Offload (IPv6) Rx & Tx Enabled
Transmit Buffers 2048
UDP Checksum Offload (IPv4) Rx & Tx Enabled
UDP Checksum Offload (IPv6) Rx & Tx Enabled
Wake on Link Settings Disabled
Wake on Magic Packet Disabled
Wake on Pattern Match Disabled

For Microsoft Network Adapter Multiplexor Driver:

Property: Value:
Encapsulated Task Offload Enabled
Header Data Split Enabled
IPsec Offload Disabled
IPv4 Checksum Offload Rx & Tx Enabled
Large Send Offload Version 2 (IPv4) Disabled
Large Send Offload Version 2 (IPv6) Disabled
MAC Address Not Present
Receive Side Scaling Disabled
Recv Segment Coalescing (IPv4) Enabled
Recv Segment Coalescing (IPv6) Enabled
TCP Checksum Offload (IPv4) Rx & Tx Enabled
TCP Checksum Offload (IPv6) Rx & Tx Enabled
UDP Checksum Offload (IPv4) Rx & Tx Enabled
UDP Checksum Offload (IPv6) Rx & Tx Enabled
Virtual Machine Queues Disabled
Virtual Machine Queues - Shared Memory Disabled
Virtual Machine Queues - VLAN Id Disabled

Note that some of these properties of course are configuration depended, and those should have no impact on speed. Also, quite some of these settings go against the advise found online.

@SaturnusDJ
Copy link

The above was for Windows to Windows.

Last weekend I looked at Windows <-> Linux.

Property: Value:
Receive Side Scaling Enabled

Linux MTU of adapters and bond: 9710

This last one makes a huge difference. Speed from unstable 400-900MB/s to 1.5GB/s.
Other ethtool settings did not seem to make a difference. Will post again when finding significant improvements.

@Hashbrown777
Copy link

@Nodens- thanks for adding your update! I finally managed to get the driver installed and although I did need to use 'Have Disk' it did say it was signed. I rewrote the extract as a powershell script that auto-mounts the iso and pulls out the files. This _NetLbfoTeam.ps1 installer also doesn't need to be ran from an admin terminal as it self elevates, and it handles mounting the DRIVERS hive for you as well. The only out-of-console thing you need to do that I couldn't figure out a commandline for is performing the 'Have Disk' part (I even open the Network Connections window automatically).

It was all for naught though.. because I was slapped with this same issue, and I cannot get past it.

Although it has less options, New-NetSwitchTeam is supposed to do what I was going to use New-NetLbfoTeam for but yeah, all traffic just went through a single NIC (actually, NO traffic went through, until I explicitly unticked Windows Loadbalancing/Failover Service in its properties!). So I think there's a new-ish bug with it.

Apparently this is a common thing to do within linux; so I'm wondering whether we can leverage WSL2 and somehow pass it control over the NICs and then use its IP as the gateway in windows.

@Nodens-
Copy link

Nodens- commented Jul 18, 2023

@Hashbrown777 The issue you've hit is not using the command with proper parameters.
new-NetLBFOTeam -Name Name -TeamNicName NicName -TeamMembers "Adapter 1","Adapter 2" -TeamingMode LACP -LoadBalancingAlgorithm Dynamic -LacpTimer Fast
Notice that Team Member names must be in double quotes.

New-NetSwitchTeam is not bugged, it just does not do what you think it does. You will not see any loadbalancing unless you're running VMs which is its intended use case. It is not a replacement for LBFO.

@Hashbrown777
Copy link

@Nodens- it's just what he wrote in the article, and he provides screenshots sans vms *shrug

@Nodens-
Copy link

Nodens- commented Jul 19, 2023

@Hashbrown777

From the url you linked:

The network switch team is a team that is controlled by a Hyper-V extensible switch forwarding extension. No other cmdlets can be used to manage a switch team and the Network Switch Team Cmdlets in Windows PowerShell must not be used to manage standard, or non-switch, network adapter teams.

It is for teaming up NICs with virtual switches for use with Hyper-V. It is not NIC Teaming.
Read this: https://learn.microsoft.com/en-us/windows-hardware/drivers/network/overview-of-the-hyper-v-extensible-switch#types-of-hyper-v-extensible-switches-and-network-adapters
And this: https://www.veeam.com/blog/hyperv-set-management-using-powershell.html

But like I said, unless you're using Hyper-V with VMs, you won't have any use for this.

@Hashbrown777
Copy link

Hashbrown777 commented Jul 19, 2023

"use"!="manage"

I took that to mean "you must use *-NetSwitchTeam cmdlets, dont attempt any others to administer the virtual nic". If what you're trying to say is true idk how the guy in the also-linked-there Medium article got his results, then.

Also, the whole "proper parameters" with emphasis on double quotes came; I'm intimately familiar with pwsh and its arg passing, the error message can come up for a number of reasons (such as a valid but inappropriate nic). In this case my params were fine; it worked identically with SwitchTeam. The patch over getting LBFOTeam to work seems broken.

I'll try again with the timer param on lbfoteam, but Im thinking of trying to do it natively in linux, esp if switchteam is just meant to work and work on a family of hyperv wsl's anyway, and have windows push all traffic through a master.

@Nodens-
Copy link

Nodens- commented Jul 19, 2023

I did not mean to imply that you're not familiar with powershell and I apologize if it came out as such. It is just that New-NetLBFOTeam example syntax in documentation is not accurate. And by that I mean that the examples on https://learn.microsoft.com/en-us/powershell/module/netlbfo/new-netlbfoteam?view=windowsserver2022-ps fail on my side as well. I can not remember which of the parameters is required but not present in the examples (maybe it was using both -Name and -TeamNicName ..it's been a while) but I have noted down the exact working example syntax on my end for this specific reason and it's that that I pasted in my reply. The double quotes remark was me being very descriptive, as usual, as more people may find this information useful.

Apart from the syntax, the only other case I've seen it fail like that was when a windows build upgrade broke the team down and the NICs were stuck in a state of being team members without a team NIC present (so Remove-NetLbfoTeam could not be used to fix it). The fast fix for this case was resetting the network stack.

@Hashbrown777
Copy link

Yeah I used tab auto complete to pull the list of accepted switches and I think I tried every combo of name and teamnicname :(

I was just really glad to finally get it installed after attempting those outdated instructions for so long, then when it didn't work I found people running it legit on real server editions getting hit with the same issues :/

It'd be nice if the error was more descriptive, like which parameters are wrong (is it a bad combo of algorithm and teaming mode?), is it that the named adapter wasn't found (mispelled or not quote-encapsulated), or is it an adapter it doesn't work with (vm bridge or something or maybe wifi cards aren't supported). Im also wondering whether it somehow uses a different list of adapter names to the ones we see, mine may have been renamed and the cmdlet is still somehow referencing old values.

@kingofotaku
Copy link

kingofotaku commented Jul 20, 2023

Yeah I used tab auto complete to pull the list of accepted switches and I think I tried every combo of name and teamnicname :(

I was just really glad to finally get it installed after attempting those outdated instructions for so long, then when it didn't work I found people running it legit on real server editions getting hit with the same issues :/

It'd be nice if the error was more descriptive, like which parameters are wrong (is it a bad combo of algorithm and teaming mode?), is it that the named adapter wasn't found (mispelled or not quote-encapsulated), or is it an adapter it doesn't work with (vm bridge or something or maybe wifi cards aren't supported). Im also wondering whether it somehow uses a different list of adapter names to the ones we see, mine may have been renamed and the cmdlet is still somehow referencing old values.

I tried to extract files from many versions of the image using the method here, but I always got only three files. Why is that?

@Hashbrown777
Copy link

Hashbrown777 commented Jul 20, 2023

I tried to extract files from many versions of the image using the method here, but I always got only three files. Why is that?

You may need to go into your images and see. If you're referencing my script it pulled this directory for me (with the correct hardcoded strings..):

image
Are you able to manually locate them in your .win? Which three are popping out for you?

@kingofotaku
Copy link

I tried to extract files from many versions of the image using the method here, but I always got only three files. Why is that?

You may need to go into your images and see. If you're referencing my script it pulled this directory for me (with the correct hardcoded strings..):

image Are you able to manually locate them in your .win? Which three are popping out for you?

I even tried options other than Volume 4, but always got the following results:
E:.
├─1
│ └─Windows
│ └─System32
│ ├─drivers
│ │ │ MsLbfoProvider.sys
│ │ │
│ │ └─en-US
│ │ MsLbfoProvider.sys.mui
│ │
│ └─DriverStore
│ └─FileRepository
│ └─mslbfoprovider.inf_amd64_f9d27a6b05ef21aa
│ mslbfoprovider.inf

├─2
│ └─Windows
│ └─System32
│ ├─drivers
│ │ │ MsLbfoProvider.sys
│ │ │
│ │ └─en-US
│ │ MsLbfoProvider.sys.mui
│ │
│ └─DriverStore
│ └─FileRepository
│ └─mslbfoprovider.inf_amd64_f9d27a6b05ef21aa
│ mslbfoprovider.inf

├─3
│ └─Windows
│ └─System32
│ ├─drivers
│ │ │ MsLbfoProvider.sys
│ │ │
│ │ └─en-US
│ │ MsLbfoProvider.sys.mui
│ │
│ └─DriverStore
│ └─FileRepository
│ └─mslbfoprovider.inf_amd64_f9d27a6b05ef21aa
│ mslbfoprovider.inf

└─4
└─Windows
└─System32
├─drivers
│ │ MsLbfoProvider.sys
│ │
│ └─en-US
│ MsLbfoProvider.sys.mui

└─DriverStore
└─FileRepository
└─mslbfoprovider.inf_amd64_f9d27a6b05ef21aa
mslbfoprovider.inf

I was never able to get the .cat files

@kingofotaku
Copy link

I tried to extract files from many versions of the image using the method here, but I always got only three files. Why is that?

You may need to go into your images and see. If you're referencing my script it pulled this directory for me (with the correct hardcoded strings..):

image Are you able to manually locate them in your .win? Which three are popping out for you?

As you say, I've also tried manually navigating to the directory where the .cat files is located, with no luck.

@Hashbrown777
Copy link

@kingofotaku, if you're using an old version of Windows Server you'll be looking for a single Microsoft-Windows-Server-Features-Package017~31bf3856ad364e35~amd64~~10.0.17763.1.cat not 015&Merged. But it is strange you cannot locate the .inf_loc either..

@kingofotaku
Copy link

kingofotaku commented Jul 20, 2023

@kingofotaku, if you're using an old version of Windows Server you'll be looking for a single Microsoft-Windows-Server-Features-Package017~31bf3856ad364e35~amd64~~10.0.17763.1.cat not 015&Merged. But it is strange you cannot locate the .inf_loc either..

Can you tell me what version you are using? I just need a file name. Many thanks!

@Dahita
Copy link

Dahita commented Aug 28, 2023

I have successfully installed LFBO on the latest Win 11 pro 22H2 Build 22621.2215 as of 8/27/2023.

I wasn't successful using the original method though. I ended up being stuck with the exact same problem as @Pengen at the origin of this thread (which is how I landed here). Installing the driver with "have disk" did not fix the mslbfoprovider stop code issue. I couldn't start the service manually either.

Reading this thread, I saw @maxim-kiselevich 's post and files, gave it a shot (using Windows safe mode with disabled signature check), bingo. No dice with his 2022 .bat file, but it works with the 2019 one.

Thank you to @gsuberland , you are a true hero. Thank you to @maxim-kiselevich for sharing his files, and thanks to all of you for the feedback which helped a lot.

Edit: mslbfoprovider was not starting at boot, needed to manually start it. Figured out how to change that in the registry (
Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\mslbfoprovider and change the Start Dword key value to 0. @Nodens- actually mentioned it above too, copy/pasting his comment below) :

"Edit mslbfo_service.reg and set Start"=dword:00000003 to either 1 (automatic start for kernel mode drivers only) or 2 automatic. Because 3 is manual and on reboot your NIC team will not go up. 1 is better in this case as it gets up sooner."

I changed it to zero instead since it was defined as:
0 - Boot: Loaded by kernel loader. Components of the driver stack for the boot (startup) volume must be loaded by the kernel loader.

I don't know whether 0, 1 or 2 is best, but 0 works like a charm. I've edited @maxim-kiselevich 's mslbfo_service.reg file, hopefully to bulletproof future reinstalls.

Thanks again, played with transfers all night, I've never had such steady maxed out speeds!

@kingofotaku
Copy link

I have successfully installed LFBO on the latest Win 11 pro 22H2 Build 22621.2215 as of 8/27/2023.

I wasn't successful using the original method though. I ended up being stuck with the exact same problem as @Pengen at the origin of this thread (which is how I landed here). Installing the driver with "have disk" did not fix the mslbfoprovider stop code issue. I couldn't start the service manually either.

Reading this thread, I saw @maxim-kiselevich 's post and files, gave it a shot (using Windows safe mode with disabled signature check), bingo. No dice with his 2022 .bat file, but it works with the 2019 one.

Thank you to @gsuberland , you are a true hero. Thank you to @maxim-kiselevich for sharing his files, and thanks to all of you for the feedback which helped a lot.

Edit: mslbfoprovider was not starting at boot, needed to manually start it. Figured out how to change that in the registry ( Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\mslbfoprovider and change the Start Dword key value to 0. @Nodens- actually mentioned it above too, copy/pasting his comment below) :

"Edit mslbfo_service.reg and set Start"=dword:00000003 to either 1 (automatic start for kernel mode drivers only) or 2 automatic. Because 3 is manual and on reboot your NIC team will not go up. 1 is better in this case as it gets up sooner."

I changed it to zero instead since it was defined as: 0 - Boot: Loaded by kernel loader. Components of the driver stack for the boot (startup) volume must be loaded by the kernel loader.

I don't know whether 0, 1 or 2 is best, but 0 works like a charm. I've edited @maxim-kiselevich 's mslbfo_service.reg file, hopefully to bulletproof future reinstalls.

Thanks again, played with transfers all night, I've never had such steady maxed out speeds!

May I ask which version of Windows Server you have used?

@Dahita
Copy link

Dahita commented Sep 2, 2023

Sure, in @maxim-kiselevich 's files on the post above he included two versions:
Win_srv_2019_v1809_build_17763.1852
Win_srv_2022_v21H2_build_20308.1

Only the first one worked for me. Link to his post here.

@kingofotaku
Copy link

kingofotaku commented Sep 3, 2023

Sure, in @maxim-kiselevich 's files on the post above he included two versions: Win_srv_2019_v1809_build_17763.1852 Win_srv_2022_v21H2_build_20308.1

Only the first one worked for me. Link to his post here.

thanks a lot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests