Mesh Agent running and connected, but no desktop or file session possible

[chrkli]

Describe the bug

Running MeshCentral on ~150 domain-joined clients, mostly Windows 10 (different build versions, see below) as
well as three or four Windows 11 Clients. The device groups are configured to show only connected devices. The
agent is connected, however it is not possible to access either the functionality "Desktop" or "Files".
Additionally, the list view of all connected nodes does not show anything in the column "username"
However, if the command "users" is issued in the agent console of an client "non-controllable", it will print the logged on users correctly. In the agent console, I can issue any command successfully, also osinfo can be accessed.

To Reproduce
I'm monitoring several clients showing this strange behavior but up to now, I was not able to find a
reproduction or at least any related client-specific config.

Expected behavior
While the agent is shown inside the device groupt it shall be possible to directly connect the desktop as well as files.

Current workaround
In agent console, first use action "delete core" immediately followed by "Upload default server core"
The tabs for "Desktop" as well as "Files" appear almost instantly. Alternatively, trigger agent update
via agent consoles "run command" feature.

Server Software:

• OS: Windows Server 2022 21H2 (Build 20348.1487)
• Virtualization: ESXi hosted virtual maschine
• Network: Hybrid LAN+WAN
• Version: 1.1.2
• Node: 14.17.1

Client Device:

• Device: Laptop
• OS: Windows 10 Enterprise 22H2 (19045.2486)
• Network: Local to Meshcentral
• Browser: Google Chrome, MS Edge, Firefox

Remote Device:

• Device: Laptop as well as PC
• OS: Windows 10 Enterprise, Windows 10 Enterprise LTSC (see below)
• Network: Local to Meshcentral, BranchOffice via VPN-Tunnel,
• Current Core Version : Dec 9 2022, 3593025475

Additional context

• AV-Exceptions for server as well as agent paths and processes have been added to all endpoints.
• For codesigning as well as for the website the certificates used are issues by our in-house PKI.
• The certificate for the agent server uses the default self-signed created on first startup.
• Alreade fully removed the agent on two clients showing this behavior and reinstalled a fresh one of another device group, however the issue persists.
• When using the workaround described above, sometimes the affected client remains controllable several hours, sometimes it will only last for a few minutes (observed timespan: 3 to 36 minutes) in "controllable" mode
• It does not seem to be related to a specifix OS-Version as I've observed the issue on following builds:
  Windows 10 Enterprise LTSC 21H2/19044
  Windows 10 Enterprise LTSC 17763
  Windows 10 Enterprise 22H2/19045
  Windows 10 Enterprise 20H2/19042
  Windows 10 Enterprise 18363
• Inside the agenterrorlogs.txt collected on the server there are several entries like those ones:

node//randomString, 1674746652896, {"action":"errorlog","log":[{"t":1674707034,"m":"Info: No certificate was found in db","h":"F9D7CD20898987F1","f":"..\\microstack\\ILibParsers.c","l":"10978"}]}
node//randomString, 1674722928724, {"action":"errorlog","log":[{"t":1674669706,"m":"FATAL EXCEPTION @ [FuncAddr: 0x00007ff730799f2f / BaseAddr: 0x00007ff7307d0160 / Delta: 221745]","h":"F9D7CD20898987F1","f":"..\\microstack\\ILibParsers.c","l":"2776"}]}
node//randomString, 1675146640510, {"action":"errorlog","log":[{"t":1675095132,"m":"NCryptFinalizeKey() failed","h":"F9D7CD20898987F1","f":"..\\microstack\\ILibParsers.c","l":"10978"},{"t":1675095132,"m":"Error opening Microsoft Certificate Store","h":"F9D7CD20898987F1","f":"..\\microstack\\ILibParsers.c","l":"10978"},{"t":1675095134,"m":"NCryptFinalizeKey() failed","h":"F9D7CD20898987F1","f":"..\\microstack\\ILibParsers.c","l":"10978"},{"t":1675150224,"m":"Info: No certificate was found in db","h":"F9D7CD20898987F1","f":"..\\microstack\\ILibParsers.c","l":"10978"}]}

Screenshots

config.json file

{
   "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json", 
   "settings": {
      "cert": "host.redacted.fqdn", 
      "minify": true, 
      "compression": true, 
      "dbencryptkey": "28-characters-alphanumerical+special-characters",
	  "sessionSameSite": "strict",
	  "StrictTransportSecurity": false,
	  "browserPong": 299,
	  "agentPong": 179,
      "agentlogdump": true, 
      "agentcoredump": true, 
	  "agentsignlock": true,
      "webpagelenghtrandomization": true, 
      "cleannpmcacheonupdate": true, 
      "allowHighQualityDesktop": true, 
	  "desktopMultiplex": true,
	  "trustedProxy": "IPv4_of_currently_unused_reverseproxy",
      "autobackup": false, 
      "authlog": "C:\\Program Files\\MeshCentral\\meshcentral-data\\auth.log"
   }, 
   "domains": {
      "": {
         "title": "Remote Support", 
         "title2": "IT Abteilung", 
		 "welcomePicture": "Logo_C.png",
		 "loginfooter": "<a href=\"tel:+491234567-890\">+491234567-890</a> | <a href=\"mailto:meshk@redacted.fqdn\">mesh@redacted.fqdn</a>  | company - street - zipcode city",
         "footer": "<a href=\"tel:+491234567-890\">+491234567-890</a> | <a href=\"mailto:meshk@redacted.fqdn\">mesh@redacted.fqdn</a>  | company - street - zipcode city",,
         "agentcustomization": {
            "displayName": "company Support Agent", 
            "description": "Ermöglicht Fernwartung durch die IT-Abteilung der Firma.", 
            "companyName": "Company Name", 
            "serviceName": "SupportAgentCompany", 
            "fileName": "Company-SupportAgent"
         }, 
         "agentfileinfo": {
            "icon": "MCagent.ico", 
            "filedescription": "Remote Support powered by MeshCentral", 
            "fileversion": "0.23.01.2023", 
            "internalname": "SupportAgentCompany", 
            "legalcopyright": "Company name & MeshCentral ", 
            "productname": "Support Agent Company", 
            "productversion": "v0.23.01.2023"
         }, 
         "assistantcustomization": {
            "title": "Company Support Assistant", 
            "image": "MCassistant.png", 
            "fileName": "Company-SupportAssistant"
         }, 
         "androidcustomization": {
            "title": "IT Remote Support", 
            "subtitle": "Company Name", 
            "image": "MCandroid.png"
         }, 
         "newaccounts": false, 
         "passwordrequirements": {
            "force2factor": true, 
            "autofido2fa": true, 
            "email2factor": true, 
            "push2factor": true, 
            "otp2factor": true, 
            "msg2factor": false, 
            "backupcode2factor": true, 
            "loginTokens": false,
			"twoFactorTimeout": 90,
			"allowaccountreset": false
         }, 
         "twofactorcookiedurationdays": 5,  
         "auth": "ldap", 
         "ldapuserbinarykey": "objectSid", 
         "ldapusername": "sAMAccountName", 
         "ldapuseremail": "mail", 
         "ldapuserrealname": "name", 
         "ldapuserphonenumber": "mobile", 
         "ldapsaveusertofile": "C:\\Program Files\\MeshCentral\\meshcentral-data\\ldap.log", 
         "ldapusergroups": "memberOf", 
         "ldapsiteadmingroups": ["CN=res_meshcentral_admin,OU=MeshCentral,DC=redacted,DC=fqdn"], 
         "ldapuserrequiredgroupmembership": ["CN=res_meshcentral_admin,OU=MeshCentral,DC=redacted,DC=fqdn", 
         "CN=res_meshcentral_control,OU=MeshCentral,DC=redacted,DC=fqdn", 
         "CN=res_meshcentral_control,OU=MeshCentral,DC=redacted,DC=fqdn"], 
         "ldapsyncwithusergroups": {
            "filter": ["CN=res_meshcentral_control,OU=MeshCentral,DC=redacted,DC=fqdn", 
            "CN=res_meshcentral_view,OU=MeshCentral,DC=redacted,DC=fqdn"]
         }, 
         "ldapoptions": {
            "url": "ldaps://redacted.fqdn:636/", 
            "tlsOptions": {
               "rejectUnauthorized": false
            }, 
            "bindDN": "CN=MeshCentral,OU=Operators,DC=redacted,DC=fqdn", 
            "bindCredentials": "bindDNPassword", 
            "searchBase": "DC=redacted,DC=fqdn", 
            "searchFilter": "(\u0026(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(sAMAccountName={{username}}))", 
            "reconnect": true
         }
      }
   }, 
   "smtp": {
      "host": "mail.redacted.fqdn",
      "port": 25,
      "from": "mesh@redacted.fqdn",
      "user": "MeshMailUser",
      "pass": "MeshMailPassword",
      "tls": true,
	  "tslcertcheck": false
   }
}

[chrkli]

Maybe related to #4387

[OutbackMatt]

I tend to find that this happens after a windows update on the client machine
Does that match your experience?

[chrkli]

Thanks for the hint with windows update. Unfortunately, this has not shown up in my experience.

The last two workdays I've observed the following behaviour: Whenever a client changes the network connection from ethernet to wifi or vice versa the "desktop" and "files" get lost. On a few clients (around 15) the workstation lock also triggers the issue.

Restarting the agent service on the client also restores the "manageability" - desktop and files tab is visible and usable again.
It surely is not the most effective way but at the moment I've implemented the following workaround on every client:
Create a task on every client with the following triggers

• Microsoft-Windows-Wired-AutoConfig/Operational -> Source Wired-AutoConfig -> ID 15505
• Microsoft-Windows-WiFi-AutoConfig/Operational -> Source "Wifi-AutoConfig" -> ID 8001
• Workstation unlocked by any user
  As action, restart the MeshCentral Agent service via PowerShell, including a 15 seconds delay for the network connection to really stabilise.

[chrkli]

Duplicate of #5008

[chrkli]

See duplikate issue for further updates