Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MDM status is displayed as off when it's actually on #17692

Closed
pacamaster opened this issue Mar 18, 2024 · 17 comments
Closed

MDM status is displayed as off when it's actually on #17692

pacamaster opened this issue Mar 18, 2024 · 17 comments
Assignees
Labels
bug Something isn't working as documented ~critical bug This is a critical bug and may require a patch release. ~csa Issue was created by or deemed important by the Customer Solutions Architect. customer-preston #g-mdm MDM product group P1 Prioritize as critical :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. ~released bug This bug was found in a stable release.
Milestone

Comments

@pacamaster
Copy link
Member

pacamaster commented Mar 18, 2024

Fleet version:
Reported in Fleet Fleet 4.47.0 Go go1.21.7
osquery 5.11.0
Fleetd 1.22.0
Web browser and operating system:
Current version


💥  Actual behavior

  • customer-preston is doing MDM migrations from a different MDM solution to Fleet.
  • fleetd is installed on the host.
  • Automatic Windows MDM Enrollment is enabled in Fleet UI.
  • some of the hosts in this instance ( 28/60 ) are not enrolling into Fleet MDM.
  • computers that have failed enrollment show that MDM is "off"
  • these computers do not show an enrollment "failed" state in the Fleet UI

🧑‍💻  Steps to reproduce

  1. Enroll a host in a different Windows MDM.
  2. Attempt to migrate MDM enrollment to Fleet.

🕯️ More info (optional)

  • Fleet uses a detailed query to pull information and MDM enrollment state.
  • This query checks for previous enrollments and state to make sure that Fleet is only MDM (similar to macOS).
  • enable-scripts was part of the tagged clients' enrollment package
  • customer is able to run a script that unenrolls the device from their previous MDM and successfully enrolls into Fleet, however, Fleet is reporting that some of these hosts do not have scripts enabled.
  • because of the reporting issue (i.e., Fleet UI showing that MDM is "off" when it actually has "failed") it is difficult to know which hosts to remediate. EnrollmentState gets set to 4 for "failed" and there does not appear to retry and the endpoint gets stuck in this state
  • some of these hosts show the script queue stuck in a pending state.
  • the script queue can't be manually cleared

Please see possibly related issue: #17695

@pacamaster pacamaster added bug Something isn't working as documented customer-preston :incoming New issue in triage process. labels Mar 18, 2024
@nonpunctual nonpunctual changed the title Enrollment Windows MDM migration Unable to Complete Windows MDM Enrollment Migration Mar 18, 2024
@JoStableford
Copy link
Contributor

@nonpunctual nonpunctual added the ~csa Issue was created by or deemed important by the Customer Solutions Architect. label Mar 18, 2024
@nonpunctual
Copy link
Contributor

@roperzh @georgekarrv heads up. Thanks!

@nonpunctual nonpunctual added #g-mdm MDM product group ~released bug This bug was found in a stable release. :incoming New issue in triage process. :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. and removed :incoming New issue in triage process. :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. labels Mar 18, 2024
@valentinpezon-primo
Copy link

Thanks - What we would expect as the end-user of fleet solution is that fleet automatically retry the enrollment whenever it's in "failed" state of "off" state

@valentinpezon-primo
Copy link

Also, the issue name is "MDM Enrollment Migration" but it happens enven with device that were never enrolled on any MDM, and for all kind of windows version / os (home, pro, 10, 11..)

@primo-vincent
Copy link

Today (D+2) only 19/61 windows hosts are turned on, it's seems the MDM is able to move from enrolled to failed status

Screenshot 2024-03-21 at 00 02 18

Screenshot 2024-03-21 at 00 01 48

@primo-vincent
Copy link

primo-vincent commented Mar 20, 2024

logs in fleet instance :

  • "limit exceeded"
  • "authentication error: invalid device authentication token"
  • "soap fault: management response message: message syncML creation error creation of SyncML message: invalid session ID"

Screenshot_21_03_2024__00_22

Screenshot_21_03_2024__00_29

@georgekarrv georgekarrv added the :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. label Mar 21, 2024
@valentinpezon-primo
Copy link

Hi, update on our side

As stated, we have some Windows MDM device that where properly enrolled with MDM ON, and for un unknown reason they switch to MDM OFF

Here are the osquery logs for one of this device
Log was pulled thanks to Get-Content C:\Windows\system32\config\systemprofile\AppData\Local\FleetDM\Orbit\Logs\orbit-osquery.log script



2024-03-25T09:10:27+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:10:47+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:10:57+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:11:17+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:11:25+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:11:28+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:11:47+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:11:58+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:12:17+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:12:25+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:12:28+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:12:47+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:12:58+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:13:17+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:13:25+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:13:28+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:13:47+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:13:58+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:14:17+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:14:25+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:14:29+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:14:47+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:14:59+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

I0325 09:15:06.787837  9836 interfaces.cpp:102] Failed to retrieve network statistics for interface 13

I0325 09:15:07.321223  9836 interfaces.cpp:102] Failed to retrieve network statistics for interface 16

I0325 09:15:07.859834  9836 interfaces.cpp:102] Failed to retrieve network statistics for interface 19

I0325 09:15:08.399746  9836 interfaces.cpp:102] Failed to retrieve network statistics for interface 20

I0325 09:15:08.955835  9836 interfaces.cpp:102] Failed to retrieve network statistics for interface 2

I0325 09:15:09.513701  9836 interfaces.cpp:102] Failed to retrieve network statistics for interface 1

I0325 09:15:09.961730  9836 interfaces.cpp:130] Failed to retrieve physical state for interface 1

I0325 09:15:10.081773  9836 interfaces.cpp:157] Failed to retrieve DHCP and DNS information for interface 1

W0325 09:15:12.857724  9836 chocolatey_packages.cpp:65] Did not find chocolatey path environment variable

2024-03-25T09:15:17+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:15:25+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:15:29+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:15:47+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:15:59+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:16:17+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:16:25+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:16:29+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:16:47+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:16:59+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:17:17+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:17:25+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:17:29+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:17:47+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:18:00+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:18:17+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:18:26+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:18:30+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:18:47+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"

2024-03-25T09:19:00+01:00 INF calling RegisterDeviceWithManagement to enroll Windows device failed error="RegisterDeviceWithManagement failed: The operation completed successfully. (0x8018000a - 2149056522)"




@valentinpezon-primo
Copy link

Another one

with "2024-03-21T09:31:43+01:00 ERR there was an error calling RegisterDeviceWithLocalManagement(): (0x80070032)"

6b58a274e93db2cea721c4d400fce29c41c7147ad6aa13bf0101319b9bb579f05

2024-03-21T09:18:25+01:00 INF hash(desktop)=4d2287c3747ed49158613b896c8d1662c1d7ae401ba0e5aead554b48426756889d74a025c9ec45229034863379f3a17e320eb1467b28925b8bfa24deae90128e

2024-03-21T09:18:25+01:00 INF Found osquery version: 5.11.0

2024-03-21T09:18:27+01:00 ERR failed to encrypt the volume

2024-03-21T09:18:27+01:00 ERR failed to encrypt the volume

2024-03-21T09:18:27+01:00 ERR failed to encrypt the volume

2024-03-21T09:18:28+01:00 INF token rotation is enabled

2024-03-21T09:18:28+01:00 INF start osqueryd cmd="C:\\Program Files\\Orbit\\bin\\osqueryd\\windows\\stable\\osqueryd.exe --pidfile=C:\\Program Files\\Orbit\\osquery.pid --extensions_socket=\\\\.\\pipe\\orbit-osquery-extension --logger_path=C:\\Program Files\\Orbit\\osquery_log --enroll_secret_env ENROLL_SECRET --tls_hostname=aventa.mdm.getprimo.com --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls,filesystem --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=8000000 --tls_server_certs C:\\Program Files\\Orbit\\certs.pem --force --flagfile C:\\Program Files\\Orbit\\osquery.flags --host-identifier uuid --database_path C:\\Program Files\\Orbit\\osquery.db"

2024-03-21T09:18:28+01:00 INF killing any pre-existing fleet-desktop instances

2024-03-21T09:18:28+01:00 INF opening path="C:\\Program Files\\Orbit\\bin\\desktop\\windows\\stable\\fleet-desktop.exe"

I0321 09:18:29.006330 29928 interface.cpp:137] Registering extension (com.fleetdm.orbit.osquery_extension.v1, 11808, version=, sdk=)

I0321 09:18:31.163288 26868 eventfactory.cpp:156] Event publisher not enabled: etw_process_publisher: etw_process_publisher publisher disabled via configuration.

I0321 09:18:31.164206 26868 eventfactory.cpp:156] Event publisher not enabled: ntfs_event_publisher: NTFS event publisher disabled via configuration

2024-03-21T09:18:58+01:00 ERR failed to encrypt the volume

2024-03-21T09:19:28+01:00 ERR failed to encrypt the volume

2024-03-21T09:19:58+01:00 ERR failed to encrypt the volume

2024-03-21T09:20:28+01:00 ERR failed to encrypt the volume

2024-03-21T09:20:58+01:00 ERR failed to encrypt the volume

2024-03-21T09:21:28+01:00 ERR failed to encrypt the volume

2024-03-21T09:21:30+01:00 ERR there was an error calling RegisterDeviceWithLocalManagement(): (0x80070032)

2024-03-21T09:21:59+01:00 INF found enroll secret in keystore: Credential Manager

2024-03-21T09:21:59+01:00 INF hash(orbit)=9348488d6db394112a36fea5de336809b44b21f16097fef8e9f2cb44aa0ccfa44f134e3a92ae20c3a7094b269a1f1887643ec13e74ca419b7fcafb6e0f5b8691

2024-03-21T09:21:59+01:00 INF hash(osqueryd)=3c507251b94223616bc18527fe81899af7c292260f44fe3361d5c29734906e66b58a274e93db2cea721c4d400fce29c41c7147ad6aa13bf0101319b9bb579f05

2024-03-21T09:22:00+01:00 INF hash(desktop)=4d2287c3747ed49158613b896c8d1662c1d7ae401ba0e5aead554b48426756889d74a025c9ec45229034863379f3a17e320eb1467b28925b8bfa24deae90128e

2024-03-21T09:22:00+01:00 INF Found osquery version: 5.11.0

2024-03-21T09:22:02+01:00 ERR failed to encrypt the volume

2024-03-21T09:22:02+01:00 ERR failed to encrypt the volume

2024-03-21T09:22:02+01:00 ERR failed to encrypt the volume

2024-03-21T09:22:03+01:00 INF token rotation is enabled

2024-03-21T09:22:03+01:00 INF killing any pre-existing fleet-desktop instances

2024-03-21T09:22:03+01:00 INF start osqueryd cmd="C:\\Program Files\\Orbit\\bin\\osqueryd\\windows\\stable\\osqueryd.exe --pidfile=C:\\Program Files\\Orbit\\osquery.pid --extensions_socket=\\\\.\\pipe\\orbit-osquery-extension --logger_path=C:\\Program Files\\Orbit\\osquery_log --enroll_secret_env ENROLL_SECRET --tls_hostname=aventa.mdm.getprimo.com --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls,filesystem --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=8000000 --tls_server_certs C:\\Program Files\\Orbit\\certs.pem --force --flagfile C:\\Program Files\\Orbit\\osquery.flags --host-identifier uuid --database_path C:\\Program Files\\Orbit\\osquery.db"

I0321 09:22:03.672083 21992 interface.cpp:137] Registering extension (com.fleetdm.orbit.osquery_extension.v1, 6272, version=, sdk=)

2024-03-21T09:22:03+01:00 INF opening path="C:\\Program Files\\Orbit\\bin\\desktop\\windows\\stable\\fleet-desktop.exe"

I0321 09:22:05.949203 11980 eventfactory.cpp:156] Event publisher not enabled: etw_process_publisher: etw_process_publisher publisher disabled via configuration.

I0321 09:22:05.950112 11980 eventfactory.cpp:156] Event publisher not enabled: ntfs_event_publisher: NTFS event publisher disabled via configuration

2024-03-21T09:22:33+01:00 ERR failed to encrypt the volume

2024-03-21T09:23:03+01:00 ERR failed to encrypt the volume

2024-03-21T09:23:33+01:00 ERR failed to encrypt the volume

2024-03-21T09:24:03+01:00 ERR failed to encrypt the volume

2024-03-21T09:24:33+01:00 ERR failed to encrypt the volume

2024-03-21T09:25:03+01:00 ERR failed to encrypt the volume

2024-03-21T09:25:03+01:00 ERR failed to encrypt the volume

2024-03-21T09:25:33+01:00 ERR failed to encrypt the volume

2024-03-21T09:26:03+01:00 ERR failed to encrypt the volume

2024-03-21T09:26:03+01:00 ERR failed to encrypt the volume

2024-03-21T09:26:33+01:00 ERR failed to encrypt the volume

2024-03-21T09:26:40+01:00 ERR there was an error calling RegisterDeviceWithLocalManagement(): (0x80070032)

2024-03-21T09:27:03+01:00 ERR failed to encrypt the volume

2024-03-21T09:27:04+01:00 INF found enroll secret in keystore: Credential Manager

2024-03-21T09:27:05+01:00 INF hash(orbit)=9348488d6db394112a36fea5de336809b44b21f16097fef8e9f2cb44aa0ccfa44f134e3a92ae20c3a7094b269a1f1887643ec13e74ca419b7fcafb6e0f5b8691

2024-03-21T09:27:05+01:00 INF hash(osqueryd)=3c507251b94223616bc18527fe81899af7c292260f44fe3361d5c29734906e66b58a274e93db2cea721c4d400fce29c41c7147ad6aa13bf0101319b9bb579f05

2024-03-21T09:27:05+01:00 INF hash(desktop)=4d2287c3747ed49158613b896c8d1662c1d7ae401ba0e5aead554b48426756889d74a025c9ec45229034863379f3a17e320eb1467b28925b8bfa24deae90128e

2024-03-21T09:27:05+01:00 INF Found osquery version: 5.11.0

2024-03-21T09:27:07+01:00 ERR failed to encrypt the volume

2024-03-21T09:27:07+01:00 ERR failed to encrypt the volume

2024-03-21T09:27:07+01:00 ERR failed to encrypt the volume

2024-03-21T09:27:07+01:00 INF token rotation is enabled

2024-03-21T09:27:08+01:00 INF killing any pre-existing fleet-desktop instances

2024-03-21T09:27:08+01:00 INF start osqueryd cmd="C:\\Program Files\\Orbit\\bin\\osqueryd\\windows\\stable\\osqueryd.exe --pidfile=C:\\Program Files\\Orbit\\osquery.pid --extensions_socket=\\\\.\\pipe\\orbit-osquery-extension --logger_path=C:\\Program Files\\Orbit\\osquery_log --enroll_secret_env ENROLL_SECRET --tls_hostname=aventa.mdm.getprimo.com --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls,filesystem --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=8000000 --tls_server_certs C:\\Program Files\\Orbit\\certs.pem --force --flagfile C:\\Program Files\\Orbit\\osquery.flags --host-identifier uuid --database_path C:\\Program Files\\Orbit\\osquery.db"

I0321 09:27:08.472947 28676 interface.cpp:137] Registering extension (com.fleetdm.orbit.osquery_extension.v1, 21024, version=, sdk=)

2024-03-21T09:27:08+01:00 INF opening path="C:\\Program Files\\Orbit\\bin\\desktop\\windows\\stable\\fleet-desktop.exe"

I0321 09:27:10.693408 30284 eventfactory.cpp:156] Event publisher not enabled: etw_process_publisher: etw_process_publisher publisher disabled via configuration.

I0321 09:27:10.693408 30284 eventfactory.cpp:156] Event publisher not enabled: ntfs_event_publisher: NTFS event publisher disabled via configuration

2024-03-21T09:27:38+01:00 INF token TTL expired, rotating token

2024-03-21T09:27:38+01:00 ERR failed to encrypt the volume

2024-03-21T09:28:08+01:00 ERR failed to encrypt the volume

2024-03-21T09:28:38+01:00 ERR failed to encrypt the volume

2024-03-21T09:29:08+01:00 ERR failed to encrypt the volume

2024-03-21T09:29:38+01:00 ERR failed to encrypt the volume

2024-03-21T09:30:08+01:00 ERR failed to encrypt the volume

2024-03-21T09:30:08+01:00 ERR failed to encrypt the volume

2024-03-21T09:30:38+01:00 ERR failed to encrypt the volume

2024-03-21T09:31:08+01:00 ERR failed to encrypt the volume

2024-03-21T09:31:08+01:00 ERR failed to encrypt the volume

2024-03-21T09:31:38+01:00 ERR failed to encrypt the volume

2024-03-21T09:31:43+01:00 ERR there was an error calling RegisterDeviceWithLocalManagement(): (0x80070032)

2024-03-21T09:32:08+01:00 ERR failed to encrypt the volume

2024-03-21T09:32:08+01:00 ERR failed to encrypt the volume


@martinpannier
Copy link

martinpannier commented Mar 26, 2024

Hi team, just to provide some context - as shared by @valentinpezon-primo above, this problem is not limited to migrations:

Also, the issue name is "MDM Enrollment Migration" but it happens even with device that were never enrolled on any MDM, and for all kind of windows version / os (home, pro, 10, 11..)

And

Today (D+2) only 19/61 windows hosts are turned on, it's seems the MDM is able to move from enrolled to failed status

Just to put some numbers to this:

  • Customer 1, 80 devices total, 57 devices enrolled: over time, the number of devices with MDM "on" has diminished from 19 ten days ago to 5 four days ago to 1 today
  • In total, there are 200 devices across 25 customers that have switched to MDM "off" in the last few weeks

@lukeheath
Copy link
Member

@martinpannier I'm sorry you're experiencing this. We are prioritizing and will investigate today.

cc @georgekarrv @roperzh

@martinpannier
Copy link

Appreciate it - thanks @lukeheath and team

dantecatalfamo added a commit that referenced this issue Mar 26, 2024
#17692

Recently there was a change that filtered out hosts in `EnrollmentState`
3. This change may cause some hosts that are in otherwise good health to
appear unresponsive to MDM in the management UI.

This change will allow hosts with `EnrollmentStatus` 3 show as enrolled.

The root cause of some hosts being in state 3 is still not entirely
clear, but may have to do with either trying to re-enroll once already
enrolled, or windows updates causing some sort of issue with fleet.

Despite the "failed" `EnrollmentState` 3, the host will still display
that the system is managed by Fleet, and will actively sync.
@georgekarrv georgekarrv removed the :incoming New issue in triage process. label Mar 26, 2024
@georgekarrv georgekarrv added this to the 4.47.3 milestone Mar 26, 2024
georgekarrv pushed a commit that referenced this issue Mar 26, 2024
#17692

Recently there was a change that filtered out hosts in `EnrollmentState`
3. This change may cause some hosts that are in otherwise good health to
appear unresponsive to MDM in the management UI.

This change will allow hosts with `EnrollmentStatus` 3 show as enrolled.

The root cause of some hosts being in state 3 is still not entirely
clear, but may have to do with either trying to re-enroll once already
enrolled, or windows updates causing some sort of issue with fleet.

Despite the "failed" `EnrollmentState` 3, the host will still display
that the system is managed by Fleet, and will actively sync.
@lukeheath lukeheath added P1 Prioritize as critical ~critical bug This is a critical bug and may require a patch release. labels Mar 26, 2024
georgekarrv pushed a commit that referenced this issue Mar 26, 2024
#17692

Recently there was a change that filtered out hosts in `EnrollmentState`
3. This change may cause some hosts that are in otherwise good health to
appear unresponsive to MDM in the management UI.

This change will allow hosts with `EnrollmentStatus` 3 show as enrolled.

The root cause of some hosts being in state 3 is still not entirely
clear, but may have to do with either trying to re-enroll once already
enrolled, or windows updates causing some sort of issue with fleet.

Despite the "failed" `EnrollmentState` 3, the host will still display
that the system is managed by Fleet, and will actively sync.
@noahtalerman noahtalerman changed the title Unable to Complete Windows MDM Enrollment Migration MDM status is displayed as off when it's actually on Mar 26, 2024
@fleet-release
Copy link
Contributor

In the cloud city,
Fleet's truth aligns with view,
Clarity shines through.

@valentinpezon-primo
Copy link

Hi @georgekarrv @dantecatalfamo ,

It means now fleet will display them as "Enrolled" even if it's failed ?

Are we still able to run mdm commands on them ? Does the custom OS settings (Configuration profile ) are well applied on the host ?

@georgekarrv
Copy link
Member

Yes this 'failure' is not necessarily a full failure of mdm capabilities. (We still haven't tracked 100% what causes this) Ultimately this looks like an event status where we do something that causes a failure and this enrollment registry value changes while still maintaining mdm connections and syncs.

@valentinpezon-primo
Copy link

Yes this 'failure' is not necessarily a full failure of mdm capabilities. (We still haven't tracked 100% what causes this) Ultimately this looks like an event status where we do something that causes a failure and this enrollment registry value changes while still maintaining mdm connections and syncs.

Okay !
I think it would be good to test if all MDM capabilities are maintained, since you haven't 100% tracked what causes this, the same cause could make some MDM capabilites fails for ex.. We will try to do some tests in our end but if you can also verify it, It would be perfect 🙏

Thanks for the udpate

@nonpunctual
Copy link
Contributor

@noahtalerman @georgekarrv I know we discussed adding this in a separate issue but I don't want it to be lost.

I think we should consider deleting previous Windows enrollment artifacts from the Registry when a migration to Fleet Windows MDM occurs. This (in my mind) would solve the problem. if there are no artifacts from prior enrollments, there would not be any question about reporting.

There probably are edge cases where customers may want to retain previous enrollments on the Host, or, they may want to try enrolling in multiple management systems. If this is a concern we could have a switch in the UI for disabling the capability to "clean up" old Registry artifacts on a new Fleet enrollment.

@georgekarrv
Copy link
Member

Absolutely, if you open a FR and we get it prioritized that sounds fine to me. I will say that in this case our reporting was based on the Fleet registry entry and didn't look to be interfering from our initial investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working as documented ~critical bug This is a critical bug and may require a patch release. ~csa Issue was created by or deemed important by the Customer Solutions Architect. customer-preston #g-mdm MDM product group P1 Prioritize as critical :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. ~released bug This bug was found in a stable release.
Development

No branches or pull requests

10 participants