Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Device doesn't reboot after update #82

Closed
SimonSimCity opened this issue Jul 14, 2021 · 8 comments
Closed

Device doesn't reboot after update #82

SimonSimCity opened this issue Jul 14, 2021 · 8 comments
Labels
area: yocto There is an issue with Yocto meta-layers

Comments

@SimonSimCity
Copy link

SimonSimCity commented Jul 14, 2021 •
edited
Loading

Following my writeup in #79 - after some initial trouble with my NFS root filesystem - I got the system started and could connect my device to the Azure IoT Hub. All this worked as expected, and I could build an image which I uploaded to the Azure Device Update service according to the manual (https://docs.microsoft.com/en-us/azure/iot-hub-device-update/device-update-raspberry-pi). Sadly, the manual is quite outdated and some information about the manifest had to be taken from the documentation found in the repo (https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/iot-hub-device-update/update-manifest.md).

After taking all this ... I ended up with a system which I could flash (at least out of the logs it seemed to work) but which didn't reboot. Here are the logs I got from the service:

root@dagonet:/adu/logs# tail -f aduc.20210714-140755.log
2021-07-14T14:07:55.0591Z [I] Agent (linux; 0.6.0-public-preview) starting. [main]
2021-07-14T14:07:55.0592Z [I] Agent built with handlers: microsoft/swupdate. [main]
2021-07-14T14:07:55.0717Z [I] Initalizing PnP components. [ADUC_PnP_Components_Create]
2021-07-14T14:07:55.0718Z [I] ADUC agent started. Using IoT Hub Client SDK 1.6.0 [AzureDeviceUpdateCoreInterface_Create]
2021-07-14T14:07:55.0718Z [I] Calling ADUC_Register [ADUC_MethodCall_Register]
2021-07-14T14:07:55.0719Z [I] IoTHub Device Twin callback registered. [ADUC_DeviceClient_Create]
2021-07-14T14:07:55.0847Z [W] Failed to pass connection string to DO, error: -3 [StartupAgent]
2021-07-14T14:07:55.0848Z [I] Agent running. [main]
2021-07-14T14:07:55.7106Z [D] IotHub connection status: 0, reason:6 [ADUC_ConnectionStatus_Callback]
2021-07-14T14:07:56.1125Z [I] Processing existing Device Twin data after agent started. [ADUC_PnPDeviceTwin_Callback]
2021-07-14T14:07:56.1125Z [D] Notifies components that all callback are subscribed. [ADUC_PnPDeviceTwin_Callback]
2021-07-14T14:07:56.1125Z [I] DeviceInformation component is ready - reporting properties [DeviceInfoInterface_Connected]
2021-07-14T14:07:56.1127Z [I] Property manufacturer changed to Contoso [RefreshDeviceInfoInterfaceData]
2021-07-14T14:07:56.1129Z [I] Property model changed to Virtual-Machine [RefreshDeviceInfoInterfaceData]
2021-07-14T14:07:56.1129Z [I] Property osName changed to Linux [RefreshDeviceInfoInterfaceData]
2021-07-14T14:07:56.1130Z [I] Property swVersion changed to 5.4.72 [RefreshDeviceInfoInterfaceData]
2021-07-14T14:07:56.1130Z [I] Property processorArchitecture changed to armv7l [RefreshDeviceInfoInterfaceData]
2021-07-14T14:07:56.1421Z [I] Property processorManufacturer changed to ARM [RefreshDeviceInfoInterfaceData]
2021-07-14T14:07:56.1422Z [I] Property totalMemory changed to 3919740 [RefreshDeviceInfoInterfaceData]
2021-07-14T14:07:56.1423Z [I] Property totalStorage changed to 2575284 [RefreshDeviceInfoInterfaceData]
2021-07-14T14:07:56.1425Z [I] No update content. Reporting Idle state. [ADUC_Workflow_HandleStartupWorkflowData]
2021-07-14T14:07:56.1425Z [I] Setting UpdateState to Idle [ADUC_SetUpdateStateHelper]
2021-07-14T14:07:56.1426Z [I] Reporting state: 0, Idle (0); HTTP 200; result 1, 0 [AzureDeviceUpdateCoreInterface_ReportStateAndResultAsync]
2021-07-14T14:07:56.1427Z [I] UpdateAction: Idle. WorkflowId: 210714140756 [ADUC_MethodCall_Idle]
2021-07-14T14:07:56.1427Z [I] Calling IdleCallback [ADUC_MethodCall_Idle]
2021-07-14T14:07:56.1427Z [I] Now idle. workflowId: 210714140756 [Idle]
2021-07-14T14:09:52.2336Z [D] ComponentName:azureDeviceUpdateAgent, propertyName:service [ADUC_PnP_ComponentClient_PropertyUpdate_Callback]
2021-07-14T14:09:52.2340Z [D] OrchestratorUpdateCallback received property JSON string ({"action":0,"updateManifest":"{\"manifestVersion\":\"2.0\",\"updateId\":{\"provider\":\"Munit\",\"name\":\"Greenbelt\",\"version\":\"1.0.0.0\"},\"updateType\":\"microsoft\/swupdate:1\",\"installedCriteria\":\"0.0.0.1\",\"files\":{\"06e7385af5cf132a\":{\"fileName\":\"core-image-base-raspberrypi4-20210714070044.rootfs.ext4.gz\",\"sizeInBytes\ [OrchestratorUpdateCallback]
2021-07-14T14:09:52.2377Z [I] Processing 'Download' action [ADUC_Workflow_HandleUpdateAction]
2021-07-14T14:09:52.2378Z [I] UpdateAction: Prepare - calling PrepareCallback [ADUC_MethodCall_Prepare]
2021-07-14T14:09:52.2379Z [I] {210714140756} Received Metadata, UpdateType: microsoft/swupdate:1, UpdateTypeName: microsoft/swupdate, UpdateTypeVersion: 1, FileCount: 1 [Prepare]
2021-07-14T14:09:52.2380Z [I] microsoft_swupdate_CreateFunc called. [microsoft_swupdate_CreateFunc]
2021-07-14T14:09:52.2380Z [I] Prepare succeeded. [Prepare]
2021-07-14T14:09:52.2381Z [I] UpdateAction: Download [ADUC_MethodCall_Download]
2021-07-14T14:09:52.2381Z [I] Calling SandboxCreateCallback [ADUC_MethodCall_Download]
2021-07-14T14:09:52.2383Z [I] Setting sandbox /tmp/aduc-dl-210714140756 [SandboxCreate]
2021-07-14T14:09:52.2383Z [I] Using sandbox /tmp/aduc-dl-210714140756 [ADUC_MethodCall_Download]
2021-07-14T14:09:52.2383Z [I] Setting UpdateState to DownloadStarted [ADUC_SetUpdateStateHelper]
2021-07-14T14:09:52.2384Z [I] Calling DownloadCallback [ADUC_MethodCall_Download]
2021-07-14T14:09:52.2385Z [I] Download thread started [DownloadCallback]
2021-07-14T14:09:52.2387Z [D] Downloading 1 files to /tmp/aduc-dl-210714140756 [Download]
2021-07-14T14:09:52.2388Z [I] OrchestratorPropertyUpdateCallback ended [OrchestratorUpdateCallback]
2021-07-14T14:09:52.2390Z [I] Downloading File 'core-image-base-raspberrypi4-20210714070044.rootfs.ext4.gz' from 'http://updateinstance--updateservice.b.nlu.dl.adu.microsoft.com/northeurope/UpdateInstance--updateservice/919236e874694fcb83921da95bc2ac43/core-image-base-raspberrypi4-20210714070044.rootfs.ext4.gz' to '/tmp/aduc-dl-210714140756/core-image-base-raspberrypi4-20210714070044.rootfs.ext4.gz' [Download]
2021-07-14T14:10:06.6648Z [I] Validating file hash [Download]
2021-07-14T14:10:14.1029Z [I] microsoft_swupdate_CreateFunc called. [microsoft_swupdate_CreateFunc]
2021-07-14T14:10:14.1030Z [I] Download called - no-op for swupdate [Download]
2021-07-14T14:10:14.1030Z [I] Content Handler Download resultCode: 1, extendedCode: 0 [Download]
2021-07-14T14:10:14.1031Z [I] ProgressCallback: workflowId: 210714140756; Id 06e7385af5cf132a; State: Completed; Bytes: 125770543/125770543 [DownloadProgressCallback]
2021-07-14T14:10:14.1031Z [I] Download resultCode: 1, extendedCode: 0 [Download]
2021-07-14T14:10:14.1031Z [I] Action 'Download' complete. Result: 1, 0 [ADUC_Workflow_WorkCompletionCallback]
2021-07-14T14:10:14.1032Z [I] WorkCompletionCallback: Download succeeded. Going to state DownloadSucceeded [ADUC_Workflow_WorkCompletionCallback]
2021-07-14T14:10:14.1032Z [I] Setting UpdateState to DownloadSucceeded [ADUC_SetUpdateStateHelper]
2021-07-14T14:10:14.1032Z [I] Reporting state: DownloadSucceeded (2) [AzureDeviceUpdateCoreInterface_ReportStateAndResultAsync]
2021-07-14T14:10:18.1841Z [D] ComponentName:azureDeviceUpdateAgent, propertyName:service [ADUC_PnP_ComponentClient_PropertyUpdate_Callback]
2021-07-14T14:10:18.1845Z [D] OrchestratorUpdateCallback received property JSON string ({"action":1,"updateManifest":"{\"manifestVersion\":\"2.0\",\"updateId\":{\"provider\":\"Munit\",\"name\":\"Greenbelt\",\"version\":\"1.0.0.0\"},\"updateType\":\"microsoft\/swupdate:1\",\"installedCriteria\":\"0.0.0.1\",\"files\":{\"06e7385af5cf132a\":{\"fileName\":\"core-image-base-raspberrypi4-20210714070044.rootfs.ext4.gz\",\"sizeInBytes\ [OrchestratorUpdateCallback]
2021-07-14T14:10:18.1883Z [I] Processing 'Install' action [ADUC_Workflow_HandleUpdateAction]
2021-07-14T14:10:18.1883Z [I] UpdateAction: Install [ADUC_MethodCall_Install]
2021-07-14T14:10:18.1884Z [I] Setting UpdateState to InstallStarted [ADUC_SetUpdateStateHelper]
2021-07-14T14:10:18.1884Z [I] Calling InstallCallback [ADUC_MethodCall_Install]
2021-07-14T14:10:18.1884Z [I] Install thread started [InstallCallback]
2021-07-14T14:10:18.1886Z [I] Installing from /tmp/aduc-dl-210714140756 [Install]
2021-07-14T14:10:18.1887Z [I] OrchestratorPropertyUpdateCallback ended [OrchestratorUpdateCallback]
2021-07-14T14:10:18.1888Z [I] Installing image file: core-image-base-raspberrypi4-20210714070044.rootfs.ext4.gz [Install]
2021-07-14T14:10:18.2055Z [I] Install succeeded [Install]
2021-07-14T14:10:18.2056Z [I] Action 'Install' complete. Result: 1, 0 [ADUC_Workflow_WorkCompletionCallback]
2021-07-14T14:10:18.2057Z [I] WorkCompletionCallback: Install succeeded. Going to state InstallSucceeded [ADUC_Workflow_WorkCompletionCallback]
2021-07-14T14:10:18.2057Z [I] Setting UpdateState to InstallSucceeded [ADUC_SetUpdateStateHelper]
2021-07-14T14:10:18.2058Z [I] Reporting state: InstallSucceeded (4) [AzureDeviceUpdateCoreInterface_ReportStateAndResultAsync]
2021-07-14T14:10:23.0977Z [D] ComponentName:azureDeviceUpdateAgent, propertyName:service [ADUC_PnP_ComponentClient_PropertyUpdate_Callback]
2021-07-14T14:10:23.0980Z [D] OrchestratorUpdateCallback received property JSON string ({"action":2,"updateManifest":"{\"manifestVersion\":\"2.0\",\"updateId\":{\"provider\":\"Munit\",\"name\":\"Greenbelt\",\"version\":\"1.0.0.0\"},\"updateType\":\"microsoft\/swupdate:1\",\"installedCriteria\":\"0.0.0.1\",\"files\":{\"06e7385af5cf132a\":{\"fileName\":\"core-image-base-raspberrypi4-20210714070044.rootfs.ext4.gz\",\"sizeInBytes\ [OrchestratorUpdateCallback]
2021-07-14T14:10:23.1017Z [I] Processing 'Apply' action [ADUC_Workflow_HandleUpdateAction]
2021-07-14T14:10:23.1017Z [I] UpdateAction: Apply [ADUC_MethodCall_Apply]
2021-07-14T14:10:23.1018Z [I] Setting UpdateState to ApplyStarted [ADUC_SetUpdateStateHelper]
2021-07-14T14:10:23.1018Z [I] Calling ApplyCallback [ADUC_MethodCall_Apply]
2021-07-14T14:10:23.1018Z [I] Apply thread started [ApplyCallback]
2021-07-14T14:10:23.1020Z [I] Applying data from /tmp/aduc-dl-210714140756 [Apply]
2021-07-14T14:10:23.1021Z [I] OrchestratorPropertyUpdateCallback ended [OrchestratorUpdateCallback]
2021-07-14T14:10:23.1159Z [I] Action 'Apply' complete. Result: 3, 0 [ADUC_Workflow_WorkCompletionCallback]
2021-07-14T14:10:23.1160Z [I] Apply indicated success with RebootRequired - rebooting system now [ADUC_MethodCall_Apply_Complete]
2021-07-14T14:10:23.1161Z [I] Calling ADUC_RebootSystem [ADUC_MethodCall_RebootSystem]
2021-07-14T14:10:23.1162Z [I] ADUC_RebootSystem called. Rebooting system. [ADUC_RebootSystem]
2021-07-14T14:10:23.1351Z [I] execvp failed, ret -1, error 13
2021-07-14T14:10:23.1242Z [E] Reboot failed. [ADUC_RebootSystem]
2021-07-14T14:10:23.1246Z [E] Reboot attempt failed. [ADUC_MethodCall_Apply_Complete]
RebootRequired - rebooting system now [ADUC_MethodCall_Apply_Complete]
2021-07-14T14:10:23.1161Z [I] Calling ADUC_RebootSystem [ADUC_MethodCall_RebootSystem]
2021-07-14T14:10:23.1162Z [I] ADUC_RebootSystem called. Reb [ADUC_RebootSystem]
2021-07-14T14:10:23.1353Z [I] WorkCompletionCallback: Apply succeeded. Going to state Idle [ADUC_Workflow_WorkCompletionCallback]
2021-07-14T14:10:23.1353Z [I] Setting UpdateState to Idle [ADUC_SetUpdateStateHelper]
2021-07-14T14:10:23.1353Z [I] Calling SandboxDestroyCallback [ADUC_MethodCall_Idle]
2021-07-14T14:10:23.1354Z [I] Destroying sandbox /tmp/aduc-dl-210714140756. workflowId: 210714140756 [SandboxDestroy]
2021-07-14T14:10:23.1354Z [E] Unable to remove sandbox, error -1 [SandboxDestroy]
2021-07-14T14:10:23.1356Z [I] UpdateAction: Idle. WorkflowId: 210714141023 [ADUC_MethodCall_Idle]
2021-07-14T14:10:23.1356Z [I] Calling IdleCallback [ADUC_MethodCall_Idle]
2021-07-14T14:10:23.1356Z [I] Now idle. workflowId: 210714141023 [Idle]

After those lines, I manually triggered a restart but couldn't verify that the operations actually were taking place ...

Any idea of what could've gone wrong?
@SimonSimCity
Copy link
Author

After comparing the bitbake files to the postinst script found in the .deb package, I discovered that the permissions are not set correctly in Yocto. If you have the time, feel free to post me an update ... Here are the commands I had to run:

mkdir -p /home/adu /home/do
chown adu:adu /home/adu
chown do:do /home/do
chmod u=rwx,g=rx /home/adu
chmod u=rwx,g=rx/home/do
chmod u=rwx,g=rx /home/do

chown "root:adu" /usr/lib/adu/adu-shell
chmod u=rxs /usr/lib/adu/adu-shell
setfacl -m "group::---" /usr/lib/adu/adu-shell
setfacl -m "user::r--" /usr/lib/adu/adu-shell
setfacl -m "user:adu:r-x" /usr/lib/adu/adu-shell

@ValOlson
Copy link
Contributor

Hi @SimonSimCity,

Thank you for raising this issue. We will investigate and add a fix in our code.

@SimonSimCity
Copy link
Author

SimonSimCity commented Aug 20, 2021 •
edited
Loading

I tried to pack those lines mentioned into my recipe, but that complained about not finding setfacl. As I've tried, it doesn't work if I just set the permission to something different.

I noticed that the script didn't set the group of the file to adu, but left it as root. Therefore, I replaced all ${PN}-adu parameters by ${PN} - removing the -adu and removed the EXTRA_USERS_PARAMS setting from the file meta-azure-device-update/conf/layer.conf - otherwise the user and group adu and do were created too late for the script to use it on install. Before I had a ton of problems because the permissions weren't applied.

Now that the permissions are set correctly, all makes sense again; let me explained:

The update service is running as adu, so this user should be able to flash the device; but flashing is only permitted by root. This can be achieved by setting the s flag for the file. As pointed out at the comment on the question https://unix.stackexchange.com/questions/118853/what-does-the-s-attribute-in-file-permissions-mean#comment186210_118854:

If setuid bit is set, it executes with ITS owner's permissions (usually root's), no matter who calls it.

This also means, that there wasn't a need for changing the permissions - the problem really was, that the users and groups were created too late in the process, so the install script gave the file a group-relation of root instead of adu.

@SimonSimCity
Copy link
Author

I've created a repository to document my changes of the yocto layer. Here's the commit to fix this problem: munit-solutions/meta-azure-device-update@58a74e3

@shiyi-peng
Copy link
Contributor

I've created a repository to document my changes of the yocto layer. Here's the commit to fix this problem: SimonSimCity/meta-azure-device-update@58a74e3

@SimonSimCity Hi, thank you for the investigation and contribution! But I cannot access this link (maybe I don't have permission to your fork) - do you mind creating a Pull Request to our repo so I can read/merge it?

@shiyi-peng
Copy link
Contributor

I've created a repository to document my changes of the yocto layer. Here's the commit to fix this problem: SimonSimCity/meta-azure-device-update@58a74e3

@SimonSimCity Hi, thank you for the investigation and contribution! But I cannot access this link (maybe I don't have permission to your fork) - do you mind creating a Pull Request to our repo so I can read/merge it?

Okie I found the link from another open issue. munit-solutions/meta-azure-device-update@53a54e7
This is actually out of scope of this iot-hub-device-update repo, so I understand that no PR will be created.
Is your problem resolved with your changes?

@SimonSimCity
Copy link
Author

@shiyi-peng Sorry for the late answer on this ticket. I've moved the repo to a different organization lately, so that's where the error comes from. Here's the correct link of the referred change: munit-solutions/meta-azure-device-update@58a74e3

If you apply this patch the users should be created by Yocto as expected.

As said, those changes are only related to yocto-scripts, which are not yet in a repository but only provided as an artefact. I've created a repository for the yocto meta-layer, so the community can work on improving the layer for everyone.

@eshashah-msft eshashah-msft added the area: yocto There is an issue with Yocto meta-layers label Jul 19, 2022
@eshashah-msft
Copy link
Contributor

Closing this issue as there has been no activity on this thread. Please feel free to open an issue if needed.
Our Yocto layers published in this repo and will be maintained for our releases: https://github.com/Azure/iot-hub-device-update-yocto

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: yocto There is an issue with Yocto meta-layers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@SimonSimCity @ValOlson @shiyi-peng @eshashah-msft