Does deleting a deployment delete the docker image from the device?

[baharw]

I am successfully able to add and remove custom modules from a deployment. However, I notice that the modules that I remove, remain on the device, occupying space. How do I remove those images?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 877a68cb-b6ad-52a2-64ff-2965e936b3a0
• Version Independent ID: 07f38c9a-bafb-bfeb-01e9-cfc420d844b5
• Content: Deploy, monitor modules for Azure IoT Edge
• Content Source: articles/iot-edge/how-to-deploy-monitor.md
• Service: iot-edge
• GitHub Login: @kgremban
• Microsoft Alias: kgremban

[asergaz]

Thanks for your feedback @baharw . That is by design - you can easily integrate a direct method for example that would be responsible to remove all unused docker images on your edge device. I nevertheless suggest that you request that feature here with a business justification so others can vote.

Cheers!

[kgremban]

@baharw When you create a deployment for a device, the IoT Edge runtime compares the modules listed in the deployment manifest to the modules running on the device. Any new modules listed in the deployment manifest are added. Any existing modules not in the deployment manifest are removed. If you wanted to keep IoT Edge running on a device but remove all modules, you could create a deployment manifest with just the runtime modules (edgeAgent and edgeHub) with no additional modules.

Or are you saying that there are modules still running on your device that are not listed in the deployment manifest?

[baharw]

@baharw When you create a deployment for a device, the IoT Edge runtime compares the modules listed in the deployment manifest to the modules running on the device. Any new modules listed in the deployment manifest are added. Any existing modules not in the deployment manifest are removed. If you wanted to keep IoT Edge running on a device but remove all modules, you could create a deployment manifest with just the runtime modules (edgeAgent and edgeHub) with no additional modules.

Or are you saying that there are modules still running on your device that are not listed in the deployment manifest?

I am saying that after removing a module from the deployment, the module stops running, but remains on the IoT Edge Device, using up space on the device.

[kgremban]

@baharw I can't reproduce the experience that you're describing. After you remove a module from the deployment, what commands are you using to verify that the modules are still on your device? iotedge list, docker ps, docker ps -a, or docker images?

[baharw]

@baharw I can't reproduce the experience that you're describing. After you remove a module from the deployment, what commands are you using to verify that the modules are still on your device? iotedge list, docker ps, docker ps -a, or docker images?

sudo docker images -a

[kgremban]

Oh, sorry. I've been misinterpreting your question. Yes you're right. I'll refer back to @sergaz-msft's original response.

[baharw]

Oh, sorry. I've been misinterpreting your question. Yes you're right. I'll refer back to @sergaz-msft's original response.

Just to be sure, what @sergaz-msft and you (@kgremban ) are saying is that by design there is no cleanup when modules are removed from the deployment and that modules keep accumulating on the device. So at some point, the device's storage will get filled up with junk (outdated or unnecessary modules).

[kgremban]

Right, there is no automatic removal of the container image from your device. You need to run docker rmi on the device itself to remove images. I misread your initial question to be about the modules themselves, not their images. I recommend following the link that sergaz-msft shared, and requesting this feature in the product.

[baharw]

Right, there is no automatic removal of the container image from your device. You need to run docker rmi on the device itself to remove images. I misread your initial question to be able the modules themselves, not their images. I recommend following the link that sergaz-msft shared, and requesting this feature in the product.

So how would one do this at scale? Would one have to SSH into every device and run docker rmi?

[kgremban]

I suppose so, or follow the earlier suggestion to use direct methods.

[kgremban]

I was thinking more about this question, and heard some users mention installing logrotate on their devices to periodically clean up their logs so they don't get too large. I assume a similar option is available for cleaning up images. A quick search brought up https://hub.docker.com/r/meltwater/docker-cleanup/ as an example of an image that cleans up unused images periodically. Perhaps something like that in your deployment would solve the problem.

[baharw]

@kgremban Thank you for suggesting the options. I will look into them and let you know how it goes. I believe I will not be the only one running into this issue.

[baharw]

@kgremban btw- how would one prune the "iotedge logs"?

[asergaz]

Great question @baharw , let me check it with product group how to do it today. I would also suggest that you add a new feature request here.

Thanks!

[baharw]

Great question @baharw , let me check it with product group how to do it today. I would also suggest that you add a new feature request here.

Thanks!

Thank you for looking into it @sergaz-msft ! I made the request here last week, per your request.

[darobs]

Hello @baharw,

Thank you for the feedback.

There is some management setup in Linux that may be helpful to keep log file sizes down.

For the IoT Edge Security Daemon in Linux, we by default use journald as the logging system. Journald supports log management, read the journald.conf manpage to configure it. If you also switch to using journald as the logging driver for Docker, it will also automatically manage the logs for containers.

If you use json-file logging for Docker, you can specify a limit to docker logfile sizes in the docker options, either at the daemon level (/etc/docker/daemon.json on Linux hosts or C:\ProgramData\docker\config\daemon.json on Windows server hosts) or at each container via the CreateOptions for the module. However, this doesn’t seem to consistently limit log file sizes. If that's the case, you can also install a logrotate (see man page) for Docker, with the following file spec.

/var/lib/docker/containers//-json.log {
copytruncate
daily
rotate 7
delaycompress
compress
notifempty
missingok
}

Other than periodically running docker system prune on the system, I don't have an awesome answer to image cleanup. I have had systems run out of disk space due to lots of large containers, so I know how annoying that can be.

[baharw]

Hello @baharw,

Thank you for the feedback.

There is some management setup in Linux that may be helpful to keep log file sizes down.

For the IoT Edge Security Daemon in Linux, we by default use journald as the logging system. Journald supports log management, read the journald.conf manpage to configure it. If you also switch to using journald as the logging driver for Docker, it will also automatically manage the logs for containers.

If you use json-file logging for Docker, you can specify a limit to docker logfile sizes in the docker options, either at the daemon level (/etc/docker/daemon.json on Linux hosts or C:\ProgramData\docker\config\daemon.json on Windows server hosts) or at each container via the CreateOptions for the module. However, this doesn’t seem to consistently limit log file sizes. If that's the case, you can also install a logrotate (see man page) for Docker, with the following file spec.

/var/lib/docker/containers//-json.log {
copytruncate
daily
rotate 7
delaycompress
compress
notifempty
missingok
}

Other than periodically running docker system prune on the system, I don't have an awesome answer to image cleanup. I have had systems run out of disk space due to lots of large containers, so I know how annoying that can be.

@darobs thank you for the thoughtful response. How does one go about doing this on IoT devices that are hard to access. SSH? How does one do it a scale?

Thank you in advance for any insights.
-Bahar

[chipalost]

Hi @baharw, easing and automating device management tasks is a high priority that we want to build into the runtime. Right now tasks like updating the device OS or managing native components is an exercise left to the user. We've seen some folks use scripts, tools like Ansible, or write device management modules which can be triggered from the cloud. Looking at your specific case of pruning logs, you could create your OS images to limit the amount of space used for logs and automatically prune your logs.

We recently added documentation to talk about issues like this that folks should think about when creating devices headed out to production. Expect to see the experience in this area improve next year.
https://docs.microsoft.com/en-us/azure/iot-edge/production-checklist

[asergaz]

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

[HectorSVC]

Encounter this issue as well. Just went through the thread. I can't believe that, when I deploy a model, the IoT know how to do it. I mean it trigger the device to pull the image and starts a container. However, when I remove a deployment, it only stops the container instance, still left the docker image there! User may run out of disk space unaware of this if they keep trying deploy different modules (some module maybe huge) then remove it.

[kpm-at-hfi]

@asergaz I just ran into this in a field test. I see this issue is closed, but I'm not sure what the resolution actually is for this pretty-easy-to-encounter problem. I have read up over on the Production Checklist, but it's not clear to me that there's any guidance there concerning how to remotely or automatically manage unused Docker images uselessly taking up valuable disk space. Is there an official best practice for those of us wanting to use Azure's IoT offerings?

[Alberto-Vega]

@kpm-at-hfi Please see the suggestions from Edge team on the feedback entry https://feedback.azure.com/forums/907045-azure-iot-edge/suggestions/36069130-removing-unused-modules-from-iot-edge-devices
You can also upvote it if you want to.

[kpm-at-hfi]

Yeah I'd found this issue via that link, and I did upvote :)