umount /run/cloud-init/tmp/tmpl5n7csdd failed

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1982902

Launchpad details

affected_projects = ['ubuntu']
assignee = mahdi3742
assignee_name = MehdiAmirirad
date_closed = None
date_created = 2022-07-27T01:38:08.171504+00:00
date_fix_committed = 2022-07-28T23:38:55.516212+00:00
date_fix_released = 2022-07-28T23:38:55.516212+00:00
id = 1982902
importance = undecided
is_complete = False
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1982902
milestone = None
owner = jshen28
owner_name = norman shen
private = False
status = incomplete
submitter = jshen28
submitter_name = norman shen
tags = []
duplicates = []

Launchpad user norman shen(jshen28) wrote on 2022-07-27T01:38:08.171504+00:00

Hello,

I am using cloud-init version: /usr/bin/cloud-init 20.4.1-0ubuntu118.04.1, ubuntu version is root@ubuntu:# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

I found that umount configdrive fails with device busy reported, it further causes temp folder failed to be deleted.

Logs are

2022-07-25 02:13:01,732 - handlers.py[DEBUG]: finish: init-local/search-ConfigDrive: FAIL: no local data found from DataSourceConfigDrive
2022-07-25 02:13:01,733 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceConfigDrive.DataSourceConfigDrive'> failed
2022-07-25 02:13:01,733 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceConfigDrive.DataSourceConfigDrive'> failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/temp_utils.py", line 90, in tempdir
    yield tdir
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1687, in mount_cb
    return ret
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1571, in unmounter
    subp.subp(umount_cmd)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
    cmd=args)
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['umount', '/run/cloud-init/tmp/tmpl5n7csdd']
Exit code: 32
Reason: -
Stdout:
Stderr: umount: /run/cloud-init/tmp/tmpl5n7csdd: target is busy.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 771, in find_source
    if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 660, in update_metadata
    result = self.get_data()
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 279, in get_data
    return_value = self._get_data()
  File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceConfigDrive.py", line 81, in _get_data
    mtype=mtype)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1687, in mount_cb
    return ret
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/cloudinit/temp_utils.py", line 92, in tempdir
    shutil.rmtree(tdir, ignore_errors=rmtree_ignore_errors)
  File "/usr/lib/python3.6/shutil.py", line 486, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 444, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 442, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
OSError: [Errno 30] Read-only file system: 'network_data.json'
2022-07-25 02:13:01,783 - main.py[DEBUG]: No local datasource found

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2022-07-28T23:38:35.097996+00:00

Thanks for filing a bug and helping make cloud-init better.

I'm having a hard time with these logs deterimining what may be contributing to the configdrive device busy. I suppose we could add a bit of retry logic on device busy errors here, but I'm not sure whether this is due to the underlying system being under high load or some other symptom that leads to this.

Would it be possible to run cloud-init collect-logs and attach the resulting tarfile of logs/system configuration to this bug to help us triage.
Note if providing
collect-logs --include-userdata that you may want to inspect the userdata included to make sure no secrets or passwords are included in the tarfile.

Much thanks.

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2022-07-28T23:39:12.762277+00:00

I'll set this bug to incomplete status, please reset it to new when you are able to provide the complete /var/log/cloud-init.log.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2022-07-29T02:36:12.634597+00:00

Status changed to 'Confirmed' because the bug affects multiple users.

[ubuntu-server-builder]

Launchpad user norman shen(jshen28) wrote on 2022-07-29T04:30:10.058289+00:00

thank you for reply,

I am sorry but the virtual machine has been deleted and I current found no way to reproduce it..

Best,
Norman

[ubuntu-server-builder]

Launchpad user Brett Holman(holmanb) wrote on 2022-08-01T14:13:29.323392+00:00

Cloud-init assumes it can mount/unmount the config drive filesystem. This kind of failure could easily be caused by external processes to cloud-init that don't realize they are interfering by accessing files on this filesystem.

/usr/bin/cloud-init 20.4.1-0ubuntu1~18.04.1

FYI: This reported release is over a year old and contains bugs that have been fixed in recent releases on Bionic.