Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

virt.init fails to seed under certain conditions #55348

Open
tash opened this issue Nov 18, 2019 · 2 comments
Open

virt.init fails to seed under certain conditions #55348

tash opened this issue Nov 18, 2019 · 2 comments
Labels
Bug broken, incorrect, or confusing behavior Confirmed Salt engineer has confirmed bug/feature - often including a MCVE severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around
Projects
[Test] Triage
Milestone
Chlorine v3007.0

Comments

@tash
Copy link

tash commented Nov 18, 2019 •
edited

Description of Issue

When creating a VM with virt.init, the seeding fails on the given system:

salt-call -l debug virt.init gitlab 2 4096 start=False image=/central/vm/test/openSUSE-Leap-15.1-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot9.114.qcow2

Logging:

[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/testdefault.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/testdefault.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: test02
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/testdefault.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/testdefault.conf
[DEBUG   ] Connecting to master. Attempt 1 of 1
[DEBUG   ] "salt-master" Not an IP address? Assuming it is a hostname.
[DEBUG   ] Master URI: tcp://10.10.81.124:4506
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'test02', u'tcp://10.10.81.124:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (7065)
[DEBUG   ] Setting zmq_reconnect_ivl to '7065ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'test02', u'tcp://10.10.81.124:4506', 'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://10.10.81.124:4506
[DEBUG   ] Trying to connect to: tcp://10.10.81.124:4506
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] salt.crypt._get_key_with_evict: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] Connecting the Minion to the Master publish port, using the URI: tcp://10.10.81.124:4505
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'test02', u'tcp://10.10.81.124:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'test02', u'tcp://10.10.81.124:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://10.10.81.124:4506
[DEBUG   ] Trying to connect to: tcp://10.10.81.124:4506
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'test02', u'tcp://10.10.81.124:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'test02', u'tcp://10.10.81.124:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://10.10.81.124:4506
[DEBUG   ] Trying to connect to: tcp://10.10.81.124:4506
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded virt.init
[DEBUG   ] LazyLoaded config.get
[DEBUG   ] Using hyperisor kvm
[DEBUG   ] NIC profile is [{u'mac': u'52:54:00:9F:BB:6D', u'source': u'virbr0', u'model': u'virtio', u'type': u'bridge', u'name': u'eth0'}]
[DEBUG   ] /central/vm/test/openSUSE-Leap-15.1-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot9.114.qcow2 image from module arguments will be used for disk "system" instead of None
[DEBUG   ] Creating disk for VM [ gitlab ]: {u'system': {u'model': u'virtio', u'format': u'qcow2', u'image': u'/central/vm/test/openSUSE-Leap-15.1-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot9.114.qcow2', u'pool': u'/nas-01/images', u'siz$
': u'8192'}}
[DEBUG   ] Image directory from config option `virt.images` is /nas-01/images
[DEBUG   ] Image destination will be /nas-01/images/gitlab/system.qcow2
[DEBUG   ] Image destination directory is /nas-01/images/gitlab
[DEBUG   ] Create disk from specified image /central/vm/test/openSUSE-Leap-15.1-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot9.114.qcow2
[DEBUG   ] LazyLoaded cp.cache_file
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'test02', u'tcp://10.10.81.124:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'test02', u'tcp://10.10.81.124:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://10.10.81.124:4506
[DEBUG   ] Trying to connect to: tcp://10.10.81.124:4506
[DEBUG   ] LazyLoaded cmd.run
[INFO    ] Executing command 'qemu-img info /central/vm/test/openSUSE-Leap-15.1-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot9.114.qcow2' in directory '/root'
[DEBUG   ] stdout: image: /central/vm/test/openSUSE-Leap-15.1-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot9.114.qcow2
file format: qcow2
virtual size: 24G (25769803776 bytes)
disk size: 231M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
[DEBUG   ] output: image: /central/vm/test/openSUSE-Leap-15.1-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot9.114.qcow2
file format: qcow2
virtual size: 24G (25769803776 bytes)
disk size: 231M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
[DEBUG   ] Copying /central/vm/test/openSUSE-Leap-15.1-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot9.114.qcow2 to /nas-01/images/gitlab/system.qcow2
[DEBUG   ] Resize qcow2 image to 8192M
[INFO    ] Executing command 'qemu-img resize /nas-01/images/gitlab/system.qcow2 8192M' in directory '/root'
[ERROR   ] Command '[u'qemu-img', u'resize', u'/nas-01/images/gitlab/system.qcow2', u'8192M']' failed with return code: 1
[ERROR   ] stdout: qemu-img: qcow2 doesn't support shrinking images yet
[ERROR   ] retcode: 1
[ERROR   ] Command 'qemu-img resize /nas-01/images/gitlab/system.qcow2 8192M' failed with return code: 1
[ERROR   ] output: qemu-img: qcow2 doesn't support shrinking images yet
[DEBUG   ] Apply umask and remove exec bit
[DEBUG   ] Seed command is seed.apply
[DEBUG   ] LazyLoaded seed.apply
[DEBUG   ] LazyLoaded file.stats
[DEBUG   ] Mounting file at /nas-01/images/gitlab/system.qcow2
[DEBUG   ] LazyLoaded guestfs.mount
[DEBUG   ] LazyLoaded mount.mount
[DEBUG   ]  ---> location: /nas-01/images/gitlab/system.qcow2
[DEBUG   ]  ---> root: None
[DEBUG   ] Using root /tmp/guest/nas-01.images.gitlab.system.qcow2
[DEBUG   ] Establishing new root as /tmp/guest/nas-01.images.gitlab.system.qcow22c7c8055ce08653ebb4e59f77263bc2e309f8273a02bcb7d383c809a4f609b40
[DEBUG   ]  ---> cmd: guestmount -i -a /nas-01/images/gitlab/system.qcow2 --rw /tmp/guest/nas-01.images.gitlab.system.qcow22c7c8055ce08653ebb4e59f77263bc2e309f8273a02bcb7d383c809a4f609b40
[INFO    ] Executing command 'guestmount -i -a /nas-01/images/gitlab/system.qcow2 --rw /tmp/guest/nas-01.images.gitlab.system.qcow22c7c8055ce08653ebb4e59f77263bc2e309f8273a02bcb7d383c809a4f609b40' in directory '/root'
[DEBUG   ] output:
[DEBUG   ] Attempting to create directory /tmp/guest/nas-01.images.gitlab.system.qcow22c7c8055ce08653ebb4e59f77263bc2e309f8273a02bcb7d383c809a4f609b40/tmp
[DEBUG   ] LazyLoaded pillar.ext
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'test02', u'tcp://10.10.81.124:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'test02', u'tcp://10.10.81.124:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://10.10.81.124:4506
[DEBUG   ] Trying to connect to: tcp://10.10.81.124:4506
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] salt-minion pre-installed on image, configuring as gitlab
[DEBUG   ] Reading configuration from /tmp/guest/nas-01.images.gitlab.system.qcow22c7c8055ce08653ebb4e59f77263bc2e309f8273a02bcb7d383c809a4f609b40/tmp/minion
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
OSError: [Errno 18] Invalid cross-device link
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 410, in salt_call
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 57, in run
    caller.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 212, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/virt.py", line 773, in init
    priv_key=priv_key,
  File "/usr/lib/python2.7/site-packages/salt/modules/seed.py", line 169, in apply_
    mpt, pki_dir.lstrip('/'), 'minion.pem'))
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 410, in salt_call
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 57, in run
    caller.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 212, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/virt.py", line 773, in init
    priv_key=priv_key,
  File "/usr/lib/python2.7/site-packages/salt/modules/seed.py", line 169, in apply_
    mpt, pki_dir.lstrip('/'), 'minion.pem'))
OSError: [Errno 18] Invalid cross-device link

Setup

libvirt-bash-completion-4.5.0-10.el7_6.2.x86_64
libvirt-client-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-nwfilter-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-lxc-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-storage-disk-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-storage-4.5.0-10.el7_6.2.x86_64
libvirt-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-kvm-4.5.0-10.el7_6.2.x86_64
libvirt-libs-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-network-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-config-nwfilter-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-qemu-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-storage-scsi-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-storage-logical-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-storage-iscsi-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-secret-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-nodedev-4.5.0-10.el7_6.2.x86_64
libvirt-glib-1.0.0-1.el7.x86_64
libvirt-python-4.5.0-1.el7.x86_64
libvirt-daemon-driver-storage-core-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-config-network-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-storage-mpath-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-storage-rbd-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-interface-4.5.0-10.el7_6.2.x86_64

qemu-kvm-common-ev-2.10.0-21.el7_5.7.1.x86_64
qemu-kvm-tools-ev-2.10.0-21.el7_5.7.1.x86_64
libvirt-daemon-kvm-4.5.0-10.el7_6.2.x86_64
libvirt-daemon-driver-qemu-4.5.0-10.el7_6.2.x86_64
qemu-kvm-ev-2.10.0-21.el7_5.7.1.x86_64
qemu-img-ev-2.10.0-21.el7_5.7.1.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch

libguestfs-1.38.2-12.el7.x86_64
libguestfs-tools-c-1.38.2-12.el7.x86_64
libguestfs-tools-1.38.2-12.el7.noarch
perl-Sys-Guestfs-1.38.2-12.el7.x86_64
libguestfs-bash-completion-1.38.2-12.el7.noarch

Steps to Reproduce Issue

Call is made locally on the Hypervisor (test02):
salt-call -l debug virt.init gitlab 2 4096 start=False image=/central/vm/test/openSUSE-Leap-15.1-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot9.114.qcow2

Versions Report

Salt Version:
           Salt: 2018.3.4

Dependency Versions:
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.5.1804 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.el7.x86_64
         system: Linux
        version: CentOS Linux 7.5.1804 Core
@tash
Copy link
Author

tash commented Nov 18, 2019 •
edited

modules/seed.py _check_install calls os.rename which won't work under certain circumstances when source and destination are on different filesystems:

salt/salt/modules/seed.py

Lines 168 to 173 in d0cad3e

os.rename(cfg_files['privkey'], os.path.join(
mpt, pki_dir.lstrip('/'), 'minion.pem'))
os.rename(cfg_files['pubkey'], os.path.join(
mpt, pki_dir.lstrip('/'), 'minion.pub'))
os.rename(cfg_files['config'], os.path.join(mpt, 'etc/salt/minion'))
res = True

A fix should be fairly easy:

        shutil.move(cfg_files['privkey'], os.path.join(
            mpt, pki_dir.lstrip('/'), 'minion.pem'))
        shutil.move(cfg_files['pubkey'], os.path.join(
            mpt, pki_dir.lstrip('/'), 'minion.pub'))
        shutil.move(cfg_files['config'], os.path.join(mpt, 'etc/salt/minion'))
        res = True

@waynew waynew added this to Needs triage in [Test] Triage Dec 12, 2019
@waynew waynew added Confirmed Salt engineer has confirmed bug/feature - often including a MCVE Bug broken, incorrect, or confusing behavior P4 Priority 4 labels Dec 12, 2019
@waynew waynew added this to the Approved milestone Dec 12, 2019
@waynew waynew moved this from Needs triage to Low priority in [Test] Triage Dec 12, 2019
@waynew
Copy link
Contributor

waynew commented Dec 12, 2019

@tash Thanks for the report!

Looks like a pretty reasonable fix, too. Are you interested in writing the tests/fix for this? If not, that's totally fine.

For whoever does address this:

There's some code already in tests/unit/modules/test_seed.py that could either be modified to apply to this case, or used as a starting point here. In an ideal world we could write functional tests that would actually test the movement, but I'm pretty sure our existing CI pipeline doesn't have multiple mounts to break with the cross-device link.

I think it would be reasonable to just mock os.rename and have it raise OSerror (just to make sure it's not called).

@sagetherage sagetherage removed the P4 Priority 4 label Jun 3, 2020
@sagetherage sagetherage added openSUSE severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around Phosphorus v3005.0 Release code name and version and removed openSUSE labels Jun 16, 2021
@sagetherage sagetherage modified the milestones: Approved, Phosphorus Jun 16, 2021
@Ch3LL Ch3LL removed the Phosphorus v3005.0 Release code name and version label Mar 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug broken, incorrect, or confusing behavior Confirmed Salt engineer has confirmed bug/feature - often including a MCVE severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around
Projects
No open projects
[Test] Triage
  
Low priority
Milestone
Chlorine v3007.0
Development

No branches or pull requests
4 participants
@waynew @tash @Ch3LL @sagetherage