Make the lorax-composer ks templates more generic #603

bcl · 2019-02-13T01:12:52Z

Some platforms do not have grub2, and some require other partitions.

Anaconda will add platform specific partitions if the 'reqpart' command
is included, and it will add bootloader specific packages to the list if
they are needed.

--- Merge policy ---

Travis CI PASS
*-aws-runtest PASS
*-azure-runtest PASS
*-images-runtest PASS
*-openstack-runtest PASS
*-vmware-runtest PASS
For rhel8-* and rhel7-* branches commit log references an approved
bug in Bugzilla. Do not merge if the bug doesn't have the 3 ACKs set to +!

--- Jenkins commands ---

ok to test to accept this pull request for testing
test this please for a one time test run
retest this please to start a new build

bcl · 2019-02-13T01:14:30Z

@atodorov I think the tricky problem here is how to handle the iso template. The extra packages are actually required by the lorax template used to post-process the filesystem image so anaconda can't be expected to know what to install. I may have more ideas tomorrow.

atodorov · 2019-02-13T12:16:25Z

I tried the qcow2.ks changes on a Power system (rhel8-branch) and I got:

============================

04:40:43,610 DBG exception: running handleException
04:40:43,611 CRT exception: Traceback (most recent call last):

  File "/usr/lib/python3.6/site-packages/simpleline/render/screen_scheduler.py", line 245, in _process_screen
    top_screen.ui_screen.get_input_with_error_check(top_screen.args)

  File "/usr/lib/python3.6/site-packages/simpleline/render/screen/__init__.py", line 285, in get_input_with_error_check
    self._input_manager.get_input(args=args)

  File "/usr/lib/python3.6/site-packages/simpleline/render/screen/input_manager.py", line 91, in get_input
    prompt = self._ui_screen.prompt(args)

  File "/usr/lib64/python3.6/site-packages/pyanaconda/ui/tui/hubs/summary.py", line 119, in prompt
    raise CmdlineError(errtxt)

pyanaconda.errors.CmdlineError: The following mandatory spokes are not completed:
Installation Destination

atodorov · 2019-02-19T14:18:05Z

2019-02-19 09:02:19,618: Rebuilding initrds
2019-02-19 09:02:19,618: dracut args = ['--xz', '--add', 'livenet dmsquash-live convertfs pollcdrom qemu qemu-net', '--omit', 'plymouth', '--no-hostonly', '--debug', '--no-early-microcode']
2019-02-19 09:02:19,619: rebuilding boot/initramfs-4.18.0-67.el8.ppc64le.img
2019-02-19 09:02:55,231: Building boot.iso
2019-02-19 09:02:55,803: traceback: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/mako/lookup.py", line 247, in get_template
    return self._check(uri, self._collection[uri])
KeyError: 'ppc64le.tmpl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/lorax/src/pylorax/api/queue.py", line 143, in monitor
    make_compose(cfg, os.path.realpath(dst))
  File "/root/lorax/src/pylorax/api/queue.py", line 248, in make_compose
    run_creator(install_cfg, cancel_func=cancel_build)
  File "/root/lorax/src/pylorax/creator.py", line 701, in run_creator
    result_dir = make_livecd(opts, mount_dir, work_dir)
  File "/root/lorax/src/pylorax/creator.py", line 369, in make_livecd
    tb.build()
  File "/root/lorax/src/pylorax/treebuilder.py", line 313, in build
    self._runner.run(templatefile, kernels=self.kernels)
  File "/root/lorax/src/pylorax/ltmpl.py", line 141, in run
    commands = t.parse(templatefile, variables)
  File "/root/lorax/src/pylorax/ltmpl.py", line 53, in parse
    template = lookup.get_template(template_file)
  File "/usr/lib/python3.6/site-packages/mako/lookup.py", line 261, in get_template
    "Cant locate template for uri %r" % uri)
mako.exceptions.TopLevelLookupException: Cant locate template for uri 'ppc64le.tmpl'

# ps aux | grep lorax-composer
root     138256  3.2  0.0 288704 182464 pts/0   S    09:09   0:05 /usr/bin/python3 ./src/sbin/lorax-composer --sharedir /tmp/composer-share.dmSfj /tmp/composer-blueprints.piNfI
root     138258  0.2  0.0 196096 57216 pts/0    Sl   09:09   0:00 /usr/bin/python3 ./src/sbin/lorax-composer --sharedir /tmp/composer-share.dmSfj /tmp/composer-blueprints.piNfI
root     138623  0.0  0.0   4544  1664 pts/1    S+   09:12   0:00 grep --color=auto lorax-composer

# find /tmp/composer-share.dmSfj -name "ppc64le.tmpl"
/tmp/composer-share.dmSfj/templates.d/99-generic/ppc64le.tmpl

So this file is obviously there.

NOTE: we still don't have a test for live-iso in Jenkins, this is on the list of tasks to do this/next week.

atodorov · 2019-02-19T15:09:33Z

with the latest patch set I tested on ppc64le and the result is as follows:

almost all compose types are able to be built
live-iso fails with the error above
vmdk fails with open-vm-tools-* (No match) - I guess this is arch specific
for most of the compose types we either don't have test scenarios(ext4) or no vendors providing access to infra
I was able to boot the qcow2 image in KVM
I tried importing the tar image in podman but the podman package for ppc64le can't be installed (see RHBZ 1678773)

I will test s390x and aarch64 later today.

bcl · 2019-02-19T21:13:49Z

/tmp/composer-share.dmSfj/templates.d/99-generic/ppc64le.tmpl

Ah! So it looks like we need to make a ./live/ppc64le.tmpl

bcl · 2019-02-19T21:55:36Z

@atodorov Give that template a try, if it fails I'll need access to the machine for debugging.

atodorov · 2019-02-20T09:45:06Z

Note: xorriso is not added as dependency to lorax.spec for rhel8-branch so I had to install it manually.

After that I got the following error

2019-02-20 04:55:28,898: template command error in ppc64le.tmpl:
2019-02-20 04:55:28,898:   runcmd xorrisofs -v -U -J -R -o /var/tmp/lmc-work-mc7irc_p/images/boot.iso -r -l -sysid PPC -A Red Hat Enterprise Linux 8.0 -V Red_Hat_Enterprise_Linux_8_0_ppc -volset 8.0 -volset-size 1 -volset-seqno 1 -chrp-boot -graft-points .discinfo=/var/tmp/lmc-work-mc7irc_p/.discinfo ppc=/var/tmp/lmc-work-mc7irc_p/ppc boot/grub=/var/tmp/lmc-work-mc7irc_p/boot/grub LiveOS=/var/tmp/lmc-work-mc7irc_p/LiveOS
2019-02-20 04:55:28,899:   subprocess.CalledProcessError: Command '['xorrisofs', '-v', '-U', '-J', '-R', '-o', '/var/tmp/lmc-work-mc7irc_p/images/boot.iso', '-r', '-l', '-sysid', 'PPC', '-A', 'Red Hat Enterprise Linux 8.0', '-V', 'Red_Hat_Enterprise_Linux_8_0_ppc', '-volset', '8.0', '-volset-size', '1', '-volset-seqno', '1', '-chrp-boot', '-graft-points', '.discinfo=/var/tmp/lmc-work-mc7irc_p/.discinfo', 'ppc=/var/tmp/lmc-work-mc7irc_p/ppc', 'boot/grub=/var/tmp/lmc-work-mc7irc_p/boot/grub', 'LiveOS=/var/tmp/lmc-work-mc7irc_p/LiveOS']' returned non-zero exit status 5.
2019-02-20 04:55:29,430: traceback: Traceback (most recent call last):
  File "/root/lorax/src/pylorax/api/queue.py", line 143, in monitor
    make_compose(cfg, os.path.realpath(dst))
  File "/root/lorax/src/pylorax/api/queue.py", line 248, in make_compose
    run_creator(install_cfg, cancel_func=cancel_build)
  File "/root/lorax/src/pylorax/creator.py", line 701, in run_creator
    result_dir = make_livecd(opts, mount_dir, work_dir)
  File "/root/lorax/src/pylorax/creator.py", line 369, in make_livecd
    tb.build()
  File "/root/lorax/src/pylorax/treebuilder.py", line 313, in build
    self._runner.run(templatefile, kernels=self.kernels)
  File "/root/lorax/src/pylorax/ltmpl.py", line 142, in run
    self._run(commands)
  File "/root/lorax/src/pylorax/ltmpl.py", line 161, in _run
    f(*args)
  File "/root/lorax/src/pylorax/ltmpl.py", line 521, in runcmd
    stdout = runcmd_output(cmd)
  File "/root/lorax/src/pylorax/executils.py", line 349, in runcmd_output
    return execWithCapture(cmd[0], cmd[1:], **kwargs)
  File "/root/lorax/src/pylorax/executils.py", line 251, in execWithCapture
    reset_handlers=reset_handlers, reset_lang=reset_lang)[1]
  File "/root/lorax/src/pylorax/executils.py", line 203, in _run_program
    raise subprocess.CalledProcessError(proc.returncode, argv, output)
subprocess.CalledProcessError: Command '['xorrisofs', '-v', '-U', '-J', '-R', '-o', '/var/tmp/lmc-work-mc7irc_p/images/boot.iso', '-r', '-l', '-sysid', 'PPC', '-A', 'Red Hat Enterprise Linux 8.0', '-V', 'Red_Hat_Enterprise_Linux_8_0_ppc', '-volset', '8.0', '-volset-size', '1', '-volset-seqno', '1', '-chrp-boot', '-graft-points', '.discinfo=/var/tmp/lmc-work-mc7irc_p/.discinfo', 'ppc=/var/tmp/lmc-work-mc7irc_p/ppc', 'boot/grub=/var/tmp/lmc-work-mc7irc_p/boot/grub', 'LiveOS=/var/tmp/lmc-work-mc7irc_p/LiveOS']' returned non-zero exit status 5.

bcl · 2019-02-20T16:52:05Z

Note: xorriso is not added as dependency to lorax.spec for rhel8-branch so I had to install it manually.

Actually the right thing to do for rhel8 is change it to match the non-live ppc64le.tmpl, we're still using mkisofs. I'll also work up a backport of this specific to rhel8 at some point today.

Some platforms do not have grub2, and some require other partitions. Anaconda will add platform specific partitions if the 'reqpart' command is included, and it will add bootloader specific packages to the list if they are needed.

reqpart can be used to make kickstarts more platform agnostic, creating needed partitions without lmc having to keep track of the arch-specific needs. eg. ppc64 needs prepboot and /boot This increases the size of the disk based on whether reqpart or reqpart --add-boot is in the kickstart. Note that this is only valid for partitioned disk output types, not for filesystem images or live iso output.

Some platforms, like ppc64, require that the /boot partition be present. It doesn't hurt to have it there on other platforms so instead of trying to add per-arch kickstart templates just use reqpart --add-boot everywhere.

This will make it easier to add a new subclass that only handles installpkg for use with livemedia-creator and lorax-composer iso creation.

This template runner only uses the installpkg command to gather a list of package NEVRA to be installed to support live iso creation.

If a package is excluded in the template and later added by a blueprint or dependency, anaconda will fail to finish the installation. So remove the -dracut-config-rescue exclusion and instead remove the rescue artifacts in %post

In order to support iso creation on multiple arches with the templates we need to be able to select different packages based on arch. lorax-composer uses the arch-specific Lorax templates in order to generate the output iso so this patch: 1. Creates a new template and type to parse it, live-install.tmpl which contains only installpkg commands and #if clauses for arch 2. Removes bootloader related packages from the live-iso.ks 3. Remove dracut-config-rescue exclusion because it can cause problems with some blueprints. 4. Switch logo requirement to system-logos which is satisfied by generic-logos or fedora-logos. This prevents conflicts when a blueprint installs fedora-release-workstation. So in the future, if x86.tmpl, etc. need a new package to support creating the iso it should be added to the correct section in ./share/live/live-install.tmpl

bcl mentioned this pull request Feb 20, 2019

Add support for lorax-composer on other arches #616

Merged

7 tasks

bcl force-pushed the master-simpler-ks branch 2 times, most recently from 116d73b to 0246d99 Compare February 21, 2019 18:22

bcl added 8 commits February 25, 2019 13:49

Make the lorax-composer ks templates more generic

d27b4fc

Some platforms do not have grub2, and some require other partitions. Anaconda will add platform specific partitions if the 'reqpart' command is included, and it will add bootloader specific packages to the list if they are needed.

Move the run part of LoraxTemplateRunner into new TemplateRunner class

8eaad3b

This will make it easier to add a new subclass that only handles installpkg for use with livemedia-creator and lorax-composer iso creation.

Add LiveTemplateRunner to parse per-arch live-iso package requirements

6c5e89e

This template runner only uses the installpkg command to gather a list of package NEVRA to be installed to support live iso creation.

Remove exclusions from lorax-composer templates

822bc7a

If a package is excluded in the template and later added by a blueprint or dependency, anaconda will fail to finish the installation. So remove the -dracut-config-rescue exclusion and instead remove the rescue artifacts in %post

Add a ppc64le template for live iso creation

f3c1474

bcl force-pushed the master-simpler-ks branch from 0246d99 to f3c1474 Compare February 25, 2019 21:49

bcl merged commit f3c1474 into weldr:master Feb 25, 2019

Make the lorax-composer ks templates more generic #603

Make the lorax-composer ks templates more generic #603

bcl commented Feb 13, 2019 •

edited by atodorov

bcl commented Feb 13, 2019

atodorov commented Feb 13, 2019 •

edited

atodorov commented Feb 19, 2019 •

edited

atodorov commented Feb 19, 2019

bcl commented Feb 19, 2019

bcl commented Feb 19, 2019

atodorov commented Feb 20, 2019 •

edited

bcl commented Feb 20, 2019

Make the lorax-composer ks templates more generic #603

Make the lorax-composer ks templates more generic #603

Conversation

bcl commented Feb 13, 2019 • edited by atodorov

bcl commented Feb 13, 2019

atodorov commented Feb 13, 2019 • edited

atodorov commented Feb 19, 2019 • edited

atodorov commented Feb 19, 2019

bcl commented Feb 19, 2019

bcl commented Feb 19, 2019

atodorov commented Feb 20, 2019 • edited

bcl commented Feb 20, 2019

bcl commented Feb 13, 2019 •

edited by atodorov

atodorov commented Feb 13, 2019 •

edited

atodorov commented Feb 19, 2019 •

edited

atodorov commented Feb 20, 2019 •

edited