Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cobbler get-loaders fails on Centos 7 new install #2660

Closed
AlbertBraden opened this issue Jun 7, 2021 · 23 comments
Closed

cobbler get-loaders fails on Centos 7 new install #2660

AlbertBraden opened this issue Jun 7, 2021 · 23 comments
Labels
2.8.5 Release 2.8.5 Bug Report Reporting a bug Deprecated Deprecated feature which will be removed in future versions.
Milestone

Comments

@AlbertBraden
Copy link

AlbertBraden commented Jun 7, 2021

Describe the bug

cobbler get-loaders fails with 404 error on Centos 7 new install

Steps to reproduce

  1. Install Centos 7 from https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2
  2. Install cobbler and its requirements:
yum install -y epel-release
yum upgrade -y
yum install -y net-tools bind-utils pykickstart syslinux fence-agents rsync debmirror dhcp wget
yum install -y python2-pip httpd cobbler cobbler-web xinetd vim
  1. Configure and start cobbler
  2. Run get-loaders:
[root@area51-os-cobbler-01 ~]# cobbler get-loaders
task started: 2021-06-07_202830_get_loaders
task started (id=Download Bootloader Content, time=Mon Jun  7 20:28:30 2021)
path /var/lib/cobbler/loaders/README already exists, not overwriting existing content, use --force if you wish to update
downloading https://cobbler.github.io/loaders/COPYING.elilo to /var/lib/cobbler/loaders/COPYING.elilo
Exception occured: <class 'urlgrabber.grabber.URLGrabError'>
Exception value: [Errno 14] HTTPS Error 404 - Not Found
Exception Info:
  File "/usr/lib/python2.7/site-packages/cobbler/remote.py", line 82, in run
    rc = self._run(self)
   File "/usr/lib/python2.7/site-packages/cobbler/remote.py", line 176, in runner
    return self.remote.api.dlcontent(self.options.get("force",False), self.logger)
   File "/usr/lib/python2.7/site-packages/cobbler/api.py", line 735, in dlcontent
    return grabber.run(force)
   File "/usr/lib/python2.7/site-packages/cobbler/action_dlcontent.py", line 73, in run
    urlgrabber.grabber.urlgrab(src, filename=dst, proxies=proxies)
   File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 773, in urlgrab
    return default_grabber.urlgrab(url, filename, **kwargs)
   File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1220, in urlgrab
    return _run_callback(opts.failfunc, opts)
   File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1071, in _run_callback
    return cb(obj)
   File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1065, in _do_raise
    raise obj.exception

!!! TASK FAILED !!!

Expected behavior

get-loaders completes successfully

Cobbler version

[root@area51-os-cobbler-01 ~]# cobbler version
Cobbler 2.8.5
  source: ?, ?
  build time: Tue Oct 15 01:59:43 2019

Operating system

CentOS Linux release 7.9.2009 (Core)

Cobbler log

Mon Jun  7 20:42:33 2021 - DEBUG | REMOTE CLI Authorized; user(?)
Mon Jun  7 20:42:33 2021 - INFO | REMOTE start_task(Download Bootloader Content); event_id(2021-06-07_204233_get_loaders); user(?)
Mon Jun  7 20:42:34 2021 - INFO | dlcontent

Screenshots

Additional information

It looks like https://cobbler.github.io/loaders/COPYING.elilo doesn't exist.

@AlbertBraden AlbertBraden added the Bug Report Reporting a bug label Jun 7, 2021
@AlbertBraden
Copy link
Author

This seems pretty new; I installed cobbler on centos 7 a few weeks ago and did not encounter it.

@nodeg nodeg added the 2.8.5 Release 2.8.5 label Jun 8, 2021
@nodeg nodeg added this to Inbox in Backlog via automation Jun 8, 2021
@nodeg
Copy link
Member

nodeg commented Jun 8, 2021

We removed the get-loaders support with #2572 but this should not affect you since you are still on version 2.8.5. But together with this the bootloaders from the Cobbler website were removed, too. See cobbler/cobbler.github.io@3d9d3c3
@SchoolGuy Any suggestions on how we should proceed here since version 2.x is EOL?

@nodeg nodeg added the Deprecated Deprecated feature which will be removed in future versions. label Jun 8, 2021
@SchoolGuy
Copy link
Member

@AlbertBraden The bootloaders of our website should have never been used in the first place. They were not updated for years and since I cannot guarantee for the content of it, I have taken them down. As said in other discussions and issues, I am expecting people to get them for themselves from the distros for numerous reasons. I am very sorry but updating, signing, as well es publishing the bootloaders is a task for more then a whole team, thus we cannot take care of this, even if it makes your life more complicated.

@AlbertBraden
Copy link
Author

Is there a document that explains how to manually update the bootloaders?

@SchoolGuy
Copy link
Member

@AlbertBraden They should be by the distro you want to install at least. And then you can use the normal workflows of Cobbler.

@AlbertBraden
Copy link
Author

AlbertBraden commented Jun 8, 2021

After googling around it appeared that I needed syslinux.efi so I did this:

mkdir -p /usr/local/src/loaders/
cd /usr/local/src/loaders/
wget https://mirrors.edge.kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.zip
unzip syslinux-6.03.zip
cp efi64/com32/elflink/ldlinux/ldlinux.e64 /var/lib/tftpboot/
cp efi64/efi/syslinux.efi /var/lib/tftpboot/

Then I edited dhcp.template and changed the filename to syslinux.efi, and ran cobbler sync, but the pxe-boot is hanging:

syslinux

What am I doing wrong?

@dansou901
Copy link

If I remember correctly version 6.0.3 has a bug. For efi, you should be using grub.

@AlbertBraden
Copy link
Author

I searched on my Cobbler host and found /var/www/cobbler/ks_mirror/centos7-minimal-1908-x86_64/EFI/BOOT/grubx64.efi so I copied that into /var/lib/tftpboot/ along with grub.cfg, and changed the filename in dhcp.template, and that gives me a nice menu:

menu

But it fails when I select an option:

FileNotFound

What am I missing?

@SchoolGuy
Copy link
Member

@AlbertBraden Exactly what it tells you: It can't find the kernel where it expectes the kernel to be. That can either be due to a misconfiguration, a missing cobbler sync or due to a timing problem when your environment is to large and Cobbler has not yet completed the sync it is currently doing.

@AlbertBraden
Copy link
Author

Until recently, I would install cobbler and run get-loaders and it would just work. Now that running get-loaders has been replaced with a complex and difficult process that is new to Cobbler users, can we get a document to help us download and install the correct loader and menu?

@SchoolGuy
Copy link
Member

@AlbertBraden Yes you may. Actually the (easy) process is already documented: https://cobbler.readthedocs.io/en/latest/quickstart-guide.html#importing-your-first-distribution

The more complex/advanced one is distributed over the whole docs but I am working on improving the docs...

@SchoolGuy
Copy link
Member

Keep in mind that this is for the master branch and it might not work always.

For the 2.8.5 release you may have a look here: https://cobbler.readthedocs.io/en/release28/3_general/cobbler%20direct%20command.html#import

@AlbertBraden
Copy link
Author

AlbertBraden commented Jun 10, 2021

I read those two documents and I see the discussion about importing, but importing the distribution is a separate step from setting up the loader and menu, and I'm not seeing the loader/menu steps in there. I ended up spending most of the day experimenting with grub.cfg, and I ended up with this:

[root@area51-os-cobbler-01]# cat /var/lib/tftpboot/grub.cfg

set default="0"

function load_video {
  insmod efi_gop
  insmod efi_uga
  insmod video_bochs
  insmod video_cirrus
  insmod all_video
}

load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2

set timeout=10

search --no-floppy --set=root -l 'CentOS 7 x86_64'

menuentry 'Install CentOS 7' --class fedora --class gnu-linux --class gnu --class os {
        linuxefi /images/centos7-minimal-1908-x86_64/vmlinuz ksdevice=bootif lang=  kssendmac text  ks=http://192.168.100.2/cblr/svc/op/ks/profile/centos7-minimal-1908-x86_64
        initrdefi /images/centos7-minimal-1908-x86_64/initrd.img
}

This works for pxe-booting, but I am seeing a bunch of errors in /var/log/messages on the cobbler server. Do I still have something misconfigured?

Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23167]: Client 192.168.100.14 finished grubx64.efi
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23168]: RRQ from 192.168.100.16 filename /grub.cfg-01-e4-43-4b-c0-ed-10
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23168]: Client 192.168.100.16 File not found /grub.cfg-01-e4-43-4b-c0-ed-10
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23169]: RRQ from 192.168.100.16 filename /grub.cfg-C0A86410
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23169]: Client 192.168.100.16 File not found /grub.cfg-C0A86410
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23170]: RRQ from 192.168.100.16 filename /grub.cfg-C0A8641
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23170]: Client 192.168.100.16 File not found /grub.cfg-C0A8641
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23171]: RRQ from 192.168.100.16 filename /grub.cfg-C0A864
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23171]: Client 192.168.100.16 File not found /grub.cfg-C0A864
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23172]: RRQ from 192.168.100.16 filename /grub.cfg-C0A86
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23172]: Client 192.168.100.16 File not found /grub.cfg-C0A86
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23173]: RRQ from 192.168.100.16 filename /grub.cfg-C0A8
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23173]: Client 192.168.100.16 File not found /grub.cfg-C0A8
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23174]: RRQ from 192.168.100.16 filename /grub.cfg-C0A
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23174]: Client 192.168.100.16 File not found /grub.cfg-C0A
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23175]: RRQ from 192.168.100.16 filename /grub.cfg-C0
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23175]: Client 192.168.100.16 File not found /grub.cfg-C0
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23176]: RRQ from 192.168.100.16 filename /grub.cfg-C
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23176]: Client 192.168.100.16 File not found /grub.cfg-C
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23177]: RRQ from 192.168.100.16 filename /grub.cfg
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23177]: Client 192.168.100.16 finished /grub.cfg
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23178]: RRQ from 192.168.100.16 filename /EFI/BOOT/x86_64-efi/command.lst
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23178]: Client 192.168.100.16 File not found /EFI/BOOT/x86_64-efi/command.lst
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23179]: RRQ from 192.168.100.16 filename /EFI/BOOT/x86_64-efi/fs.lst
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23179]: Client 192.168.100.16 File not found /EFI/BOOT/x86_64-efi/fs.lst
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23180]: RRQ from 192.168.100.16 filename /EFI/BOOT/x86_64-efi/crypto.lst
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23180]: Client 192.168.100.16 File not found /EFI/BOOT/x86_64-efi/crypto.lst
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23181]: RRQ from 192.168.100.16 filename /EFI/BOOT/x86_64-efi/terminal.lst
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23181]: Client 192.168.100.16 File not found /EFI/BOOT/x86_64-efi/terminal.lst
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23182]: RRQ from 192.168.100.16 filename /grub.cfg
Jun 10 02:40:03 area51-os-cobbler-01 in.tftpd[23182]: Client 192.168.100.16 finished /grub.cfg

@SchoolGuy
Copy link
Member

@AlbertBraden Yes these files are normally there if Cobbler is the one generating the TFTP directory and the grub.cfg for you. If you want to UEFI Boot your machines take a look at the mkgrub.sh script. Documentation for it can be found here: https://cobbler.readthedocs.io/en/latest/scripts.html#mkgrub-sh

The script is not deployed with Cobbler 2.8.5 but works perfectly well with it since it is independent of Cobbler. Just grab it from the master branch. Remember that your distro needs to support the noarch packages for Grub, so you can install the ppcle64 modules onto an x86_64 machine. Normally that should be the case since you need them not only for Cobbler but also to boot VMs with other archs on your machine.

@SchoolGuy
Copy link
Member

@dansou901 @hbokh Please help me out here if you have a bit of time. Maybe I am seeing this from the wrong standpoint, you have more experience as users as I have...

@hbokh
Copy link
Contributor

hbokh commented Jun 11, 2021

Interesting! This is going to bite us too in time, on our now +1 year old Cobbler 3.1.2 on Debian Buster (not CentOS7).

FWIW, we have these loaders in /var/lib/cobbler/loaders/:

$ tree /var/lib/cobbler/loaders/
/var/lib/cobbler/loaders/
├── COPYING.syslinux
├── COPYING.yaboot
├── grub
│   ├── grub.0
│   ├── grub.cfg
│   ├── grubx64.efi
│   └── x86_64-efi -> /usr/lib/grub/x86_64-efi
├── grub-x86_64.efi
├── grub-x86.efi
├── menu.c32
├── pxelinux.0
├── README
└── yaboot

2 directories, 11 files

On my even older Cobbler 3.1.2 at $HOME, running cobbler get-loaders gives this output:

# cobbler get-loaders
task started: 2021-06-11_123903_get_loaders
task started (id=Download Bootloader Content, time=Fri Jun 11 12:39:03 2021)
running python triggers from /var/lib/cobbler/triggers/task/get_loaders/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/get_loaders/pre/*
shell triggers finished successfully
path /var/lib/cobbler/loaders/README already exists, not overwriting existing content, use --force if you wish to update
path /var/lib/cobbler/loaders/COPYING.yaboot already exists, not overwriting existing content, use --force if you wish to update
path /var/lib/cobbler/loaders/COPYING.syslinux already exists, not overwriting existing content, use --force if you wish to update
path /var/lib/cobbler/loaders/yaboot already exists, not overwriting existing content, use --force if you wish to update
path /var/lib/cobbler/loaders/pxelinux.0 already exists, not overwriting existing content, use --force if you wish to update
path /var/lib/cobbler/loaders/menu.c32 already exists, not overwriting existing content, use --force if you wish to update
path /var/lib/cobbler/loaders/grub-x86.efi already exists, not overwriting existing content, use --force if you wish to update
path /var/lib/cobbler/loaders/grub-x86_64.efi already exists, not overwriting existing content, use --force if you wish to update
*** TASK COMPLETE ***

This might come in handy for @AlbertBraden to at least get some loaders in the right place.

In the mean time I'm having a hard time setting up Cobbler 2.8.5 on CentOS7 in a VM again.
If I find anything useful I'll get back here, but since this version of Cobbler is pretty dated and EOL I'm not going to run for it TBH.

@AlbertBraden
Copy link
Author

This is a completely terrible experience for cobbler operators, and the timing is particularly bad as we are scrambling to build Centos 7 dev environments to test the migration to Centos 8, and then from Centos 8 to something else before the end of the year. If you could just make 2.8.5 work for a few more months (without the operator spending many hours hacking loaders and menus), that would be a huge service to the operators who are using your product.

@hbokh
Copy link
Contributor

hbokh commented Jun 11, 2021

cobbler check on CentOS7 and Cobbler 2.8.5 gives this informative output:

Some network boot-loaders are missing from /var/lib/cobbler/loaders, you may run 'cobbler get-loaders' to download them, or, if you only want to handle x86/x86_64 netbooting, you may ensure that you have installed a recent version of the syslinux package installed and can ignore this message entirely. Files in this directory, should you want to support all architectures, should include pxelinux.0, menu.c32, elilo.efi, and yaboot. The 'cobbler get-loaders' command is the easiest way to resolve these requirements.

I wonder if anyone still needs yaboot and / or elilo.efi.
So far I have a gut feeling the cobbler get-loaders is not the cause anymore of the problem here, but (U)EFI / Grub booting.

@AlbertBraden
Copy link
Author

Yes, UEFI is the issue, but UEFI works "out of the box" after running get-loaders. Without get-loaders it is a mess.

@hbokh
Copy link
Contributor

hbokh commented Jun 11, 2021

I created an "archive" with the old loader files. I hope this can clean up your mess a bit.

https://github.com/hbokh/cobbler-loaders

@SchoolGuy
Copy link
Member

@opoplawski Could you create a patch which removes get-loaders from any version available in the RH Ecosystem please?

@opoplawski
Copy link
Contributor

Done in 3.2.2-5 (Fedora, EPEL8, copr orion/cobbler) and 2.8.5-2 (EPEL7).

@SchoolGuy
Copy link
Member

This will not be fixed and the reasons are outlined in detail here. No activity for a while. Closing issue.

@SchoolGuy SchoolGuy closed this as not planned Won't fix, can't repro, duplicate, stale Mar 4, 2023
@SchoolGuy SchoolGuy added this to the v2.8.x milestone Mar 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.8.5 Release 2.8.5 Bug Report Reporting a bug Deprecated Deprecated feature which will be removed in future versions.
Projects
Status: Done
Development

No branches or pull requests

6 participants