cobbler get-loaders --force failing with Exception value: 'str' object has no attribute 'get'

[mohantnr]

Describe the bug

root@hdrkairosvm203:~# cobbler get-loaders --force
task started: 2021-03-08_083227_get_loaders
task started (id=Download Bootloader Content, time=Mon Mar  8 08:32:27 2021)
running python triggers from /var/lib/cobbler/triggers/task/get_loaders/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/get_loaders/pre/*
shell triggers finished successfully
downloading https://cobbler.github.io/loaders/README to /var/lib/cobbler/loaders/README
Exception occurred: <class 'AttributeError'>
Exception value: 'str' object has no attribute 'get'
Exception Info:
  File "/usr/local/lib/python3.8/dist-packages/cobbler/remote.py", line 95, in run
    rc = self._run(self)

  File "/usr/local/lib/python3.8/dist-packages/cobbler/remote.py", line 209, in runner
    self.remote.api.dlcontent(self.options.get("force", False), self.logger)

  File "/usr/local/lib/python3.8/dist-packages/cobbler/api.py", line 1290, in dlcontent
    return grabber.run(force)

  File "/usr/local/lib/python3.8/dist-packages/cobbler/actions/dlcontent.py", line 75, in run
    dlmgr.download_file(src, dst)

  File "/usr/local/lib/python3.8/dist-packages/cobbler/download_manager.py", line 71, in download_file
    response = requests.get(url, stream=True, proxies=proxies, cert=cert)

  File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)

  File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)

  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 523, in request
    settings = self.merge_environment_settings(

  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 699, in merge_environment_settings
    no_proxy = proxies.get('no_proxy') if proxies is not None else None

!!! TASK FAILED !!!

Steps to reproduce

1. cobbler get-loaders

Expected behavior

Cobbler version

Cobbler 3.2.1
  source: c7fc767c, Thu Mar 4 20:07:10 2021 +0100
  build time: Fri Mar  5 06:13:19 2021

Operating system

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

Cobbler log

root@hdrkairosvm203:~# cobbler version
Cobbler 3.2.1
  source: c7fc767c, Thu Mar 4 20:07:10 2021 +0100
  build time: Fri Mar  5 06:13:19 2021

Screenshots

Additional information

[nodeg]

Could you please provide a partial output of the cobbler log (/var/log/cobbler/cobbler.log), please?

[nodeg]

We will remove the get-loaders command with the upcoming version 3.3.0. It already got removed in master with #2572.

I tested the command with a fresh Cobbler installation on openSUSE Leap 15.2 and was not able to reproduce it.

$ cobbler version
Cobbler 3.2.1
  source: c7fc767c, Thu Mar 4 20:07:10 2021 +0100
  build time: Mon Mar  8 17:29:25 2021

$ cobbler get-loaders
task started: 2021-03-08_172949_get_loaders
task started (id=Download Bootloader Content, time=Mon Mar  8 17:29:49 2021)
running python triggers from /var/lib/cobbler/triggers/task/get_loaders/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/get_loaders/pre/*
shell triggers finished successfully
downloading https://cobbler.github.io/loaders/README to /var/lib/cobbler/loaders/README
downloading https://cobbler.github.io/loaders/COPYING.yaboot to /var/lib/cobbler/loaders/COPYING.yaboot
downloading https://cobbler.github.io/loaders/COPYING.syslinux to /var/lib/cobbler/loaders/COPYING.syslinux
downloading https://cobbler.github.io/loaders/yaboot-1.3.17 to /var/lib/cobbler/loaders/yaboot
downloading https://cobbler.github.io/loaders/pxelinux.0-3.86 to /var/lib/cobbler/loaders/pxelinux.0
downloading https://cobbler.github.io/loaders/menu.c32-3.86 to /var/lib/cobbler/loaders/menu.c32
downloading https://cobbler.github.io/loaders/grub-0.97-x86.efi to /var/lib/cobbler/loaders/grub-x86.efi
downloading https://cobbler.github.io/loaders/grub-0.97-x86_64.efi to /var/lib/cobbler/loaders/grub-x86_64.efi
*** TASK COMPLETE ***


$ cobbler get-loaders --force
task started: 2021-03-08_173220_get_loaders
task started (id=Download Bootloader Content, time=Mon Mar  8 17:32:20 2021)
running python triggers from /var/lib/cobbler/triggers/task/get_loaders/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/get_loaders/pre/*
shell triggers finished successfully
downloading https://cobbler.github.io/loaders/README to /var/lib/cobbler/loaders/README
downloading https://cobbler.github.io/loaders/COPYING.yaboot to /var/lib/cobbler/loaders/COPYING.yaboot
downloading https://cobbler.github.io/loaders/COPYING.syslinux to /var/lib/cobbler/loaders/COPYING.syslinux
downloading https://cobbler.github.io/loaders/yaboot-1.3.17 to /var/lib/cobbler/loaders/yaboot
downloading https://cobbler.github.io/loaders/pxelinux.0-3.86 to /var/lib/cobbler/loaders/pxelinux.0
downloading https://cobbler.github.io/loaders/menu.c32-3.86 to /var/lib/cobbler/loaders/menu.c32
downloading https://cobbler.github.io/loaders/grub-0.97-x86.efi to /var/lib/cobbler/loaders/grub-x86.efi
downloading https://cobbler.github.io/loaders/grub-0.97-x86_64.efi to /var/lib/cobbler/loaders/grub-x86_64.efi
*** TASK COMPLETE ***

[dansou901]

As the whole get-loaders code will be removed in a future version (see also #2572), I think we can close this as won't fix.
EDIT: @nodeg was faster...

[tim1670]

After removing get-loaders in version 3.2.1, I cannot boot pxe at all, I am Ubuntu 20.04 version. It's not that I don't approve of deletion, At least version 3.2 can also guide pxe to start, and I haven't found any solution in the update log and documentation.

[SchoolGuy]

@tim1670 There is the mkgrub.sh script which should help you. We will try to improve the docs drastically for the next version to make such things more trivial. This project should make life easy and not harder.

[tim1670]

@SchoolGuy I try to run the script, the log prompts:
~/cobbler/scripts$ sudo sh mkgrub.sh

mkgrub.sh: 5: [[: not found
mkgrub.sh: 6: [[: not found
mkgrub.sh: 32: function: not found
mkgrub.sh: 34: local: not in a function

Seems not compatible with Ubuntu 20.04?

In fact, I just hope that the LAN machine pxe netboot can guide other machine in the LAN to automatically install ubuntu 16 or 18 unattended, and download the installation files from the LAN machine without using the Internet or only a few update files are downloaded from the Internet.Just like installing the system to the virtual machine in VirtualBox with the iso file, I tried for 2 weeks, but there was no progress and I was very frustrated.

[SchoolGuy]

@tim1670 Ubuntu uses Dash and the script is written in Bash. Thus try to execute with bash /path/to/script. Also you need to install the bootloaders for the appropriate distros before.

[nodeg]

Some background information about the change in Ubuntu: https://wiki.ubuntu.com/DashAsBinSh Maybe we find some time to make our Bash scripts POSIX compliant.

[mohantnr]

logs: when we run cobbler check:

2021-03-14T23:54:57 - INFO | check
2021-03-14T23:54:57 - INFO | running: wget --help
2021-03-14T23:54:57 - INFO | received on stdout: GNU Wget 1.20.3, a non-interactive network retriever.
Usage: wget [OPTION]... [URL]...

Mandatory arguments to long options are mandatory for short options too.

Startup:
-V, --version display the version of Wget and exit
-h, --help print this help
-b, --background go to background after startup
-e, --execute=COMMAND execute a `.wgetrc'-style command

Logging and input file:
-o, --output-file=FILE log messages to FILE
-a, --append-output=FILE append messages to FILE
-d, --debug print lots of debugging information
-q, --quiet quiet (no output)
-v, --verbose be verbose (this is the default)
-nv, --no-verbose turn off verboseness, without being quiet
--report-speed=TYPE output bandwidth as TYPE. TYPE can be bits
-i, --input-file=FILE download URLs found in local or external FILE
-F, --force-html treat input file as HTML
-B, --base=URL resolves HTML input-file links (-i -F)
relative to URL
--config=FILE specify config file to use
--no-config do not read any config file
--rejected-log=FILE log reasons for URL rejection to FILE

Download:
-t, --tries=NUMBER set number of retries to NUMBER (0 unlimits)
--retry-connrefused retry even if connection is refused
--retry-on-http-error=ERRORS comma-separated list of HTTP errors to retry
-O, --output-document=FILE write documents to FILE
-nc, --no-clobber skip downloads that would download to
existing files (overwriting them)
--no-netrc don't try to obtain credentials from .netrc
-c, --continue resume getting a partially-downloaded file
--start-pos=OFFSET start downloading from zero-based position OFFSET
--progress=TYPE select progress gauge type
--show-progress display the progress bar in any verbosity mode
-N, --timestamping don't re-retrieve files unless newer than
local
--no-if-modified-since don't use conditional if-modified-since get
requests in timestamping mode

When we run cobbler get-loaders

2021-03-14T23:54:57 - DEBUG | received on stderr:
2021-03-14T23:54:57 - DEBUG | get_items; ['repo']
2021-03-14T23:54:57 - DEBUG | done with get_items; ['repo']
2021-03-14T23:54:57 - DEBUG | get_items; ['profile']
2021-03-14T23:54:57 - DEBUG | done with get_items; ['profile']
2021-03-14T23:56:57 - DEBUG | REMOTE CLI Authorized; user(?)
2021-03-14T23:56:57 - INFO | REMOTE start_task(Download Bootloader Content); event_id(2021-03-14_235657_get_loaders); user(?)
2021-03-14T23:56:58 - INFO | dlcontent

[SchoolGuy]

@mohantnr What is your question now? And is it even related to this Question?

[mohantnr]

You have requested to provide cobbler.log output. so i have updated case.

[nodeg]

Were you able to run the mkgrub.sh script successfully after the suggestion from @SchoolGuy to execute it with Bash instead of Dash?

[SchoolGuy]

Since for almost two weeks there was no response, I will close this. The functionality is discouraged for a long time and has been removed on master.

[sujit-jadhav]

@SchoolGuy Are you saying that "cobbler get-loaders" will be deprecated in Cobbler 3.3 ?
When is Cobbler 3.3 release expected? Omnia already using Cobbler version 2.8.5, can you suggest us how to overcome this issue?
We are getting "HTTPS Error 404 - Not Found" for "https://cobbler.github.io/loaders/README"

[SchoolGuy]

@sujit-jadhav Yes it is removed already on master. The content from our website was also removed. The bootloaders which were present were very out of date (multiple-years), thus full of security flaws and missing features.

Cobbler 3.3.0 is expected to be released end of June but since we have two major PRs open, I expect the version to be deleyed until we have them in and Cobbler is stable again.

An update from version 2.8.5 was described here and in the Gitter Chat multiple times roughly. For completness sake I will repeat it:

1. Backup everything thoroughly (be sure you can roll-back)!
2. Stop Cobbler and disable the service
3. Update to the latest version supplied by your OS-Vendor (if not possible I recommend you build yourself the appropriate DEB or RPM via the Docker Containers, if a required OS is missing please open a seperate issue for this)
4. Run the migration script(s) which can be found here. Please do not blindly execute. Please make yourself familiar with what the scripts do before you execute them.
5. Start your Cobbler 3.X.X and hope that it worked.

In case of errors during and after the migration look at /var/log/cobbler/cobbler.log!

Please note that if you are on RHEL 7 you will be on your own as RHEL 7 has not all required dependencies in Python 3 available via dnf/yum. For further discussion for the migration I would kindly ask that we move this to a sperate GH Discussion/GH Issue or into the Gitter/Matrix Chat.

[sujit-jadhav]

@SchoolGuy Can you please let me know probable release date for Cobbler 3.3.0 ? Earlier you mentioned it will be released end of June.

[SchoolGuy]

@sujit-jadhav The roadmap is the following:

• Fixup the current broken master
• Enabled buildiso for having UEFI
• Test Cobbler then throughly
• Release Cobbler 3.3.0

BUT you will then lose the webinterface which is developed separately and is not yet done. So Cobbler itself will - if everything goes according to plan - be released by the end of July now. On the webinterface I can't put a date yet but I will do my best.

But since 80% of the work is done in my private free time, I can't promise that this plan will go as I wrote it down here.

[sujit-jadhav]

@SchoolGuy, Is Cobbler 3.3.0 available now? We are ok to consume it without webinterface.
We tried using Cobbler 3.2.1 but we are facing issue #2614, for Cobbler 3.2.1 we are taking release 32, our environment is CentOS version: 8.4 and Python version 3.6.8

[SchoolGuy]

@sujit-jadhav Sadly not. We have found that the amount of manual migration steps are not accaptable for V3.3.0 and are thus writing logic to automate that now. This sadly is more complicated then thought but it is on the way. Because of internal deadlines at my company I am required to deliver V3.3.0 by the end of this month. The error you linked was never reproduced by us or the reporter so if you are running into it, I think you should reopen it with more details how we can reproduce it.

[sujit-jadhav]

@SchoolGuy , we are able to resolve the error #2614.
But now during PXE booting is not working,

Looks like we need boot loaders for Cobbler 3.2.1, if that is the case can you please let me know from where I can pick it up.

[SchoolGuy]

@sujit-jadhav As an interim quickfix you can use this: #2660 (comment)

Otherwise if you want to do it properly: #2660 (comment)

[sujit-jadhav]

@SchoolGuy both links are same: Quickfix and proper fix. Is there any typo?

[SchoolGuy]

@sujit-jadhav Nope they link to different comments in the issue. What hbokh describes is a quickfix and what I describe further down is the propper fix.

[sujit-jadhav]

@SchoolGuy
We are using CentOS 8.3 as our base OS and we have Cobbler 3.2.1 installed on it. We are trying to PXE boot CentOS 7.9 on the remote server. We have x86_64 based servers with UEFI boot.

Quickfix link has boot loaders for Cobbler 2.8.5, it doesn't work for our scenario.
Proper fix link which uses mkgrub-sh to generate UEFI bootable bootloaders for Cobbler installation throws following error:

+ grub2-mkimage -O x86_64-efi -o /var/lib/cobbler/loaders/grub/grubx64.efi --prefix= all_video boot cat configfile echo true font gfxmenu gfxterm gzio halt iso9660 jpeg minicmd normal part_apple part_msdos part_gpt password_pbkdf2 png reboot search search_fs_uuid search_fs_file search_label sleep test video fat loadenv linux btrfs ext2 xfs jfs reiserfs tftp http luks gcry_rijndael gcry_sha1 gcry_sha256 mdraid09 mdraid1x lvm serial regexp tr chain efinet
grub2-mkimage: error: cannot open `/usr/lib/grub/x86_64-efi/moddep.lst': No such file or directory.

[SchoolGuy]

@sujit-jadhav So that error message clearly says that one of the modules you are trying to use is not existing. Please adjust the list of included GRUB modules so grub2-mkimage is working.

[zparihar]

Hey folks,

I ran into the "cobbler get-loaders" issue on CentOS 8 with Cobbler 3.2.0, and I gotta say i'm a bit frustrated by the changes and the lack of documentation to go with it. It even talks about "cobbler get-loaders" at https://cobbler.readthedocs.io/en/release32/cobbler.html

I'm reading over here about "mkgrub.sh" and confused about how to use it... I opened up and looked through it and was hard to make sense of it... and as I was going through it and comparing the directories on my filesystem, I knew it was going to fail...

I'm running it to show you:

[root@centos84 tftpboot]# /usr/share/cobbler/bin/mkgrub.sh 
+ grub2-mkimage -O arm64-efi -o /var/lib/cobbler/loaders/grub/grubaa64.efi --prefix= all_video boot cat configfile echo true font gfxmenu gfxterm gzio halt iso9660 jpeg minicmd normal part_apple part_msdos part_gpt password_pbkdf2 png reboot search search_fs_uuid search_fs_file search_label sleep test video fat loadenv linux btrfs ext2 xfs jfs reiserfs tftp http luks gcry_rijndael gcry_sha1 gcry_sha256 mdraid09 mdraid1x lvm serial regexp tr efinet
grub2-mkimage: error: cannot open `/usr/lib/grub/arm64-efi/moddep.lst': No such file or directory.
+ set +x
+ grub2-mkimage -O i386-pc-pxe -o /var/lib/cobbler/loaders/grub/grub.0 --prefix= all_video boot cat configfile echo true font gfxmenu gfxterm gzio halt iso9660 jpeg minicmd normal part_apple part_msdos part_gpt password_pbkdf2 png reboot search search_fs_uuid search_fs_file search_label sleep test video fat loadenv linux btrfs ext2 xfs jfs reiserfs tftp http luks gcry_rijndael gcry_sha1 gcry_sha256 mdraid09 mdraid1x lvm serial regexp tr chain pxe biosdisk
+ set +x
+ grub2-mkimage -O powerpc-ieee1275 -o /var/lib/cobbler/loaders/grub/grub.ppc64le --prefix= all_video boot cat configfile echo true font gfxmenu gfxterm gzio halt iso9660 jpeg minicmd normal part_apple part_msdos part_gpt password_pbkdf2 png reboot search search_fs_uuid search_fs_file search_label sleep test video fat loadenv linux btrfs ext2 xfs jfs reiserfs tftp http luks gcry_rijndael gcry_sha1 gcry_sha256 mdraid09 mdraid1x lvm serial regexp tr net ofnet
grub2-mkimage: error: cannot open `/usr/lib/grub/powerpc-ieee1275/moddep.lst': No such file or directory.
+ set +x
+ grub2-mkimage -O x86_64-efi -o /var/lib/cobbler/loaders/grub/grubx64.efi --prefix= all_video boot cat configfile echo true font gfxmenu gfxterm gzio halt iso9660 jpeg minicmd normal part_apple part_msdos part_gpt password_pbkdf2 png reboot search search_fs_uuid search_fs_file search_label sleep test video fat loadenv linux btrfs ext2 xfs jfs reiserfs tftp http luks gcry_rijndael gcry_sha1 gcry_sha256 mdraid09 mdraid1x lvm serial regexp tr chain efinet
grub2-mkimage: error: cannot open `/usr/lib/grub/x86_64-efi/moddep.lst': No such file or directory.
+ set +x
+ ln -s /usr/share/syslinux/ldlinux.c32 /var/lib/cobbler/loaders/ldlinux.c32
+ set +x

There should at least be instructions on how to get a distro provisioning!

Anyways, what I had to do to get things to work was copy all the files in the directory /var/lib/cobbler/loaders from my CentOS 7 Cobbler 2.8.5 installation to the directory /var/lib/tftpboot/ in my CentOS 8 Cobbler 3.2.0 installation.

Just some background:

• I did a cobbler import on AlmaLinux 8.4
• I added cobbler repo add / reposync for AlmaLinux 8.4

I'm really hoping this could get smoothed out soon... I'm sure people less techical than myself would probably get lost...

[PabloMK7]

Having the same issue as @zparihar. Sadly, I don't have the files from a previous installation to get around this.

[Shubhangi-dell]

Hi @zparihar and @mariohackandglitch, we faced these same issues but we were able to solve them.
We are using CentOS-8.3 as the base OS and cobbler version 3.2.2.

Perform the following steps after installing cobbler:

1. dnf install -y grub2-efi-x64-modules efibootmgr
2. yum install -y grub2-efi-x64 shim-x64
3. Run mkgrub.sh

After this everything else remains the same.

For more info, you can view our project Omnia