Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

yum is not compatible with urlgrabber >= 4.0.0 #10

Closed
gchiesa opened this issue Feb 28, 2019 · 4 comments · Fixed by #13
Closed

yum is not compatible with urlgrabber >= 4.0.0 #10

gchiesa opened this issue Feb 28, 2019 · 4 comments · Fixed by #13
Assignees
@dmnks

Comments

@gchiesa
Copy link

gchiesa commented Feb 28, 2019

This morning has been released a major updated of urlgrabber package
This package is breaking yum functionality when downloading metalink files.

How to reproduce it:

  • on a vanilla centos or docker centos:7 as root when I run the commands:
# yum update -y
# curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py &&  python
get-pip.py
# yum install -y curl-devel openssl-devel gcc python-devel
# pip install six
# export PYCURL_SSL_LIBRARY=nss
# pip install --upgrade --ignore-installed  urlgrabber
# yum install -y epel-release
# yum search vim

The ouput of the yum search is:

Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
epel/x86_64/metalink
                                            |  28 kB  00:00:00
 * base: mirror.serverius.net
 * epel: ftp.nluug.nl
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 375, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 184, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 585, in doCommands
    return self.yum_cli_commands[self.basecmd].doCommand(self,
self.basecmd, self.extcmds)
  File "/usr/share/yum-cli/yumcommands.py", line 1731, in doCommand
    return base.search(extcmds)
  File "/usr/share/yum-cli/cli.py", line 1503, in search
    for (po, keys, matched_value) in matching:
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 3250, in
searchGenerator
    for sack in self.pkgSack.sacks.values():
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1075, in
<lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 778, in
_getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 347, in
populateSack
    self.doSetup()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 122, in doSetup
    self.ayum.plugins.run('prereposetup')
  File "/usr/lib/python2.7/site-packages/yum/plugins.py", line 188, in run
    func(conduitcls(self, self.base, conf, **kwargs))
  File "/usr/lib/yum-plugins/fastestmirror.py", line 240, in
prereposetup_hook
    repo.setupGrab()
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 603, in
setupGrab
    self._setupGrab()
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 663, in
_setupGrab
    failure_callback=mirror_failure)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 288,
in __init__
    self.mirrors.sort(key=estimate, reverse=True)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 282,
in estimate
    speed, fail = _TH.estimate(m['mirror'])
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2547,
in estimate
    host = urlparse.urlsplit(baseurl).netloc.split(b'@')[-1] or baseurl
  File "/usr/lib64/python2.7/urlparse.py", line 175, in urlsplit
    cached = _parse_cache.get(key, None)
TypeError: unhashable type: 'dict'

If I downgrade urlgrabber to the previous version with the commands:

# pip install --upgrade urlgrabber==3.10.2
# yum search vim

The output of the yum search is the expected one.
@Conan-Kudo Conan-Kudo changed the title New urlgrabber package is breaking yum functionalities yum is not compatible with urlgrabber >= 4.0.0 Feb 28, 2019
@Conan-Kudo
Copy link
Member

This is not terribly surprising, to be honest, as yum did not factor into releasing urlgrabber.

@Conan-Kudo
Copy link
Member

Please just use the urlgrabber version shipped as an RPM in EL7. This release will not be backported to EL7.

@tomchiverton
Copy link

"This is not terribly surprising, to be honest, as yum did not factor into releasing urlgrabber."

You might want to have a word with Fedora 30 then, because they are shipping both, and it's broken

Traceback (most recent call last):
  File "/usr/sbin/yum-cron", line 713, in <module>
    main()
  File "/usr/sbin/yum-cron", line 710, in main
    base.updatesCheck()
  File "/usr/sbin/yum-cron", line 602, in updatesCheck
    self.populateUpdateMetadata()
  File "/usr/sbin/yum-cron", line 409, in populateUpdateMetadata
    self.pkgSack # honor skip_if_unavailable
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1095, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 799, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 344, in populateSack
    self.doSetup()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
    self.retrieveAllMD()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
    dl = repo._async and repo._commonLoadRepoXML(repo)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1460, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1234, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1014, in _getFile
    result = self.grab.urlgrab(misc.to_utf8(relative), local,
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 698, in <lambda>
    grab = property(lambda self: self._getgrab())
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 693, in _getgrab
    self._setupGrab()
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 659, in _setupGrab
    failure_callback=mirror_failure)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 288, in __init__
    self.mirrors.sort(key=estimate, reverse=True)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 282, in estimate
    speed, fail = _TH.estimate(m['mirror'])
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2547, in estimate
    host = urlparse.urlsplit(baseurl).netloc.split(b'@')[-1] or baseurl
  File "/usr/lib64/python2.7/urlparse.py", line 176, in urlsplit
    cached = _parse_cache.get(key, None)
TypeError: unhashable type: 'dict'

[root@bookcase ~]# rpm -qa|grep urlgrab
python2-urlgrabber-4.0.0-1.fc30.noarch
[root@bookcase ~]# rpm -qa|grep yum
dnf-yum-4.2.5-1.fc30.noarch
yum-cron-3.4.3-521.fc30.noarch
yum-cron-security-3.4.3-521.fc30.noarch
yum-3.4.3-521.fc30.noarch
yum-metadata-parser-1.1.4-22.fc29.x86_64
yum-plugin-fastestmirror-1.1.31-519.fc30.noarch
yum-utils-1.1.31-519.fc30.noarch
[root@bookcase ~]#

@Conan-Kudo
Copy link
Member

Conan-Kudo commented May 15, 2019 •
edited

As @dmnks maintains yum, he's probably the right person to adapt the code. File an issue against yum.

dmnks added a commit to dmnks/urlgrabber that referenced this issue May 20, 2019
@dmnks
Revert "Simplify mirror conversion to utf8"
106e101 
This reverts commit be8ee10.

A list of dicts (as opposed to strings) is valid input as well; see the
module-level doc string for details (section 2 under CUSTOMIZATION).  In
fact, the nested estimate() function in MirrorGroup.__init__() accounts
for that, too.

This fixes a traceback in YUM which does pass such a dict list.

Closes rpm-software-management#10.
@dmnks dmnks mentioned this issue May 20, 2019
Merged
Conan-Kudo pushed a commit that referenced this issue May 20, 2019
@dmnks @Conan-Kudo
Revert "Simplify mirror conversion to utf8"
7607b9f 
This reverts commit be8ee10.

A list of dicts (as opposed to strings) is valid input as well; see the
module-level doc string for details (section 2 under CUSTOMIZATION).  In
fact, the nested estimate() function in MirrorGroup.__init__() accounts
for that, too.

This fixes a traceback in YUM which does pass such a dict list.

Closes #10.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dmnks dmnks

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert "Simplify mirror conversion to utf8" dmnks/urlgrabber
4 participants
@Conan-Kudo @tomchiverton @gchiesa @dmnks