Extra yum repositories are breaking the whole world

[duck-rh]

Quack,

I was investigating why Mailman3 was not working anymore on the new server. Misc helped me find out that hyperkitty requires networkx which in turn requires numpy and the WSGI process was stuck in the later library (until it finaly timeout after several minutes and gave an error in the UI). Downgrading networkx to the one in EPEL (instead of the one from the openstack repository) solved the problem.

Moreover, the @centos-openstack-ocata repository brings in a new version of pyOpenSSL. Unfortunately this version also needs a more recent version of python2-cryptography which is not available in any of the repos. This means first there is a bug in the @centos-openstack-ocata repo I'm not sure where to report, and this breaks certbot (the Let's Encrypt tool).

IIRC you suggested to disable the extra repos after installing Sensu and other SIG packages. This might work if the draggued dependencies for the needed tools happen to not break things by chance. Unfortunately this also means it would be impossible to upgrade and have security fixes, which is not acceptable. So we need to find a better solution I'm afraid.

\_o<

[duck-rh]

I see for example a new version of python-django coming from CentOS-OpsTools, but in fact no package other than MM3 needs it. Clearly many packages are made available to the whole opstools stack, but on a server you don't need the web interface and all the tools on the monitoring server side.

So this is to say for this case the best solution which would not involve complex containerization on either MM3 or Sensu side would be to split the package repository in two: packages for the monitoring server and package for the client.

I just looked at all the packages provided in the opstools repo and except for sensu I see no other packages needed on the client. Also the sensu repository currently holds much more recent versions, so I guess the ancient ones in the opstools repo are obsolete and could be removed.

I would suggest this:

• keep the sensu repo, as it only contains sensu and uchiwa
• patch the opstools role to not drag in the opstools repo when you only install the client
• remove all other repos; they seem to come from the install image, and the list may change, so I would be in favor of using a more basic CentOS image instead of having to disable a list of repo not under control and possibly other openstacl-specific alterations which do not make sense for this use case

[quaid]

Regarding the CentOS Ops Tools group and packaging discussions, centos-devel@centos.org is a good place for that conversation.

Maybe get things working, then post your solution & ask the plans for the CentOS repo here -- what are they tracking, etc.

I'm presuming there aren't other potential conflicts that might occur in the future that makes having a separate or SCL package a possible solution. Those might arise from discussions in centos-devel.

[javierpena]

I've just tested installing pyOpenSSL from the Ocata repos:

python2-pyOpenSSL noarch 16.2.0-3.el7 centos-openstack-ocata 88 k
...
python2-cryptography x86_64 1.7.2-1.el7 base 502 k

It pulls python-cryptography from the base repo. Maybe there was a repo cache issue due to the CentOS 7.4 release?

[duck-rh]

@javierpena you're right, it is available with the proper vers in 7.4, but 7.3 had only 1.3.1-3. AFAIK the release we on 2017-09-13, so it seems the centos-openstack-ocata repository is ahead of the release.

Also the pyOpenSSL package should have had the right versionned dependency and failed to upgrade, so there is most probably a packaging problem. Thanks @quaid for the pointers.

As for a solution, I suggested one after having spent quite some time understand what is of need of all these repositories. I'm a bit reluctant to patch the opstools role before an agreement is reached first. Also if noone has time to reply to this tickets then the same people won't have time to review my changes in Gerrit, which is even more time-consuming.

As for the install image, I'm not sure what else is changed and if a base CentOS image would still be find for RDO's needs. I think all changes which are not related to proper booting on the platform should be done via Ansible so they are easily adaptable and documented in the code.

[duck-rh]

The openstack-ocata repository provides a newer Python Cryptography version, but the dependency seems lax:
package: python2-pyOpenSSL.noarch 16.2.0-3.el7
dependency: python(abi) = 2.7
provider: python.x86_64 2.7.5-58.el7
dependency: python-cryptography >= 1.3.0
provider: python2-cryptography.x86_64 1.7.2-1.el7

I guess now that 7.4 is out people would have upgraded and it's less of a problem, but, if I'm not mistaken, the dependency should probably be fixed.

[duck-rh]

Just to clarify, there was no 1.7 version of Python Cryptography available on any repository configured on this machine at the time of the problem. I don't know if it was a cache or repository problem though.

[apevec]

@duck-rh please post repo configuration from that machine so we can reproduce. AFAIK CentOS supports only latest version i.e. you cannot stay on 7.3 and from @javierpena comment this is not reproducible on 7.4

[duck-rh]

@apevec it is not reproducible anymore since 7.4, but the problem appeared before the release. Is it expected from the openstack-ocata repository to provide packages for the next unreleased version of CentOS at least a good week in advance?

Anyway, the main problem is the fact MailMan does not expect packages not coming either from the base CentOS or from its own repository. two weeks ago I suggested changes in the "I would suggest this:" paragraph, so I would gladly get your opinion. Before spending time providing a patch for the opstools role I absolutely don't know I would prefer we agree first on the plan. As for the image, which I believe is activating several extra repository, I think it would be better to move this part, on-demand, in the Ansible rules, so we can skip then if unneeded or causing conflicts.

[javierpena]

Regarding the proposal to change the ansible roles: I'm not very familiar with the way they're built, but I see all repos are pulled in by https://github.com/centos-opstools/opstools-ansible/tree/124cf6adc9fabe0f2f2bad42d35d199e83733f6f/roles/repos/opstools . The base image should only have the official CentOS repos.

If any of these additional repos is an issue for Mailman, we can try to remove them as part of a Mailman-specific role, or set "manage_repos: false" in that role. That would mean we'd have to care about which repos we include as part of the role.

[duck-rh]

@javierpena youre right it not only drags the opstools repo but also the centos-release-openstack- one, which depends on the other ones. I missed this part with grep, as there is a variable in the package name, thanks.

This would mean the mailman role would have to care about totally unrelated things, which can change over time out of our control. This would be very fragile.

I don't get why opstools depends on openstack repos at all. But even if the server needs specific updates from these repo, the client seems to be limited to installing sensu according to yum list installed.

I agree we should care about any repo we include, for any role we use. I don't have the knowledge to say which are useful or not for the opstools server side, but we could try with this new machine which is not in production to see if the client side would work with only the sensu repository and all other packages downgraded to a base system. If it works then I would recommend patching the opstools role accordingly.

[apevec]

why opstools depends on openstack repos

@duck-rh
b/c it currently contains osops-tools-monitoring-oschecks which depends on OpenStack clients:
http://cbs.centos.org/koji/rpminfo?rpmID=79870

This package is being moved to RDO
https://review.rdoproject.org/r/#/q/topic:add-osops-tools-monitoring-oschecks
after that's done we can drop opstools dependency on openstack repos

@mrunge @paramite @larsks do you have a tracker for that ^ ?

[duck-rh]

thanks @apevec

[paramite]

Hmm, weird. That was supposed to happen on March already: centos-opstools/centos-release-opstools@bc8afcc

[mrunge]

There is an unresolved bug on https://bugs.centos.org/view.php?id=13374 about removing outdated builds from centos mirrors.

Installing centos-opstools repo from centos via yum install centos-release-opstools should not pull in any openstack repos, see http://cbs.centos.org/koji/buildinfo?buildID=16800

[duck-rh]

If several builds are made for the same exact version, we might fall into the same problem I got in Copr: https://bugzilla.redhat.com/show_bug.cgi?id=1489550
I did not bug YUM itself, as it is end of life and probably noone would care to solve this problem.

[duck-rh]

I upgrade the machine to 7.4 with all these repositories disabled. I just tried to remove centos-release-openstack-ocata and it worked fine, removing all the other openstack-related repos too.

For information, this machine was created on 2017-07-21. www.rdoproject.org has the same problem and was provisioned at the same period (2017-07-06). The provisioning of opstools happened around 2017-07-27.

Thanks @mrunge