Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CAHC missing CA cert for Red Hat #329

Open
miabbott opened this issue May 14, 2018 · 8 comments
Open

CAHC missing CA cert for Red Hat #329

miabbott opened this issue May 14, 2018 · 8 comments

Comments

@miabbott
Copy link
Contributor

The sanity tests on CAHC have been failing for an embarrassing amount of time, but I just did some digging after @jlebon asked about it.

The root cause looks like a missing CA cert for Red Hat things. This was observed when doing a docker pull from the registry:

    # docker pull registry.access.redhat.com/rhel7/openscap
    Using default tag: latest
    Trying to pull repository registry.access.redhat.com/rhel7/openscap ...
    open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory

The redhat-ca.crt file is usually a symlink to /etc/rhsm/ca/redhat-uep.pem. On the CAHC stream, this was provided by python-rhsm-certificates, but recent composes have caused this package to drop out of the compose. (FWIW, on RHELAH the cert is provided by subscription-manager-rhsm-certificates, but this package is basically empty in CentOS land)

The first compose where it appears this package was removed was on April 26, commit 4d12023435213f8c639337679d3f093f0188cfe8eaf77f4d5963ba5e35aea7e7
@miabbott miabbott mentioned this issue May 14, 2018
Merged
@jlebon
Copy link
Contributor

jlebon commented May 14, 2018

Hmm, so something dropped it as a dep? I suppose we could manually add it back in the manifest, though it'd be nice to track down what dropped the dep and why.

@miabbott
Copy link
Contributor Author

I mean, this is a testing stream...i'm not opposed to the hammer approach.

@cgwalters
Copy link
Member

Without actually investigating my offhand guess is that something in CentOS chagned to explicitly neuter that package in 7.5.

@miabbott
Copy link
Contributor Author

This is still an issue in the latest CAHC builds...

I tried to dig through the dependencies and what not:

# docker run -it --rm registry.centos.org/centos repoquery --whatprovides /etc/rhsm/ca/redhat-uep.pem
python-rhsm-certificates-0:1.19.10-1.el7_4.x86_64
# docker run -it --rm registry.centos.org/centos repoquery --whatrequires python-rhsm-certificates                                                                                     
python-rhsm-0:1.19.10-1.el7_4.x86_64
# docker run -it --rm registry.centos.org/centos repoquery --whatprovides python-rhsm
subscription-manager-rhsm-0:1.20.11-1.el7.centos.x86_64
python-rhsm-0:1.19.10-1.el7_4.x86_64
# rpm -qa | grep rhsm
subscription-manager-rhsm-certificates-1.20.11-1.el7.centos.x86_64
subscription-manager-rhsm-1.20.11-1.el7.centos.x86_64

The spec file for subscription-manager seems to indicate that the redhat-uep.pem cert should be installed by subscription-manager-rhsm-certificates:

https://src.fedoraproject.org/cgit/rpms/subscription-manager.git/tree/subscription-manager.spec?id=5484dbb5a4b319d70a4cc0d440c941f0463fd53a#n556

So....kind of back to where we started. ¯_(ツ)_/¯

Seems to support what @cgwalters said about the certs getting neutered. Not sure where to look for evidence of that.

@miabbott miabbott mentioned this issue Jun 13, 2018
Merged
@miabbott
Copy link
Contributor Author

This appears to affect all of CentOS

https://lists.centos.org/pipermail/centos-devel/2018-June/016749.html

@miabbott miabbott mentioned this issue Jun 19, 2018
Closed
miabbott added a commit to miabbott/atomic-host-tests that referenced this issue Jun 26, 2018
@miabbott
dprr: modify list of "popular" images for CentOS based streams
cd984f0 
The i-s-t has been failing for longer than I would care to admit
because of CentOS/sig-atomic-buildscripts#329 and there's been little
movement in the direction of fixing that.  So we'll workaround the
deficincieny by using two lists of images, one that works for CentOS
derived streams and another that works for the rest.
@miabbott miabbott mentioned this issue Jun 26, 2018
Merged
mike-nguyen pushed a commit to projectatomic/atomic-host-tests that referenced this issue Jun 26, 2018
@mike-nguyen
dprr: modify list of "popular" images for CentOS based streams (#418)
8f05eda 
The i-s-t has been failing for longer than I would care to admit
because of CentOS/sig-atomic-buildscripts#329 and there's been little
movement in the direction of fixing that.  So we'll workaround the
deficincieny by using two lists of images, one that works for CentOS
derived streams and another that works for the rest.
@miabbott miabbott mentioned this issue Jul 17, 2018
Closed
miabbott added a commit to miabbott/atomic-host-tests that referenced this issue Oct 11, 2018
@miabbott
roles: use alternate set of images for CentOS
0771e51 
CentOS AH is missing the Red Hat CA cert, so it is unable to pull
images from the registry (see:  CentOS/sig-atomic-buildscripts#329).
To workaround this, we'll have to build the list of images to pull
differently for CentOS vs. the rest.
@miabbott miabbott mentioned this issue Oct 11, 2018
Merged
miabbott added a commit to miabbott/atomic-host-tests that referenced this issue Oct 12, 2018
@miabbott
roles: use alternate set of images for CentOS
09c1e56 
CentOS AH is missing the Red Hat CA cert, so it is unable to pull
images from the registry (see:  CentOS/sig-atomic-buildscripts#329).
To workaround this, we'll have to build the list of images to pull
differently for CentOS vs. the rest.
mike-nguyen pushed a commit to projectatomic/atomic-host-tests that referenced this issue Oct 12, 2018
@mike-nguyen
introduce podman sanity checks (#417)
494eb7c 
* roles: podman_pull_run_remove

This introduces a new role named `podman_pull_run_remove` which is a
implementation of `docker_pull_run_remove` using `podman`.

The role has been enhanced to test basic running of containers and
also testing network access from the container.  (These enhancements
should likely be applied to `docker_pull_run_remove`, too).

* i-s-t:  add basic podman tests

Let's start testing `podman` on the hosts that support it.

* roles: fix centos container image location

* roles: use alternate set of images for CentOS

CentOS AH is missing the Red Hat CA cert, so it is unable to pull
images from the registry (see:  CentOS/sig-atomic-buildscripts#329).
To workaround this, we'll have to build the list of images to pull
differently for CentOS vs. the rest.
@brianUK007
Copy link

brianUK007 commented Nov 21, 2018 •
edited
Loading

@miabbott I just built a 3.10 OKD cluster on centos7 with ansible and containerized gluster for dynamic storage. I had run a git checkout release-3.10 for https://github.com/openshift/openshift-ansible.git in late october / november to build the cluster. Upgrading with Gluster is a bit more difficult than just running the upgrade script.

Is there a workaround to this? Could I inject the cert into the worker nodes to get around this issue? I've wanted to demo the .Net examples to some of our dev team working with .Net but running into issues pulling images from RedHat on OKD 3.10.

@miabbott
Copy link
Contributor Author

@brianUK007 I decided to have another look at this to see what I can find. The idea that the CA certs are getting neutered as part of the build process appears to be confirmed. I believe I found the proper spec file which shows an explicit removal of the certs on CentOS:

https://git.centos.org/blob/rpms!subscription-manager.git/c7/SPECS!subscription-manager.spec#L562

But, the registry still has to offer its cert when negotiating SSL, so we can grab it and stick it where it needs to be.

Using some help from StackOverflow, I grabbed the cert and stuck it in the right place for docker:

# docker pull registry.access.redhat.com/rhel7/openscap
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7/openscap ... 
open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory

# openssl s_client -showcerts -servername registry.access.redhat.com -connect registry.access.redhat.com:443 </dev/null 2>/dev/null | openssl x509 -text > /etc/rhsm/ca/redhat-uep.pem

# docker pull registry.access.redhat.com/rhel7/openscap
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7/openscap ... 
latest: Pulling from registry.access.redhat.com/rhel7/openscap
9a1bea865f79: Pull complete 
602125c154e3: Pull complete 
4f39a853bed4: Pull complete 
20c68cea93f0: Pull complete 
Digest: sha256:aa5ddb23af242da108ee0cfe227a96ced06ad398e4c8bb201aa837ca2837e432
Status: Downloaded newer image for registry.access.redhat.com/rhel7/openscap:latest

I'm not sure what redhat-entitlement-authority.pem is needed for (or where to get it), but this at least should get to the point where you can pull images with docker.

@brianUK007
Copy link

brianUK007 commented Nov 28, 2018 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@cgwalters @miabbott @jlebon @brianUK007