Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CA not working on brand new fedora-33 install #385

Closed
thinkl33t opened this issue Mar 18, 2021 · 25 comments
Closed

CA not working on brand new fedora-33 install #385

thinkl33t opened this issue Mar 18, 2021 · 25 comments

Comments

@thinkl33t
Copy link
Contributor

I'm installing freeipa using the fedora-33 container from dockerhub as follows. I'm orchestrating the container using ansible to template the ipa-server-install-options file to /data on the server:

--realm="{{ ipa_realm }}"
--domain="{{ ipa_domain }}"
--ds-password="{{ ipa_dm_password }}"
--admin-password="{{ ipa_admin_password }}"

--hostname="{{ ipa_hostname }}"

--no-ntp
--no-ssh
--no-sshd

--unattended

The realm and domain are both valid, and the hostname has an A record pointing to the docker server. To run the container itself, i'm running it as follows:

- name: Start FreeIPA Container
  docker_container:
    name: freeipa
    hostname: "{{ ipa_hostname }}"
    image: "freeipa/freeipa-server:fedora-33"
    restart_policy: unless-stopped
    state: started
    volumes:
      - "/sys/fs/cgroup:/sys/fs/cgroup:ro"
      - "/srv/freeipa:/data"
      - "/opt/dehydrated/certs/{{ letsencrypt_hostnames[0] }}:/ssl"
    tmpfs:
      - "/run"
      - "/tmp"
    ports:
      - "53:53/udp"
      - "53:53"
      - "389:389"
      - "636:636"
      - "88:88"
      - "464:464"
      - "88:88/udp"
      - "464:464/udp"
      - "127.0.0.1:9123:123/udp"
      - "80:80"
      - "443:443"

The install all seems to go fine, and docker exec -ti freeipa ipactl status shows all the services as running:

docker exec -ti freeipa ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful

The webui is running fine at https://fqdn/, but the ca at https://fqdn/ca/rest/securityDomain/domainInfo gives a 503 error. That URL is proxied via apache httpd to pki-tomcat on port 8009, but if we look inside the container, nothing is running on port 8009.

I've had a dig through the logs for pki-tomcat using journalctl, the section that appears unhappy is here, but i'm at a loss how to interpret this into an actionable fix.

Mar 18 13:35:49 ipa7.dev systemd[1]: Starting PKI Tomcat Server pki-tomcat...
Mar 18 13:35:56 ipa7.dev java[432]: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
Mar 18 13:35:58 ipa7.dev server[551]: Java virtual machine used: /usr/bin/java
Mar 18 13:35:58 ipa7.dev server[551]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar
Mar 18 13:35:58 ipa7.dev server[551]: main class used: org.apache.catalina.startup.Bootstrap
Mar 18 13:35:58 ipa7.dev server[551]: flags used: -Dcom.redhat.fips=false
Mar 18 13:35:58 ipa7.dev server[551]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager     -Djava.security.manager     -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
Mar 18 13:35:58 ipa7.dev server[551]: arguments used: start
Mar 18 13:35:59 ipa7.dev ipa-pki-wait-running[552]: pki.client: /usr/libexec/ipa/ipa-pki-wait-running:60: The subsystem in PKIConnection.__init__() has been deprecated (https://www.dogtagpki.org/wiki/PKI_10.8_Python_Changes).
Mar 18 13:35:59 ipa7.dev ipa-pki-wait-running[552]: ipa-pki-wait-running: Created connection http://ipa7.dev:8080/ca
Mar 18 13:35:59 ipa7.dev ipa-pki-wait-running[552]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='ipa7.dev', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcc5603f4c0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Mar 18 13:36:00 ipa7.dev ipa-pki-wait-running[552]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='ipa7.dev', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcc5603f910>: Failed to establish a new connection: [Errno 111] Connection refused'))
Mar 18 13:36:01 ipa7.dev java[551]: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
Mar 18 13:36:01 ipa7.dev ipa-pki-wait-running[552]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='ipa7.dev', port=8080): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcc5603fcd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Mar 18 13:36:02 ipa7.dev server[551]: SEVERE: Failed to initialize component [Connector[AJP/1.3-8009]]
Mar 18 13:36:02 ipa7.dev server[551]: org.apache.catalina.LifecycleException: Protocol handler initialization failed
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.connector.Connector.initInternal(Connector.java:1049)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.core.StandardService.initInternal(StandardService.java:558)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.startup.Catalina.load(Catalina.java:724)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.startup.Catalina.load(Catalina.java:746)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
Mar 18 13:36:02 ipa7.dev server[551]: Caused by: java.net.BindException: Cannot assign requested address
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/sun.nio.ch.Net.bind0(Native Method)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/sun.nio.ch.Net.bind(Net.java:455)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/sun.nio.ch.Net.bind(Net.java:447)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.tomcat.util.net.NioEndpoint.initServerSocket(NioEndpoint.java:277)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:241)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1193)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1206)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:597)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.connector.Connector.initInternal(Connector.java:1046)
Mar 18 13:36:02 ipa7.dev server[551]:         ... 13 more
Mar 18 13:36:02 ipa7.dev server[551]: SEVERE: Failed to initialize component [Connector[AJP/1.3-8009]]
Mar 18 13:36:02 ipa7.dev server[551]: org.apache.catalina.LifecycleException: Protocol handler initialization failed
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.connector.Connector.initInternal(Connector.java:1049)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.core.StandardService.initInternal(StandardService.java:558)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.startup.Catalina.load(Catalina.java:724)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.startup.Catalina.load(Catalina.java:746)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
Mar 18 13:36:02 ipa7.dev server[551]: Caused by: java.net.BindException: Cannot assign requested address
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/sun.nio.ch.Net.bind0(Native Method)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/sun.nio.ch.Net.bind(Net.java:455)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/sun.nio.ch.Net.bind(Net.java:447)
Mar 18 13:36:02 ipa7.dev server[551]:         at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.tomcat.util.net.NioEndpoint.initServerSocket(NioEndpoint.java:277)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:241)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1193)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1206)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:597)
Mar 18 13:36:02 ipa7.dev server[551]:         at org.apache.catalina.connector.Connector.initInternal(Connector.java:1046)
Mar 18 13:36:02 ipa7.dev server[551]:         ... 13 more
@adelton
Copy link
Collaborator

adelton commented Mar 19, 2021

Given that

Caused by: java.net.BindException: Cannot assign requested address

error, I'd assume that the container has a problem resolving that ipa7.dev hostname.

What's in /etc/hosts in the container and what's in /etc/resolv.conf in the container?

@thinkl33t
Copy link
Contributor Author

[root@ipa7 /]# cat /etc/hosts 
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.18.0.2	ipa7.dev ipa7
[root@ipa7 /]# cat /etc/resolv.conf 
nameserver 127.0.0.11
options ndots:0

@adelton
Copy link
Collaborator

adelton commented Mar 19, 2021

That seems sane.

@abbra, any hints what could be causing the tomcat startup issues, what to look for?

@thinkl33t, can you try to run tests/run-master-and-replica.sh freeipa/freeipa-server:fedora-33 to see if things work reasonably well with just direct docker approach?

@abbra
Copy link

abbra commented Mar 19, 2021

No idea for this specific Java issue. However, on IPA side we do not support single label domains (like dev above).

@thinkl33t
Copy link
Contributor Author

thinkl33t commented Mar 19, 2021 via email

@thinkl33t
Copy link
Contributor Author

Is it plausible that this lookup isn't looking at /etc/hosts at all, is doing a (forwarded) dns lookup for the FQDN, and trying to assign the external ip address of the container host where it should be trying to assign the container's internal ip address?

@adelton
Copy link
Collaborator

adelton commented Mar 22, 2021

That's rather unlikely, unless nsswitch.conf was updated in some creative way. What does

grep host /etc/nsswitch.conf

in the container return?

Were you able to run the test with tests/run-master-and-replica.sh?

@thinkl33t
Copy link
Contributor Author

thinkl33t commented Mar 22, 2021 via email

@adelton
Copy link
Collaborator

adelton commented Apr 1, 2021

We seem to have lost traction there.

@adelton adelton closed this as completed Apr 1, 2021
@thinkl33t
Copy link
Contributor Author

Apologies, i've been off work for the past few weeks and have just managed to hop back on this again. I think i've tracked down the issue though!

The CA for freeipa is provided via an AJP proxy to port 8009, which doesn't have anything connected to it. This is provided by the pki-tomcat service. Having a look at journalctl for the pki-dogtag instance shows the following line:

Mar 18 13:36:02 ipa7.dev.example.org server[551]: Caused by: java.net.BindException: Cannot assign requested address

When we discussed this above we assumed that it couldn't see ipa7.dev.example.org, as the stacktrace was surrounded by things accessing that. This isn't correct though! If we have a look at the pki-tomcat config file at /etc/pki/pki-tomcat/server.xml inside the container, we find:

   <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost4" name="Connector1" secret="4TdGUjMie1o8BR9B3y861TjIqX5uBWugXdRdV7cqn0i9"/>
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost6" name="Connector2" secret="4TdGUjMie1o8BR9B3y861TjIqX5uBWugXdRdV7cqn0i9"/>

These are the ajp connector configs for tomcat, and ask the ajp connectors to bind to localhost4 and localhost6, which don't exist in /etc/hosts inside the container!

127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback

Adding localhost4 and localhost6 entries manually to the hostfile appears to fix the issue:

127.0.0.1	localhost localhost4
::1	localhost ip6-localhost ip6-loopback localhost6
# ss -tnpl | grep 8009
LISTEN 0      100        127.0.0.1:8009       0.0.0.0:*  

@adelton adelton reopened this Apr 9, 2021
@thinkl33t
Copy link
Contributor Author

thinkl33t commented Apr 9, 2021

I think this is a difference between fedora upstream (which has localhost4 by default), and our containerized version, where docker manages /etc/hosts itself.

My current thought is to add something to the entrypoint that will make sure localhost4 and localhost6 are in /etc/hosts, and in the right place - that should be after the point at which docker has generated /etc/hosts

The other option would be to patch server.xml to hard-code 127.0.0.1 / ::1 in place of localhost4 and localhost6, but i believe ipa-server-install is poking this file itself, so i'm unsure if we can provide defaults?

@thinkl33t
Copy link
Contributor Author

thinkl33t commented Apr 9, 2021

Passing in --add-host localhost4:127.0.0.1 to docker appears to work as a workaround for now, leaving a working CA after install!

@adelton
Copy link
Collaborator

adelton commented Apr 9, 2021

I agree that the assumptions that FreeIPA / Dogtag makes about the environment (specifically hostnames specified in /etc/hosts) are not always met when the /etc/hosts content gets created by the container runtime. Even on Fedora, docker and podman behave differently in that respect.

The --add-host might be a viable approach.

@abbra, what are your thoughts about this? Assuming we'd be able to reproduce this outside of container, would an issue "FreeIPA does not setup fully working CA on host that does not have localhost4/localhost6 in /etc/hosts" be something that the FreeIPA team would be willing to consider investigating?

@abbra
Copy link

abbra commented Apr 9, 2021

FreeIPA doesn't modify server.xml's host references. They come from dogtag and need to be handled in dogtag installer. They'd be affecting PKI container in the same way. So please open a bug against dogtag.

@thinkl33t
Copy link
Contributor Author

Looks like dogtagpki/pki@1906afb is the commit in dogtag that has caused this issue

@adelton
Copy link
Collaborator

adelton commented Apr 9, 2021

FreeIPA doesn't modify server.xml's host references. They come from dogtag and need to be handled in dogtag installer. They'd be affecting PKI container in the same way. So please open a bug against dogtag.

What is the Dogtag-only equivalent to ipa-server-install?

@rcritten
Copy link

rcritten commented Apr 9, 2021

IPA calls pkispawn

https://www.dogtagpki.org/wiki/Installing_Custom_CA

@thinkl33t
Copy link
Contributor Author

The defaults file in the container at /usr/share/pki/server/etc/default.cfg appears to contain the pki_ajp_host_ipv4=localhost4 / pki_ajp_host_ipv6=localhost6 defaults that are used for setting up the dogtag server.xml, so if we can't easily pass in our own values for those two we can always tweak that file.

@thinkl33t
Copy link
Contributor Author

ipa builds out the dict that is passed through to pkispawn here so its plausible we could patch ipa to allow specifying what host to use for ajp, either at the docker container level or upstream.

@abbra
Copy link

abbra commented Apr 10, 2021

you already can pass PKI override config to ipa-server-install, where these overrides can be specified. They'll be automatically applied on top of IPA configuration. See --pki-config-override option. Fraser's blog may be helpful for the details: https://frasertweedale.github.io/blog-redhat/posts/2020-01-28-freeipa-override-ca-key-size.html

@adelton
Copy link
Collaborator

adelton commented Apr 10, 2021

Sadly, following https://www.dogtagpki.org/wiki/Installing_Custom_CA does not give me error-free setup. So I tried ipa-server-install on a Fedora 33 host where I removed localhost4 and localhost6 beforehand with

# sed -i 's/localhost[46].*//' /etc/hosts

The problem -- I was not able to reproduce the issue on host (outside of containers) as

# curl -k https://$HOSTNAME/ca/rest/securityDomain/domainInfo

then passes just fine.

(Internal reference J:5258540.)

@thinkl33t
Copy link
Contributor Author

I can confirm that passing in a file pki_override.cfg containing:

[CA]
pki_ajp_host_ipv4=127.0.0.1
pki_ajp_host_ipv6=::1

And adding --pki-config-override="/data/pki_override.cfg" to my ipa-server-install-options works around this issue.

@adelton
Copy link
Collaborator

adelton commented May 4, 2021

Since there is a solution / workaround and at the same time I was not able to reproduce the problem outside of containers to be able to report it to either Dogtag or FreeIPA projects, I'm inclined to close this issue.

@adelton
Copy link
Collaborator

adelton commented May 29, 2021

Closing.

@adelton adelton closed this as completed May 29, 2021
@Westie
Copy link

Westie commented Aug 7, 2022

I'd like to add that I too have been able to replicate this issue during migration of a legacy installation.

Adding the following to the docker-compose file solved it:

    extra_hosts:
      localhost4: 127.0.0.1
      localhost6: ::1

Just for anyone else encounters this issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants