Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenID login doesn't work #2418

Closed
praiskup opened this issue Dec 2, 2022 · 6 comments
Closed

OpenID login doesn't work #2418

praiskup opened this issue Dec 2, 2022 · 6 comments
Assignees
Labels

Comments

@praiskup
Copy link
Member

praiskup commented Dec 2, 2022

When I hit "login" link, with kerberos token taken, I get "504 gateway timeout".
https://pagure.io/fedora-infrastructure/issue/11025

@praiskup praiskup assigned praiskup and unassigned praiskup Dec 2, 2022
@praiskup
Copy link
Member Author

praiskup commented Dec 2, 2022

The same version of our software works fine on staging (against staging oid).

Not sure what is going on. The @oid.after_login() hook is never called by Flask OpenID.

@praiskup praiskup added the bug label Dec 3, 2022
@praiskup praiskup self-assigned this Dec 4, 2022
@praiskup
Copy link
Member Author

praiskup commented Dec 4, 2022

I enabled production Fedora OpenID even on Fedora Copr stage, and it behaves the same way... I think there's some difference between id.stg.fedoraproject.org vs. id.fedoraproject.org.

@praiskup praiskup assigned FrostyX and unassigned praiskup Dec 5, 2022
@FrostyX
Copy link
Member

FrostyX commented Dec 5, 2022

I enabled production Fedora OpenID even on Fedora Copr stage, and it behaves the same way... I think there's some difference between id.stg.fedoraproject.org vs. id.fedoraproject.org.

The weird thing is that our docker container uses production id.fedoraproject.org and doesn't suffer this issue.

@FrostyX
Copy link
Member

FrostyX commented Dec 7, 2022

Deep, deep down in the python openid library, this is being called and timeouts

import urllib.request

url = "http://frostyx.id.fedoraproject.org/"

# On STG server it works, uncomment the variable to make sure
# url = "http://frostyx.id.stg.fedoraproject.org/"

accept = 'text/html; q=0.3, application/xhtml+xml; q=0.5, application/xrds+xml'
headers = {
    'Accept': accept,
    'User-Agent': 'python-openid/3.1.0 (linux) Python-urllib/3.11',
}

req = urllib.request.Request(url, data=None, headers=headers)
resp = urllib.request.urlopen(req)
print(resp.status)

Edit: Timeouts on our Copr servers. It works fine on my laptop.

Edit 2: It doesn't actually timeout, but takes a very long time (and thus causing the login timeout)

[root@copr-fe-dev ~][STG]# time python /home/jkadlcik/openid-reproducer.py 
200

real    2m9.666s

Edit 3: Don't think it is caused by urllib, requests suffers the same issue

import requests
resp = requests.get(url, headers=headers)

Edit 4: So the minimal reproducer is this

python -c 'import requests; requests.get("http://frostyx.id.fedoraproject.org/")'

And here is where it all ties to IPv6:

[root@copr-fe-dev ~][STG]# curl http://frostyx.id.fedoraproject.org/ -v
*   Trying 2600:1f14:fad:5c02:3556:f9be:1874:bdc0:80...
*   Trying 38.145.60.20:80...
* Connected to frostyx.id.fedoraproject.org (38.145.60.20) port 80 (#0)

See, it tried the IPv6 first, but it immediately timeouted and continued to IPv4, and therefore curl worked just fine. Now wget ...

[root@copr-fe-dev ~][STG]# wget http://frostyx.id.fedoraproject.org/
--2022-12-07 19:09:52--  http://frostyx.id.fedoraproject.org/
Resolving frostyx.id.fedoraproject.org (frostyx.id.fedoraproject.org)... 2600:1f14:fad:5c02:3556:f9be:1874:bdc0, 2600:2701:4000:5211:dead:beef:fe:fed3, 2605:bc80:3010:600:dead:beef:cafe:fed9, ...
Connecting to frostyx.id.fedoraproject.org (frostyx.id.fedoraproject.org)|2600:1f14:fad:5c02:3556:f9be:1874:bdc0|:80...

And it hangs forever ... Doing wget -4 http://frostyx.id.fedoraproject.org/ works just fine.

In all cases, IPv6 is prioritized for some reason, and our python reproducer fails because of that. Proof:

import requests
requests.packages.urllib3.util.connection.HAS_IPV6 = False
resp = requests.get(url)
print(resp)

This works without any issues.

And I hopefully fixed the issue now. I created this file:

# See https://www.youtube.com/watch?v=9qIBUt-VMds
[root@copr-fe ~][PROD]# cat /etc/gai.conf 
precedence ::ffff:0:0/96  100

and systemctl restart httpd. Seems to work.

@praiskup
Copy link
Member Author

praiskup commented Dec 7, 2022

Good job @FrostyX!

[root@copr-fe-dev ~][STG]# wget http://frostyx.id.fedoraproject.org/
--2022-12-07 19:09:52--  http://frostyx.id.fedoraproject.org/
Resolving frostyx.id.fedoraproject.org (frostyx.id.fedoraproject.org)... 2600:1f14:fad:5c02:3556:f9be:1874:bdc0, 2600:2701:4000:5211:dead:beef:fe:fed3, 2605:bc80:3010:600:dead:beef:cafe:fed9, ...
Connecting to frostyx.id.fedoraproject.org (frostyx.id.fedoraproject.org)|2600:1f14:fad:5c02:3556:f9be:1874:bdc0|:80...

And it hangs forever ... Doing wget -4 http://frostyx.id.fedoraproject.org/ works just fine.

This seems like the FAS ipv6 is broken, because ipv6 just works on our server. Can you please report to the infra team?

@FrostyX
Copy link
Member

FrostyX commented Dec 12, 2022

I have submitted some feedback for the infra team in
https://pagure.io/fedora-infrastructure/issue/11025

Kevin says that they are planning to add IPv6 to iad2 early next year. Until then, I added the modified gai.conf into the ansible playbook, and consider this fixed.

@FrostyX FrostyX closed this as completed Dec 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants