myproxy FTBFS on IPV6-only build servers #160

ellert · 2021-08-19T11:29:17Z

Forwarded from:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=992436

Anyone has any idea how to fix this?

fscheiner · 2021-08-19T12:08:08Z

Strange. Comparing the Debian build log to my log on the OBS (though for MyProxy v6.2.6, but the changes in #146 and #147 don't look like they changed something relevant), it looks to me, like there's just no service listening on the IPv6 localhost address on the Debian build host, because the server seems to listen on the IPv4 localhost address:

[...]
Attempting to connect to ::1:45721 
Attempting to connect to 127.0.0.1:45721 
Successfully connected to localhost:45721
[...]

Maybe myproxy-server (commandline in https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L269) doesn't listen on IPv6 addresses by default, when the env var MYPROXY_SERVER is configured to localhost only (done in https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L243). On Ubuntu 20.04 for example the IPv6 localhost address has the following aliases:

ip6-localhost
ip6-loopback

Not sure if (1) we just would need to add one of these to MYPROXY_SERVER (@jbasney: Can this be done and what separator is needed here?) or (2) we need to add additional -l [...] arguments to the server command in https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L269) to solve this.

fscheiner · 2021-08-19T12:14:06Z

Additonal info:

Failing tests are 3T, 20T and 41 (https://github.com/gridcf/gct/blob/master/myproxy/source/myproxy-test#L386)
"Bootstrapping MyProxy server root of trust." comes from

gct/myproxy/source/myproxy.c

Line 497 in 5882044

myproxy_bootstrap_trust(myproxy_socket_attrs_t *attrs)
"Attempting to connect to [...]" comes from

gct/myproxy/source/myproxy.c

Line 322 in 5882044

static int connect_socket_to_host(char *host, int port) {

jbasney · 2021-08-19T12:45:18Z

MYPROXY_SERVER can be a comma-separated list of hostnames. That tells clients where to connect.

gct/myproxy/source/myproxy.c

Line 387 in 246f1c3

static int get_connected_myproxy_host_socket(char *hostlist, int port) {

The myproxy-server may only listen on IPv4 by default.

gct/myproxy/source/myproxy_server.c

Line 1124 in 246f1c3

myproxy_debug("creating IPv4 listen socket without binding");

Otherwise, the myproxy-server --listen option can only specify a single hostname.

gct/myproxy/source/myproxy_server.c

Line 1017 in 246f1c3

bind_socket(const char *hostname, int port)

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Running myproxy-test -verbose may give additional clues.

fscheiner · 2021-08-19T13:42:27Z

MYPROXY_SERVER can be a comma-separated list of hostnames. That tells clients where to connect.

gct/myproxy/source/myproxy.c

Line 387 in 246f1c3

static int get_connected_myproxy_host_socket(char *hostlist, int port) {

What I don't understand is, why the client tries both IPv4 and IPv6 localhost addresses with a single localhost in MYPROXY_SERVER. Aha, because of this in /etc/hosts (checked on Raspberry Pi OS/Raspbian):

127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
[...]

And here is why: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=686970

Here's what other distributions do:

CentOS 7, CentOS 8 and Fedora 34 use:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

So our builds at Travis CI would actually have the same issue, if they'd use IPv6 in addition to IPv4.

openSUSE Leap 15.2 uses:

127.0.0.1	localhost

# special IPv6 addresses
::1             localhost ipv6-localhost ipv6-loopback

Ubuntu 18.04.x and 20.04.x use:

127.0.0.1	localhost
[...]
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback

But IMHO I'd have expected localhost to resolve to a singular address.

Otherwise, the myproxy-server --listen option can only specify a single hostname.

Can --listen then be used mutliple times? Possibly not: https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L963..L965

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Do the other tests try to connect to all localhost addresses the same way as the 3T test? If not, then I assume myproxy-server just creates IPv4 sockets for all (IPv4) addresses the host has (see https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L937..L938).

ellert · 2021-08-19T13:56:10Z

Complete logs are available here:
https://buildd.debian.org/status/logs.php?pkg=myproxy&ver=6.2.8-1&arch=amd64
Both attempts say:

MyProxy Tests Complete: 53 tests passed, 3 tests failed

All 3 failed test say:
trust root bootstrap failed
Unable to connect to ::1: .......
Connection refused

jbasney · 2021-08-19T13:58:24Z

Can --listen then be used multiple times?

No, but myproxy-server --listen localhost should bind to all localhost addresses just like MYPROXY_SERVER=localhost tries to connect to all localhost addresses. Adding --listen localhost to the myproxy-server command in the myproxy-test script may be a fix for this issue.

Do the other tests try to connect to all localhost addresses the same way as the 3T test?

I think so, which is why I'm struggling to identify why only the 3T test fails. There must be some difference that I'm missing.

ellert · 2021-08-19T14:24:40Z

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Could the problem be get_connected_myproxy_host_socket()? It tries to split a host:port string on a colon, but maybe it gets confused with an IPv6 host that has lots of colons in it (like e.g. ::1).

I still don't get why only 3 tests fail though...

fscheiner · 2021-08-19T14:27:22Z

@ellert:

Complete logs are available here:
https://buildd.debian.org/status/logs.php?pkg=myproxy&ver=6.2.8-1&arch=amd64
Both attempts say:

MyProxy Tests Complete: 53 tests passed, 3 tests failed

All 3 failed test say:
trust root bootstrap failed
Unable to connect to ::1: .......
Connection refused

Thanks for clarification, I was too quick and didn't check the whole log but just the first occurance of an error, sorry. :-/ I'll update #160 (comment) accordingly.

@jbasney: So I assume all tests where a client tries to connect to the server using the address in MYPROXY_SERVER which points to both 127.0.0.1 and ::1 on Debian seem to fail.

Can --listen then be used multiple times?

No, but myproxy-server --listen localhost should bind to all localhost addresses just like MYPROXY_SERVER=localhost tries to connect to all localhost addresses. Adding --listen localhost to the myproxy-server command in the myproxy-test script may be a fix for this issue.

But it should already per default listen on all addresses - at least that's what I understand from https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L937..L938 - and the myproxy-test script doesn't use --listen nor anything address related in the configuration file.

@ellert

Since only MyProxy Test 3T fails, I think there's not a problem in general with the clients connecting to the server, but something specific to myproxy_bootstrap_trust() though that uses get_connected_myproxy_host_socket() just like the other connections, so I'm not sure what's causing a 'Connection refused' error only in that case.

Could the problem be get_connected_myproxy_host_socket()? It tries to split a host:port string on a colon, but maybe it gets confused with an IPv6 host that has lots of colons in it (like e.g. ::1).

So you assume that the myproxy-server is actually listening on ::1, too?

fscheiner · 2021-08-19T14:35:05Z

@ellert:
But:

Unable to connect to ::1:45721 error:0200206F:system library:connect:Connection refused error:2008A067:BIO routines:BIO_connect:connect error error:0200206F:system library:connect:Connection refused hostname=localhost service=45721 error:20073067:BIO routines:conn_state:connect error error:140E0197:SSL routines:SSL_shutdown:shutdown while in init 
Connection refused

...seems to correctly identify the port used (service=45721).

jbasney · 2021-08-19T14:53:52Z

There's a subtle difference between myproxy-server --listen localhost and myproxy-server without any --listen option. For better or worse, my thinking at the time when I added IPv6 support was not to enable it by default for fear of breaking things but to require an explicit --listen option for server admins who knew they wanted IPv6 enabled.

Without any --listen option, the myproxy-server binds with AF_INET (and not AF_INET6):

gct/myproxy/source/myproxy_server.c

Line 1125 in 246f1c3

listen_sock = socket(AF_INET, SOCK_STREAM, 0);

With a --listen option, the myproxy-server binds with AF_UNSPEC so that IPv4 and/or IPv6 will be enabled based on the hostname provided:

gct/myproxy/source/myproxy_server.c

Line 1036 in 246f1c3

hints.ai_family = AF_UNSPEC;

I still don't understand why only the 3 "bootstrap" test cases are failing, but I still think changing the myproxy-test script to explicitly pass a myproxy-server --listen localhost option may fix it.

fscheiner · 2021-08-19T14:56:10Z

@jbasney
Sounds good. We're in a VC now, so please don't mind us not replying for some time.

fscheiner · 2021-08-19T19:26:35Z

Small update: For me make myproxy-check works through w/o an issue on openSUSE Leap 15.2 with the /etc/hosts configuration mentioned in #160 (comment) (gct at 246f1c3, configured using ./configure --disable-gridftp --disable-gram5 --enable-myproxy):

johndoe@gridftp-5:~/git-projects/gct> make myproxy-check
make[1]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
Making check in web
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source/web'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source/web'
Making check in systemd
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source/systemd'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source/systemd'
Making check in man
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source/man'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source/man'
make[2]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
make  myproxy-test-wrapper
make[3]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
make[3]: Nothing to be done for 'myproxy-test-wrapper'.
make[3]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make  check-TESTS
make[3]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
make[4]: Entering directory '/home/johndoe/git-projects/gct/myproxy/source'
PASS: myproxy-test-wrapper
============================================================================
Testsuite summary for myproxy 6.2.8
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[4]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make[3]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make[2]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'
make[1]: Leaving directory '/home/johndoe/git-projects/gct/myproxy/source'

johndoe@gridftp-5:~/git-projects/gct> cat myproxy/source/myproxy-test-wrapper.log
[...]
MyProxy Test 3T (retrieve stored credential w/ trustroots): SUCCEEDED
[...]
MyProxy Test 20T (retrieve stored credential w/ trustroots): SUCCEEDED
[...]
MyProxy Test 41 (retrieve trustroots w/o authentication): SUCCEEDED
[...]
MyProxy Tests Complete: 56 tests passed, 0 tests failed
PASS myproxy-test-wrapper (exit status: 0)

But when checking network connections with watch -n 0.1 "netstat -plantu | grep myproxy" I only ever saw myproxy-server listening to 0.0.0.0 which matches INADDR_ANY from https://github.com/gridcf/gct/blob/246f1c3f6b3f0fbff080c1d73c6a60dc418cf183/myproxy/source/myproxy_server.c#L937..L938. Why the clients don't complain for me the same way as on Debian, although localhost points to both 127.0.0.1 and ::1 on that host is the question.

fscheiner · 2021-09-03T17:17:46Z

Fixed in GCT 6.2.20210826 maintenance release.

ellert mentioned this issue Aug 20, 2021

Use -l (--listen) flag when starting myproxy-server in test scripts #161

Merged

ellert added the fixed in git master label Aug 23, 2021

fscheiner closed this as completed Sep 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

myproxy FTBFS on IPV6-only build servers #160

myproxy FTBFS on IPV6-only build servers #160

ellert commented Aug 19, 2021

fscheiner commented Aug 19, 2021

fscheiner commented Aug 19, 2021 •

edited

jbasney commented Aug 19, 2021 •

edited

fscheiner commented Aug 19, 2021

ellert commented Aug 19, 2021

jbasney commented Aug 19, 2021

ellert commented Aug 19, 2021

fscheiner commented Aug 19, 2021

fscheiner commented Aug 19, 2021

jbasney commented Aug 19, 2021

fscheiner commented Aug 19, 2021

fscheiner commented Aug 19, 2021

fscheiner commented Sep 3, 2021

myproxy FTBFS on IPV6-only build servers #160

myproxy FTBFS on IPV6-only build servers #160

Comments

ellert commented Aug 19, 2021

fscheiner commented Aug 19, 2021

fscheiner commented Aug 19, 2021 • edited

jbasney commented Aug 19, 2021 • edited

fscheiner commented Aug 19, 2021

ellert commented Aug 19, 2021

jbasney commented Aug 19, 2021

ellert commented Aug 19, 2021

fscheiner commented Aug 19, 2021

fscheiner commented Aug 19, 2021

jbasney commented Aug 19, 2021

fscheiner commented Aug 19, 2021

fscheiner commented Aug 19, 2021

fscheiner commented Sep 3, 2021

fscheiner commented Aug 19, 2021 •

edited

jbasney commented Aug 19, 2021 •

edited