[GSoC 2021] Socket Library

[king-11]

This PR adds an implementation of the Socket library to modules/packages.

This first version has some caveats:

• requires libevent 2.1 or newer
• requires CHPL_TASKS=qthreads right now (fifo doesn't work yet)
• requires the C backend right now (CHPL_TARGET_COMPILER=llvm doesn't work yet)
• requires libevent be available in /usr/include/event2

Design

#17899

Future Work

• DNS Resolution using libevent
• IO Plugin using libevent for non-blocking IO
• sendAll method for UDP and TCP on lines of python's sendAll
• Make work with LLVM and FIFO
• Separate Module for Libevent

[ankushbhardwxj]

Did a review pass mostly over TCP part. Looks good. Made some inline comments.
@king-11 Looking forward for tests. For documentation, I believe you'll be writing doc-strings as comments over the functions. Also, please resolve merge conflicts.

[mppf]

The way it is written now it looks like it might send only some bytes but not all of the passed bytes.
To send all of the passed bytes I think you would have to have a loop that tries again if only say 1 byte is sent by the sendto call at a time.

[king-11]

we can have another function like sendAll which could do that because with udp we are working with packets instead of loads of data so if user wants to send large data they can use sendAll. ref: This is how python does it https://docs.python.org/3/library/socket.html#socket.socket.send

[mppf]

Right. I'd support a sendAll and also a comment here along the lines of this from those Python docs

Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data. For further information on this topic, ...

[king-11]

Marked as future work

[king-11]

@e-kayrakli @mppf changes were made as per review.

[mppf]

We need to move the module from modules/standard/ to modules/packages and likewise the tests.

We will need a test/packages/Socket.skipif file. See also https://chapel-lang.org/docs/main/developer/bestPractices/TestSystem.html#limiting-where-the-test-runs or look for examples in test/ with a name ending in .skipif. This skipif file needs to skip the testing:

• if libevent is not present or if it is too old
• for CHPL_TASKS!=fifo
• for CHPL_TARGET_COMPILER=llvm

I'm seeing C compilation failures on a system with libevent 2.0. We need to clearly document in the PR message and the module documentation the current caveats:

• requires libevent 2.1 or newer (at least, the errors I see in 2.0 are for features added in 2.1)
• requires fifo and the C backend right now
• requires libevent be available in /usr/include/event2

On another system where the tests compile, I'm seeing timeouts. @king-11 are the tests working for you?

[e-kayrakli]

I looked at my notes in the PR and the recent commits, and they look good. Thanks @king-11!

[king-11]

@mppf yes the tests are compiling and executing I think there can be issues with the send_http one in tcpio.chpl because it connects to an external server which can reject our requests too maybe so it can be the flaky one but we certainly need it to test the remote connection. So can you try running it again

[mppf]

@king-11 I don't think we should be connecting to google.com to run these tests. Instead, I think we should run a local Python webserver for the purposes of that test, or maybe remove it entirely. You can see an example of running our own webserver in test/library/packages/Curl/RunServer.chpl .

[mppf]

I am still seeing timeouts. tcpio.chpl worked for me when I tested it on its own but now I am seeing connectblocking.chpl time out. Perhaps before I was not using C backend + qthreads.

[king-11]

@mppf below are the updates:

• I have updated the module to be a Package instead of Standard Module
• I have added in SKIPIF for whole tests directory by python checking for libevent as a bit hackish not sure if it will work always
• Regarding tests I tired cleaning up my whole repo and making the whole project and running again seems to be running fine 🤔 . I think you are running with FIFO because that is where timeouts are still happening.
• The reason I have kept in the send_http one is to ensure that we are able to connect to a remote machine and that is the easiest way possible otherwise all other options would be interprocess/task.

[mppf]

I tried running on a Mac with Homebrew installed libevent. (To do that I had to change the paths in the require lines in Socket.chpl). I see many failures:

[Executing diff connectblocking.good connectblocking.exec.out.tmp]
2c2
< Flavour: OK
---
> Flavour: ERROR
3a4
> SystemError: Invalid argument (Failed to bind Socket)
6c7
< Flavour: OK
---
> Flavour: ERROR
7a9
> SystemError: Invalid argument (Failed to bind Socket)
10c12
< Flavour: OK
---
> Flavour: FAIL
11a14,22
> AssertionError: Strings differ: 'Invalid argument (connect() failed)' != 'Connection refused (connect() failed)'
> First differing element at index 0:
> 'I'
> 'C'
>
> Second String contains 2 additional elements.
> First extra element is at index 36
> ')'
>
14,24c25,27
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
< test_connection_ipv6_ipaddr()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
< test_fail_ipv6_noserver()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
---
> uncaught IllegalArgumentError: illegal argument 'bad cast from int '30' to enum 'IPFamily': '
>   connectblocking.chpl:51: thrown here
>   connectblocking.chpl:51: uncaught here
[Error matching program output for library/packages/Socket/connectblocking]
[Elapsed time to compile and execute all versions of "library/packages/Socket/connectblocking" - 22.780 seconds]
[test: library/packages/Socket/connecttimeout.chpl]
[Executing compiler /Users/mferguson/w/1/bin/darwin-x86_64/chpl -o connecttimeout --cc-warnings connecttimeout.chpl < /dev/null]
[Elapsed compilation time for "library/packages/Socket/connecttimeout" - 21.866 seconds]
[Success compiling library/packages/Socket/connecttimeout]
[Executing program ./connecttimeout  < /dev/null]
[Elapsed execution time for "library/packages/Socket/connecttimeout" - 0.140 seconds]
[Executing diff connecttimeout.good connecttimeout.exec.out.tmp]
2c2
< Flavour: OK
---
> Flavour: ERROR
3a4
> SystemError: Invalid argument (Failed to bind Socket)
6c7
< Flavour: OK
---
> Flavour: ERROR
7a9
> SystemError: Invalid argument (Failed to bind Socket)
10,16c12,14
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
< test_fail_backlog_ipv6()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
---
> uncaught IllegalArgumentError: illegal argument 'bad cast from int '30' to enum 'IPFamily': '
>   connecttimeout.chpl:42: thrown here
>   connecttimeout.chpl:42: uncaught here
[Error matching program output for library/packages/Socket/connecttimeout]
[Elapsed time to compile and execute all versions of "library/packages/Socket/connecttimeout" - 22.012 seconds]
[test: library/packages/Socket/ipaddr.chpl]
[Executing compiler /Users/mferguson/w/1/bin/darwin-x86_64/chpl -o ipaddr --cc-warnings ipaddr.chpl < /dev/null]
[Elapsed compilation time for "library/packages/Socket/ipaddr" - 22.190 seconds]
[Success compiling library/packages/Socket/ipaddr]
[Executing program ./ipaddr  < /dev/null]
[Elapsed execution time for "library/packages/Socket/ipaddr" - 0.130 seconds]
[Executing diff ipaddr.good ipaddr.exec.out.tmp]
6c6
< Flavour: OK
---
> Flavour: ERROR
7a8
> IllegalArgumentError: Incompatible Address and Family
14c15
< Flavour: OK
---
> Flavour: ERROR
15a17
> IllegalArgumentError: Incompatible Address and Family
22,32c24,26
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
< test_ipaddr_sockaddr_ipv4()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
< test_ipaddr_sockaddr_ipv6()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
---
> uncaught IllegalArgumentError: illegal argument 'bad cast from int '30' to enum 'IPFamily': '
>   ipaddr.chpl:60: thrown here
>   ipaddr.chpl:60: uncaught here
[Error matching program output for library/packages/Socket/ipaddr]
[Elapsed time to compile and execute all versions of "library/packages/Socket/ipaddr" - 22.325 seconds]
[test: library/packages/Socket/ipfamily.chpl]
[Executing compiler /Users/mferguson/w/1/bin/darwin-x86_64/chpl -o ipfamily --cc-warnings ipfamily.chpl < /dev/null]
[Elapsed compilation time for "library/packages/Socket/ipfamily" - 22.178 seconds]
[Success compiling library/packages/Socket/ipfamily]
[Executing program ./ipfamily  < /dev/null]
[Elapsed execution time for "library/packages/Socket/ipfamily" - 0.184 seconds]
[Executing diff ipfamily.good ipfamily.exec.out.tmp]
2c2
< Flavour: OK
---
> Flavour: FAIL
3a4
> AssertionError: assert failed - '10' != '30'
[Error matching program output for library/packages/Socket/ipfamily]
[Elapsed time to compile and execute all versions of "library/packages/Socket/ipfamily" - 22.368 seconds]
[test: library/packages/Socket/listen.chpl]
[Executing compiler /Users/mferguson/w/1/bin/darwin-x86_64/chpl -o listen --cc-warnings listen.chpl < /dev/null]
[Elapsed compilation time for "library/packages/Socket/listen" - 22.597 seconds]
[Success compiling library/packages/Socket/listen]
[Executing program ./listen  < /dev/null]
[Elapsed execution time for "library/packages/Socket/listen" - 0.214 seconds]
[Executing diff listen.good listen.exec.out.tmp]
2c2
< Flavour: OK
---
> Flavour: ERROR
3a4
> SystemError: Invalid argument (Failed to bind Socket)
6,8c7,9
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
---
> uncaught IllegalArgumentError: illegal argument 'bad cast from int '30' to enum 'IPFamily': '
>   listen.chpl:20: thrown here
>   listen.chpl:20: uncaught here
[Error matching program output for library/packages/Socket/listen]
[Elapsed time to compile and execute all versions of "library/packages/Socket/listen" - 22.817 seconds]
[test: library/packages/Socket/networkoptimization.chpl]
[Executing compiler /Users/mferguson/w/1/bin/darwin-x86_64/chpl -o networkoptimization --cc-warnings networkoptimization.chpl < /dev/null]
[Elapsed compilation time for "library/packages/Socket/networkoptimization" - 22.353 seconds]
[Executing diff networkoptimization.good networkoptimization.comp.out.tmp]
1,16c1,10
< test_nagle_tcpListener()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
< test_nagle_tcpConn()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
< test_delayAck_tcpListener()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
< test_delayAck_tcpConn()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
---
> In file included from /var/folders/b6/1lwlz22n5d53878_x52vp3kr0007pg/T//chpl-mferguson.deleteme-6qpTIz/_main.c:60:
> /var/folders/b6/1lwlz22n5d53878_x52vp3kr0007pg/T//chpl-mferguson.deleteme-6qpTIz/Socket.c:1658:46: error: use of undeclared identifier 'TCP_QUICKACK'
> setSockOpt_chpl(socketFd_chpl2, IPPROTO_TCP, TCP_QUICKACK, &c_enable_chpl, &error_chpl, _ln_chpl, _fn_chpl);
>                                              ^
> /var/folders/b6/1lwlz22n5d53878_x52vp3kr0007pg/T//chpl-mferguson.deleteme-6qpTIz/Socket.c:1676:64: error: use of undeclared identifier 'TCP_QUICKACK'
> call_tmp_chpl39 = getSockOpt_chpl(socketFd_chpl2, IPPROTO_TCP, TCP_QUICKACK, &error_chpl, _ln_chpl, _fn_chpl);
>                                                                ^
> 2 errors generated.
> make: *** [/var/folders/b6/1lwlz22n5d53878_x52vp3kr0007pg/T//chpl-mferguson.deleteme-6qpTIz/networkoptimization.tmp] Error 1
> error: compiling generated source
[Error matching compiler output for library/packages/Socket/networkoptimization]
[Elapsed time to compile and execute all versions of "library/packages/Socket/networkoptimization" - 22.358 seconds]
[test: library/packages/Socket/tcpio.chpl]
[Executing compiler /Users/mferguson/w/1/bin/darwin-x86_64/chpl -o tcpio --cc-warnings tcpio.chpl < /dev/null]
[Elapsed compilation time for "library/packages/Socket/tcpio" - 23.848 seconds]
[Success compiling library/packages/Socket/tcpio]
[Executing program ./tcpio  < /dev/null]
[Elapsed execution time for "library/packages/Socket/tcpio" - 0.202 seconds]
[Executing diff tcpio.good tcpio.exec.out.tmp]
2c2
< Flavour: OK
---
> Flavour: ERROR
3a4
> SystemError: Invalid argument (Failed to bind Socket)
6c7
< Flavour: OK
---
> Flavour: ERROR
7a9
> SystemError: Invalid argument (Failed to bind Socket)
10,20c12,14
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
< send_number()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
< send_http()
< Flavour: OK
< ======================================================================
< ----------------------------------------------------------------------
---
> uncaught IllegalArgumentError: illegal argument 'bad cast from int '30' to enum 'IPFamily': '
>   tcpio.chpl:45: thrown here
>   tcpio.chpl:45: uncaught here
[Error matching program output for library/packages/Socket/tcpio]
[Elapsed time to compile and execute all versions of "library/packages/Socket/tcpio" - 24.055 seconds]
[test: library/packages/Socket/udpio_ipv4.chpl]
[Executing compiler /Users/mferguson/w/1/bin/darwin-x86_64/chpl -o udpio_ipv4 --cc-warnings udpio_ipv4.chpl < /dev/null]
[Elapsed compilation time for "library/packages/Socket/udpio_ipv4" - 23.674 seconds]
[Success compiling library/packages/Socket/udpio_ipv4]
[Executing program ./udpio_ipv4  < /dev/null]
[Elapsed execution time for "library/packages/Socket/udpio_ipv4" - 0.128 seconds]
[Executing diff udpio_ipv4.good udpio_ipv4.exec.out.tmp]
2c2
< Flavour: OK
---
> Flavour: ERROR
3a4
> SystemError: Invalid argument (Failed to bind Socket)
6c7
< Flavour: OK
---
> Flavour: ERROR
7a9
> SystemError: Invalid argument (Failed to bind Socket)
10c12
< Flavour: OK
---
> Flavour: ERROR
11a14
> SystemError: Invalid argument (Failed to bind Socket)
14c17
< Flavour: OK
---
> Flavour: ERROR
15a19
> SystemError: Invalid argument (Failed to bind Socket)
22c26
< Flavour: OK
---
> Flavour: ERROR
23a28
> SystemError: Invalid argument (Failed to bind Socket)
26c31
< Flavour: OK
---
> Flavour: ERROR
27a33
> SystemError: Invalid argument (Failed to bind Socket)
[Error matching program output for library/packages/Socket/udpio_ipv4]
[Elapsed time to compile and execute all versions of "library/packages/Socket/udpio_ipv4" - 23.808 seconds]
[test: library/packages/Socket/udpio_ipv6.chpl]
[Executing compiler /Users/mferguson/w/1/bin/darwin-x86_64/chpl -o udpio_ipv6 --cc-warnings udpio_ipv6.chpl < /dev/null]
[Elapsed compilation time for "library/packages/Socket/udpio_ipv6" - 22.775 seconds]
[Success compiling library/packages/Socket/udpio_ipv6]
[Executing program ./udpio_ipv6  < /dev/null]
[Elapsed execution time for "library/packages/Socket/udpio_ipv6" - 0.138 seconds]
[Executing diff udpio_ipv6.good udpio_ipv6.exec.out.tmp]
2c2
< Flavour: OK
---
> Flavour: ERROR
3a4
> IllegalArgumentError: Incompatible Address and Family
6c7
< Flavour: OK
---
> Flavour: ERROR
7a9
> IllegalArgumentError: Incompatible Address and Family
10c12
< Flavour: OK
---
> Flavour: ERROR
11a14
> IllegalArgumentError: Incompatible Address and Family
14c17
< Flavour: OK
---
> Flavour: ERROR
15a19
> IllegalArgumentError: Incompatible Address and Family
18c22
< Flavour: OK
---
> Flavour: ERROR
19a24
> IllegalArgumentError: Incompatible Address and Family
22c27
< Flavour: OK
---
> Flavour: ERROR
23a29
> IllegalArgumentError: Incompatible Address and Family
26c32
< Flavour: OK
---
> Flavour: ERROR
27a34
> IllegalArgumentError: Incompatible Address and Family
[Error matching program output for library/packages/Socket/udpio_ipv6]
[Elapsed time to compile and execute all versions of "library/packages/Socket/udpio_ipv6" - 22.918 seconds]
[Finished subtest "library/packages/Socket" - 206.686 seconds]

[Done with tests - 211114.134852]
[Log file: /Users/mferguson/w/1/test/Logs/mferguson.darwin.log ]

[Test Summary - 211114.134852]
[Error matching program output for library/packages/Socket/connectblocking]
[Error matching program output for library/packages/Socket/connecttimeout]
[Error matching program output for library/packages/Socket/ipaddr]
[Error matching program output for library/packages/Socket/ipfamily]
[Error matching program output for library/packages/Socket/listen]
[Error matching compiler output for library/packages/Socket/networkoptimization]
[Error matching program output for library/packages/Socket/tcpio]
[Error matching program output for library/packages/Socket/udpio_ipv4]
[Error matching program output for library/packages/Socket/udpio_ipv6]
[Warning: SKIPIF error.]

The testing I previously told you about failing connectblocking.chpl is using Ubuntu 21.10. It still seems to be timing out.

Can you use some VMs or Docker images or something to try it on some different systems? While I believe it is working on your system it seems it does not work on other systems. (In fact, of 3 systems I've tried it on, I've seen problems in all 3).

[king-11]

• I tried to run the tests on a new Ubuntu 20.04 VM seems to except that tcpio.chpl is hanging on the VM individually and with all the tests as well.
• The macOS issue I think the numbers that I have hardcoded in IPFamily are responsible

[mppf]

thanks @king-11. I think it's important to get all of the tests passing somewhere reproduce-able before we merge it. Can you look into the tcpio.chpl hang you are seeing?

[mppf]

I've run testing on 2 systems with libevent 2.1.12 and all tests are passing and the skipifs seem to be working as expected. Great! I'm not going to merge today though because I'll be away the rest of the week. We'll consider including it in the 1.25.1 point release.