Network socket changes #136

rw-r-r-0644 · 2020-05-22T12:01:53Z

Opening this PR to keep track of wutsocket changes/issues/suggestions.

The wutsocket library provides standard network headers and a devoptab for the nsysnet socket library.

Status:

Currently the wutsocket library appears to work, though it was only tested with a limited number of applications and more testing is probably required.

Potential issues:

The wutsocket library is currently initialized from wutcrt, and internally handles nn_ac and the nsysnet socket library initialization. This is not ideal as some applications might not need to use the network, and the AC initialization might require some time (this last issue can partially be resolved using async initialization).
Also the nsysnet retrocompatibility layer, while present, could potentially cause issues with a limited number of applications (for example apps manually defining nsysnet error codes, which won't match the returned errno codes).

Any suggestion regarding those or other possible issues is welcome

…rary

ashquarky · 2020-05-22T12:07:37Z

ACConnectWithConfigIdAsync does exist, though it's not in Decaf or wut's headers atm. May need some light reverse-engineering work, with any luck it's just passing a callback through to IoctlAsync with no extra weirdness on top.

ashquarky

Looks pretty good, all up! I'm not convinced about having all the socket headers in include/ and include/sys/. Did you write them yourself? Might be a little more maintainable to nick 'em from BSD, like what libnx does (they also bury it all under external/). Just minor nitpicks, though!

include/nsysnet/_socket.h

ashquarky · 2020-05-22T12:15:06Z

libraries/wutcrt/wut_crt.c

@@ -2,11 +2,13 @@ void __init_wut_malloc();
 void __init_wut_newlib();
 extern void __init_wut_stdcpp() __attribute__((weak));
 void __init_wut_devoptab();
+void __init_wut_socket();


These could potentially be weak functions? Could be one way around the AC issue

Definitively an option; waiting for more comments on this

libraries/wutsocket/wut_socket.c

libraries/wutsocket/wut_socket_common.c

Theorically this shouldn't happen

rw-r-r-0644 · 2020-05-22T13:28:55Z

Looks pretty good, all up! I'm not convinced about having all the socket headers in include/ and include/sys/. Did you write them yourself? Might be a little more maintainable to nick 'em from BSD, like what libnx does (they also bury it all under external/). Just minor nitpicks, though!

Thanks for the suggestions! 👍

Not sure about the bsd socket headers; while they are what libnx uses, other platforms, such as libctru, also provide their own headers. They work very well for the Switch since the os provides an almost complete bsd sockets implementation, but on the wiiu many of the additional symbols provided by those headers are not available/wouldn't have any effect, so they might result in unnecessary clutter. Also as nintendo won't update nsysnet anymore, we probably wouldn't really need that much maintainance for the headers after they are working

rw-r-r-0644 · 2020-05-22T13:46:46Z

Just realized we might have one additional issue related to these changes: libcurl allows applications to interact with the network sockets used by the library, which for nintendo's libcurl port would be coming from the nsysnet library rather than newlib.
It could make sense to replace nintendo's libcurl altogether and provide a libcurl and mbedtls packages as with the switch or the 3ds. The builtin libcurl port is outdated, uses NSSL for https, and won't work with anything newer than TLSv1.0 (so most https websites are currently not working)

ashquarky · 2020-05-23T00:36:28Z

Not sure about the bsd socket headers; while they are what libnx uses, other platforms, such as libctru, also provide their own headers. They work very well for the Switch since the os provides an almost complete bsd sockets implementation, but on the wiiu many of the additional symbols provided by those headers are not available/wouldn't have any effect, so they might result in unnecessary clutter. Also as nintendo won't update nsysnet anymore, we probably wouldn't really need that much maintainance for the headers after they are working

This is a fair point! Just wanted to make sure we're following standards here. POSIX all the way! Maybe there's some kind of test suite we could run these headers against? Maybe I'm just nervous, I dunno.

Just realized we might have one additional issue related to these changes: libcurl allows applications to interact with the network sockets used by the library, which for nintendo's libcurl port would be coming from the nsysnet library rather than newlib.
It could make sense to replace nintendo's libcurl altogether and provide a libcurl and mbedtls packages as with the switch or the 3ds. The builtin libcurl port is outdated, uses NSSL for https, and won't work with anything newer than TLSv1.0 (so most https websites are currently not working)

I agree that a new libcurl is a better long-term solution, but mbedtls is kinda annoying (I've tried messing with it before, see yawut/mbedtls) so colour me unconvinced we'll be able to pull it off quickly. As a shorter-term solution, wutsocket could rplwrap the relevant libcurl functions too? I'd imagine the actual code would be super trivial, just looking for the conditions where libcurl gives an fd and passing it through the appropriate lookup tables.

exjam · 2020-05-28T18:45:36Z

Yes both some curl methods and also NSSLCreateConnection takes a socket file descriptor.

BullyWiiPlaza · 2020-11-10T11:18:32Z

It could make sense to replace nintendo's libcurl altogether and provide a libcurl and mbedtls packages as with the switch or the 3ds. The builtin libcurl port is outdated, uses NSSL for https, and won't work with anything newer than TLSv1.0 (so most https websites are currently not working)

Yeah. Any progress on updating the HTTPS support for libcurl (using NSSL or something else) to work seemlessly from a developer's point of view like when the same code is compiled for Windows/Linux? I tried to use it on Wii U and it didn't work as you said but it would be great if that was possible. Only being able to connect to HTTP websites is almost useless. After all, Wii U games are communicating over SSL just fine so the code is definitely there already.

fincs · 2021-03-07T12:10:40Z

I'm going to have a go at finishing this PR.

GaryOderNichts · 2021-03-07T12:21:05Z

I added a few more functions in my fork here: https://github.com/GaryOderNichts/wut/tree/wutsocket
Might be useful for finishing this PR.
These changes are enough to build the latest version of curl, which is why I removed the system curl.

fincs · 2021-03-07T12:23:07Z

Good to know. I'm first going to try implementing the weak linking mechanism suggested in one of the conversations above, though. In the mean time, can you PR some of your other fixes? I saw a gettod fix that is unrelated to socket stuff in your branch.

GaryOderNichts · 2021-03-07T12:31:32Z

Yeah, will do.

ashquarky and others added 8 commits May 22, 2020 12:16

cafe: Move nsysnet BSD symbols to rplwrap

4f4d3e5

nsysnet: Rename to _socket, change to rplwrap names (untested)

b899a6d

nsysnet: add back select() and create relative structures

220e301

Initial wutsocket changes

84d7303

nn_ac: Add ACConnect and ACIsApplicationConnected

1a4910d

wutsocket: update nn_ac initialization

7df90f4

wutsocket: formatting fixes

fd5f3e4

nsysnet/socket.h: Add retrocompatibility layer for the new socket lib…

de1b4c3

…rary

ashquarky added devkitPPC interop enhancement help wanted labels May 22, 2020

ashquarky reviewed May 22, 2020

View reviewed changes

rw-r-r-0644 added 2 commits May 22, 2020 14:43

wutsocket: replace NSYSNET_C with RPLWRAP

99c5bd3

wutsocket: handle socketlasterr negative results

74c329d

Theorically this shouldn't happen

wutsocket: Rename soc devoptab to sock

b9ace56

rw-r-r-0644 added 2 commits May 29, 2020 16:25

nsysnet/nssl.h: Import NSSLErrors from Decaf

5fe963a

wutsocket: Pass nsysnet fd to NSSLCreateConnection

28b76e8

ashquarky mentioned this pull request Jun 12, 2020

FD_SETSIZE redefined #93

Closed

WinterMute mentioned this pull request Jun 26, 2020

Conflict between devkitPPC ppc-zlib and wut zlib #143

Closed

fincs changed the base branch from master to socket March 7, 2021 12:08

fincs marked this pull request as ready for review March 7, 2021 12:08

fincs merged commit c377cff into devkitPro:socket Mar 7, 2021

GaryOderNichts mentioned this pull request Mar 7, 2021

wutsocket: add more missing functions #168

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Network socket changes #136

Network socket changes #136

rw-r-r-0644 commented May 22, 2020

ashquarky commented May 22, 2020

ashquarky left a comment

ashquarky May 22, 2020

rw-r-r-0644 May 22, 2020

rw-r-r-0644 commented May 22, 2020

rw-r-r-0644 commented May 22, 2020

ashquarky commented May 23, 2020

exjam commented May 28, 2020

BullyWiiPlaza commented Nov 10, 2020 •

edited

fincs commented Mar 7, 2021

GaryOderNichts commented Mar 7, 2021 •

edited

fincs commented Mar 7, 2021 •

edited

GaryOderNichts commented Mar 7, 2021

Network socket changes #136

Network socket changes #136

Conversation

rw-r-r-0644 commented May 22, 2020

ashquarky commented May 22, 2020

ashquarky left a comment

Choose a reason for hiding this comment

ashquarky May 22, 2020

Choose a reason for hiding this comment

rw-r-r-0644 May 22, 2020

Choose a reason for hiding this comment

rw-r-r-0644 commented May 22, 2020

rw-r-r-0644 commented May 22, 2020

ashquarky commented May 23, 2020

exjam commented May 28, 2020

BullyWiiPlaza commented Nov 10, 2020 • edited

fincs commented Mar 7, 2021

GaryOderNichts commented Mar 7, 2021 • edited

fincs commented Mar 7, 2021 • edited

GaryOderNichts commented Mar 7, 2021

BullyWiiPlaza commented Nov 10, 2020 •

edited

GaryOderNichts commented Mar 7, 2021 •

edited

fincs commented Mar 7, 2021 •

edited