transport: native sockets on iOS/Android

[junr03]

Using native sockets on iOS/Android is the recommended approach for performing low-level networking. We should investigate:

• What benefits we can get from using native sockets/APIs
• The work required to use these APIs, both within Envoy Mobile and upstream Envoy

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[junr03]

Current proposal:

• Envoy uses BSD sockets for sending/receiving data. On iOS, there are known issues that arise with this approach (such as these with gRPC). To work around these problems, Envoy will need to support using the native networking stack to send/receive data in place of the built-in mechanism
• Note: We’ll need to take into account TLS support (we should likely use TLS from Envoy instead of native)
• Implement a socket provider on iOS that will replace the existing BSD socket usage in Envoy. This will need further investigation, but can likely be accomplished through CFStream, Network.framework, etc.

[rebello95]

Potentially related, here are some key announcements Apple made regarding networking at WWDC earlier this month: https://github.com/Blackjacx/WWDC#advances-in-networking-part-1 (parts 1 and 2)

[conradev]

You'll definitely want to use Network.framework as the socket provider on iOS – it works across all of Apple's platforms. BSD sockets do not work on Apple Watch, for example.

[sayrer]

an even simpler test: iirc, BSD sockets don't handle cellular/wifi transitions, but Open Transport and Network.framework do. Other areas to check might be multipath TCP support and VPNs.

[mattklein123]

For watchers the main work items here are:

1. Slightly refactor the TLS transport socket to allow for swapping out the actual read/write/connection APIs
2. Create a new iOS transport socket that derives from the upstream TLS transport socket and uses appropriate iOS APIs for the underlying connections.

This should be pretty straightforward given existing Envoy extensibility mechanisms.

[rebello95]

Detailed write-up of the current state of the transport layer issues on iOS in this comment: #128 (comment)

[junr03]

Current proposal doc.

[rebello95]

Related: #541

[goaway]

https://docs.google.com/document/d/16jq5YO542BbAnBGwldgPfaJel9H3K6ufWD7Lqvrx0Bg/edit?usp=sharing

The above doc captures the current state of the world with copious linked references and examples. In brief, our current usage of POSIX sockets on both platforms, with our current workarounds, presents no serious general-case issues. There are some somewhat niche cases (e.g. on-demand VPN on iOS) which could be better addressed.

In the medium term, implementing socket providers using the newer platform APIs would yield some incremental benefit, but more importantly, help to future-proof the library.

[conradev]

While there are no general issues, Multipath TCP could yield pretty significant gains:

Last year we announced that in addition to Siri, we'd also enabled Multipath TCP for Apple Music. Since then, we've seen a 13 percent reduction in music stalls and in the event that a stall occurs, there has been a 22 percent reduction in the stall duration. https://developer.apple.com/videos/play/wwdc2020/10111/

(It might even be more pronounced for Lyft because users are leaving/entering WiFi range all the time)