Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

android: move to targetSdkVersion 30 #2293

Closed
bradfitz opened this issue Jun 30, 2021 · 20 comments · Fixed by tailscale/tailscale-android#21
Closed

android: move to targetSdkVersion 30 #2293

bradfitz opened this issue Jun 30, 2021 · 20 comments · Fixed by tailscale/tailscale-android#21
Labels
L2 Few Likelihood OS-android T8 Crash Issue type

Comments

@bradfitz
Copy link
Member

Clock's ticking:

Starting in November 2021, app updates will be required to target API level 30 or above and adjust for behavioral changes in Android 11.

Currently we're at 29.

One release moved us to 30 and it broke things due to our use of netlink: #2290.

This may or may not be a dup of #1965 (we're broken on Android 12), but I'd like to keep this open separately, specifically tracking the November 2021 deadline for SDK version 30.

/cc @eliasnaur @DentonGentry

@DentonGentry
Copy link
Contributor

fatal error: runBackend: NewUserspaceEngineAdvanced: route ip+net: netlinkrib: permission denied

Appears to be golang/go#40569
With API version 30:

  • Apps cannot use the bind() function on NETLINK_ROUTE sockets.
  • Apps cannot send RTM_GETLINK messages.

@bradfitz
Copy link
Member Author

bradfitz commented Jul 1, 2021

Good find! The Catfriend1/syncthing-android#800 link from there is interesting.

@DentonGentry
Copy link
Contributor

net.InterfaceAddrs() calls interfaceAddrTable(nil)
https://cs.opensource.google/go/go/+/refs/tags/go1.16.6:src/net/interface.go;l=115

interfaceAddrTable(nil) calls syscall.NetlinkRIB(syscall.RTM_GETADDR, syscall.AF_UNSPEC)
https://cs.opensource.google/go/go/+/refs/tags/go1.16.6:src/net/interface_linux.go;l=120

NetlinkRIB gets permission denied with targetSdkVersion 30.

@DentonGentry DentonGentry added T8 Crash Issue type and removed T3 Performance/Debugging Issue type labels Jul 15, 2021
@DentonGentry
Copy link
Contributor

No progress on golang/go#40569

As of August 1, it is no longer possible to release new Android apps written in Go. The Play Store will require new apps to target SDK version 30.

@bradfitz
Copy link
Member Author

bradfitz commented Aug 2, 2021

As of August 1, it is no longer possible to release new Android apps written in Go.

That's a little exaggerated. :) The limitation is around use of the netlink calls that implement https://pkg.go.dev/net#Interfaces etc, which few Go programs do.

But it's certainly bad for us.

The short-term fix for us is to ignore whatever Go's doing and use our own net.Interfaces-like wrapper that can use net.Interfaces on most platforms but uses Java/JNI on Android, like we did for tailscale/tailscale-android@b97cc70 ... blech, but doable.

@DentonGentry
Copy link
Contributor

DentonGentry commented Aug 14, 2021

Iterating over each Interface from https://developer.android.com/reference/java/net/NetworkInterface#getNetworkInterfaces() looks like it would work.

Will hold off until a bit closer to November, hoping that golang/go#40569 will be fixed before then.

@DentonGentry
Copy link
Contributor

I have an Android implementation using getNetworkInterfaces() and getInetAddresses() to retrieve the information needed to reconstruct Go's net.Interfaces() - except for the HardwareAddr on the Interface, which is the bit of information that Android SDK 30 is preventing access to.

It can be seen at tailscale/tailscale-android#21. Passing primitive types across JNI is relatively straightforward, passing a single object of a complex class is annoying but still possible, but passing lists and other more complex data structures is way harder. As such, I added a Java routine to render the interface information to a string and pass that across JNI as a primitive type for Go code to parse.


I'm less clear what to do in OSS net/interface/interfaces.go. I don't think we can have an Android-specific implementation return an actual net.Interface: the Go implementation handles interface.Addrs() using the NETLINK messages which Android SDK30 rejects. I guess we need a new interface{}, provide a trivial implementation of the interface{} using net.Interface, and allow the Android platform to supply a different one.

bradfitz added a commit that referenced this issue Oct 4, 2021
Updates #2293

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
@bradfitz
Copy link
Member Author

bradfitz commented Oct 5, 2021

@DentonGentry, I sent #2990

@DentonGentry
Copy link
Contributor

Annoyingly, even with #2990 and tailscale/tailscale-android#21 applied the app still fails with route ip+net: netlinkrib: permission denied

10-05 19:35:04.216  5140  5173 I com.tailscale.ipn: logtail started
10-05 19:35:04.217  5140  5175 I com.tailscale.ipn: goSetupLogs: success
10-05 19:35:04.232  5140  5175 I com.tailscale.ipn: getInterfaces: parsing "rmnet_data0 10 2000 true false false false false | fe80::4059:dc16:7ed3:9c6e%rmnet_data0/64"
10-05 19:35:04.232  5140  5175 I com.tailscale.ipn: getInterfaces: parsing "dummy0 3 1500 true false false false false | fe80::1450:5cff:fe13:f891%dummy0/64"
10-05 19:35:04.232  5140  5175 I com.tailscale.ipn: getInterfaces: parsing "wlan0 30 1500 true true false false true | fe80::2f60:2c82:4163:8389%wlan0/64,2602:248:7b4a:ff60:837b:515b:34ec:a4ea/64,2602:248:7b4a:ff60:4ce7:1788:fdec:3e62/64,10.1.10.131/24"
10-05 19:35:04.232  5140  5175 I com.tailscale.ipn: getInterfaces: parsing "r_rmnet_data0 21 1500 true false false false false | fe80::9318:6093:d1ad:ba7f%r_rmnet_data0/64"
10-05 19:35:04.232  5140  5175 I com.tailscale.ipn: getInterfaces: parsing "rmnet_data2 12 1500 true false false false false | fe80::3c8c:44dc:46a9:9907%rmnet_data2/64,2607:fb90:9d5b:389e:3c8c:44dc:46a9:9907/64"
10-05 19:35:04.232  5140  5175 I com.tailscale.ipn: getInterfaces: parsing "r_rmnet_data1 22 1500 true false false false false | fe80::b6cd:5cb0:8ae6:fe92%r_rmnet_data1/64"
10-05 19:35:04.232  5140  5175 I com.tailscale.ipn: getInterfaces: parsing "rmnet_data1 11 1500 true false false false false | fe80::51f2:ee00:edce:d68b%rmnet_data1/64,2607:fc20:9cb8:1251:51f2:ee00:edce:d68b/64"
10-05 19:35:04.232  5140  5175 I com.tailscale.ipn: getInterfaces: parsing "lo 1 65536 true false true false false | ::1/128,127.0.0.1/8"
10-05 19:35:04.233  5140  5175 I com.tailscale.ipn: getInterfaces: parsing "v4-rmnet_data2 68 1472 true true false true true | 192.0.0.4/32"
10-05 19:35:04.233  5140  5175 I com.tailscale.ipn: fatal error: runBackend: NewUserspaceEngineAdvanced: route ip+net: netlinkrib: permission denied
10-05 19:35:04.229  5140  5140 W Thread-4: type=1400 audit(0.0:2472): avc: denied { bind } for scontext=u:r:untrusted_app:s0:c202,c257,c512,c768 tcontext=u:r:untrusted_app:s0:c202,c257,c512,c768 tclass=netlink_route_socket permissive=0 bug=b/155595000 app=com.tailscale.ipn
10-05 19:35:04.317  5140  5140 D vulkan  : searching for layers in '/data/app/~~zOzLyBhtJIvLs623YOyKfA==/com.tailscale.ipn-3iNy5r8ag7PFZ65Wg9n2wQ==/lib/arm64'

There is one call to net.Interface{} left, in net/dns/resolved.go:

iface, err := net.InterfaceByName(interfaceName)

However I wouldn't expect to be calling newResolvedManager() on Android, as it is for talking to systemd-resolved which Android does not use.

It is possible that net.Interface is no longer involved at all, and we're running into some other place which sends a NETLINK message which SDK 30 prohibits.

@bradfitz
Copy link
Member Author

bradfitz commented Oct 6, 2021

WireGuard itself tries to open netlink.

@DentonGentry
Copy link
Contributor

In SDK30, apps cannot use the bind() function on NETLINK_ROUTE sockets.
I think this happens in two places in wireguard-go:

I see one use of net.Interface in conn/bind_linux.go, but it does not call the Addrs() method which is the origin of the RTM_GETLINK that Android objects to.

@bradfitz
Copy link
Member Author

bradfitz commented Oct 6, 2021

I'm looking at all the return statements from NewUserspaceEngine, and which ones can make the error you see:

route ip+net: netlinkrib: permission denied

Notably, it's not decorated with anything else.

It's not conf.DNS, because that's not-nil from the caller.

Not bird.

At first, because conf.LinkMonitor is nil, I thought it looked like it was wgengine/monitor/monitor_linux.go's func newOSMon which does a netlink.Dial(unix.NETLINK_ROUTE, but then I noticed that file has a !android build tag.

Not magicsock.NewConn, because that error is prefaced via "wgengine: %v"

And then there's no other options, because we never see:

        e.logf("Bringing wireguard device up...")

So is it the link monitor and the build tags are screwed up somehow?

Edit: we also never see:

logf("link state: %+v", ...)

If logs are busted, then the only two remaining options are e.router.Up or e.router.Set.

@bradfitz
Copy link
Member Author

bradfitz commented Oct 6, 2021

Router.Up/Set look fine. The Android backend.go uses wireguard-go's CreateUnmonitoredTUNFromFD which doesn't do the netlink stuff.

@DentonGentry
Copy link
Contributor

The logid for this device is 9c4567b7ed9516453f737ca08d71e12eab7cac45c2cb9162cf7d028fa5b932e7. I don't see "Bringing wireguard device up" nor "link state:"

In the adb logcat output a few comments ago we do see: 10-05 19:35:04.229 5140 5140 W Thread-4: type=1400 audit(0.0:2472): avc: denied { bind } for scontext=u:r:untrusted_app:s0:c202,c257,c512,c768 tcontext=u:r:untrusted_app:s0:c202,c257,c512,c768 tclass=netlink_route_socket permissive=0 bug=b/155595000 app=com.tailscale.ipn

which seems to imply the failure is a bind() for a NETLINK socket. monitor_linux.go creates a unix.NETLINK_ROUTE socket but I don't see anywhere that it calls bind().

@bradfitz
Copy link
Member Author

bradfitz commented Oct 6, 2021

The route ip+net: netlinkrib: permission denied part sure looks like it comes from a net.Interface* call from the standard library:

dev:go $ git grep -F '"route"'
src/net/interface.go:           return nil, &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: errInvalidInterface}
src/net/interface.go:           err = &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: err}
src/net/interface.go:           return nil, &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: errInvalidInterface}
src/net/interface.go:           err = &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: err}
src/net/interface.go:           return nil, &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: err}
src/net/interface.go:           err = &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: err}
src/net/interface.go:           return nil, &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: errInvalidInterfaceIndex}
src/net/interface.go:           return nil, &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: err}
src/net/interface.go:           err = &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: err}
src/net/interface.go:           return nil, &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: errInvalidInterfaceName}
src/net/interface.go:           return nil, &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: err}
src/net/interface.go:   return nil, &OpError{Op: "route", Net: "ip+net", Source: nil, Addr: nil, Err: errNoSuchInterface}

@bradfitz
Copy link
Member Author

bradfitz commented Oct 6, 2021

Ah: tailscale/tailscale-android@8e1a6b2#r57595250 ... AltAddrs wasn't always non-nil.

@DentonGentry
Copy link
Contributor

DentonGentry commented Oct 6, 2021

Oh! That made it take the path where it issues an RTM_GETADDR.

Initializing newIf.AltAddrs = []net.Addr{} is working, the app starts and can access the Tailnet.

I'm going to remove some Printfs and clean this up, then remove the WIP.

bradfitz added a commit that referenced this issue Oct 6, 2021
Updates #2293

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
DentonGentry added a commit to tailscale/tailscale-android that referenced this issue Oct 7, 2021
SDK 30 prohibits syscall.NetlinkRIB(syscall.RTM_GETADDR, ...)
which Go's net.Interfaces uses. Implement an Android
specific version of net.Interfaces to use instead.

Fixes tailscale/tailscale#2293
DentonGentry added a commit to tailscale/tailscale-android that referenced this issue Oct 7, 2021
SDK 30 prohibits syscall.NetlinkRIB(syscall.RTM_GETADDR, ...)
which Go's net.Interfaces uses. Implement an Android
specific version of net.Interfaces to use instead.

Passing primitive types across JNI is relatively straightforward,
passing a single object of a complex class is annoying but still
possible, but passing lists and other more complex data structures is
way harder. As such, this commit added a Java routine to render the
interface information to a string and pass that across JNI as a
primitive type for Go code to parse.

Fixes tailscale/tailscale#2293
DentonGentry added a commit to tailscale/tailscale-android that referenced this issue Oct 8, 2021
SDK 30 prohibits syscall.NetlinkRIB(syscall.RTM_GETADDR, ...)
which Go's net.Interfaces uses. Implement an Android
specific version of net.Interfaces to use instead.

Passing primitive types across JNI is relatively straightforward,
passing a single object of a complex class is annoying but still
possible, but passing lists and other more complex data structures is
way harder. As such, this commit added a Java routine to render the
interface information to a string and pass that across JNI as a
primitive type for Go code to parse.

Fixes tailscale/tailscale#2293
DentonGentry added a commit to tailscale/tailscale-android that referenced this issue Oct 8, 2021
SDK 30 prohibits syscall.NetlinkRIB(syscall.RTM_GETADDR, ...)
which Go's net.Interfaces uses. Implement an Android
specific version of net.Interfaces to use instead.

Passing primitive types across JNI is relatively straightforward,
passing a single object of a complex class is annoying but still
possible, but passing lists and other more complex data structures is
way harder. As such, this commit added a Java routine to render the
interface information to a string and pass that across JNI as a
primitive type for Go code to parse.

Fixes tailscale/tailscale#2293
DentonGentry added a commit to tailscale/tailscale-android that referenced this issue Oct 8, 2021
SDK 30 prohibits syscall.NetlinkRIB(syscall.RTM_GETADDR, ...)
which Go's net.Interfaces uses. Implement an Android
specific version of net.Interfaces to use instead.

Passing primitive types across JNI is relatively straightforward,
passing a single object of a complex class is annoying but still
possible, but passing lists and other more complex data structures is
way harder. As such, this commit added a Java routine to render the
interface information to a string and pass that across JNI as a
primitive type for Go code to parse.

Fixes tailscale/tailscale#2293
DentonGentry added a commit to tailscale/tailscale-android that referenced this issue Oct 8, 2021
SDK 30 prohibits syscall.NetlinkRIB(syscall.RTM_GETADDR, ...)
which Go's net.Interfaces uses. Implement an Android
specific version of net.Interfaces to use instead.

Passing primitive types across JNI is relatively straightforward,
passing a single object of a complex class is annoying but still
possible, but passing lists and other more complex data structures is
way harder. As such, this commit added a Java routine to render the
interface information to a string and pass that across JNI as a
primitive type for Go code to parse.

Fixes tailscale/tailscale#2293
@wlynxg
Copy link

wlynxg commented Jul 1, 2023

I have found a solution to this problem using pure Go language, which I believe is more elegant than using Go with JNI. You can find the detailed information here: https://github.com/wlynxg/anet. I hope this is helpful for everyone!

@Ponewor
Copy link

Ponewor commented Oct 16, 2023

Hello, I am still getting
can't start tsnet server: tsnet: route ip+net: netlinkrib: permission denied

I could start a new issue but it seems like this one hasn't been handled properly yet.

@DentonGentry
Copy link
Contributor

This issue was fixed two years ago. We moved to SDK 30 and met the Play Store requirements in October of 2021.

can't start tsnet server: tsnet: route ip+net: netlinkrib: permission denied

The Android app does not use tsnet in any way, I don't believe this is related to the Android app. Please open a new issue.

I'm going to close further comments on this issue.

@tailscale tailscale locked as resolved and limited conversation to collaborators Oct 16, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
L2 Few Likelihood OS-android T8 Crash Issue type
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants