Planetscale CLI hangs entire system on login (M1 Mac)

[mbesemann]

I was trying to set up the Planetscale CLI on my MacBook Air via Homebrew, which seems like it should work in theory as the release is compiled for arm64.

I installed it via Homebrew and got to the stage of confirming the authorization code on the website, however after that step it runs into a network error and almost right away the whole network stack (and eventually the entire system) becomes unresponsive:

Error: Post "https://auth.planetscale.com/oauth/authorize_device": dial tcp: lookup auth.planetscale.com: i/o timeout

I tried kill -9 pscale but that does not work, neither does rebooting, so I eventually have to do a hard shutdown.

Has anyone encountered this on an M1 machine?

[jgreet]

🤔 We haven't heard any similar reports yet, and a lot of us at PlanetScale use m1 macs.

Oddly enough I did have a similar OSX crash recently on 12.2 where networking and then the whole system locked up. I was using Firefox at the time and assumed that was related.

Is this reproducible for you? Which version of OSX are you using?

[mbesemann]

Yes this happens every single time I run pscale auth login. I don't get prompted to verify the authorization code anymore, I assume it's still cached. I did try reinstalling pscale via brew but the only thing that changed was getting me to do the authorization step again.

I am running the latest macOS 12.3.1.

[yych42]

Can confirm the same problem, but it's not isolated to auth. I ran into the same thing with pscale connect .... All the processes started throwing NSURLErrorDomain -1001 after the command failure.

12.3.1 with M1 Pro.

[mbesemann]

are you a Little Snitch user per chance? That's the only thing I thought might be interfering with the connections.

[yych42]

No. Did you fix it on your device?

[mbesemann]

No but I honestly haven't tried troubleshooting much since a hard reset is rather cumbersome

[nickvanw]

Hey Folks,

I really appreciate the reports of this - we've been very confused for quite some time, many of us at PlanetScale use the pscale CLI every day with macOS on both amd64 and ARM-based processors and have never run into this issue. We asked everyone to keep an eye out for it, but couldn't reproduce it as hard as we tried.

Until.. today! Someone was able to continuously run into this problem - down to requiring a hard restart - and we dug in to debug.

Where we are right now is that we think this is a combination of IPv6 and Go's DNS resolution code with CGO disabled. We have a scoped down reproduction case that appears to create the same effect that we experienced with the pscale CLI.

In short, when our test laptop is:

• Connected to an IPv6-native network and has both an IPv6 and IPv4 address in /etc/resolv.conf
• That resolv.conf points to a v6 link-local address with a zone (e.g. fe80::1%lo0 which appears to be the default in some cases)
• Tries to run a go program compiled with CGO_ENABLED=0 that resolves a domain name

Everything locks up.

The test program we've been using to reproduce this is as simple as:

package main

import (
	"context"
	"fmt"
	"net"
	"os"
	"time"
)

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	ips, err := net.DefaultResolver.LookupIP(ctx, "ip", "google.com")
	if err != nil {
		fmt.Fprintf(os.Stderr, "Could not get IPs: %v\n", err)
		os.Exit(1)
	}
	for _, ip := range ips {
		fmt.Printf("google.com. IN A %s\n", ip.String())
	}
}

It appears on both ARM and amd64, but only when connected to an IPv6 network. For me, that's meant when I'm tethered to my phone (as most mobile networks in 🇺🇸 are v6-native), but I'm sure it happens on normal home ISPs as well around the world.

We think our next steps are:

1. Report this as a bug to the language maintainers, as it appears to be easily reproduced with nothing PlanetScale-specific
2. Look at compiling the pscale CLI with cgo enabled.

Apologies for how long this has taken, I wish we could have found it sooner!

[mattrobenolt]

Yup, thank you, that checks out with my assessment so far! Thanks for confirming.

[nickvanw]

@mbesemann for now, the fastest way to get a working pscale CLI is unfortunately to self-compile it from Git, with CGO enabled (which is the default).

Unfortunately it's not straightforward to get a cross-platform cross-OS build going with CGO enabled, which is what's stopping us from just releasing a new version that fixes this problem.

[mattrobenolt]

EDIT: Although funny enough, the following comment appears at the top of the file:

I guess I responded after you made this edit.

Fun fact, Go re-invents DNS resolution an explicitly consults this file and bypasses what macOS does. This is the reason why it works with CGO enabled, because GCO uses the OS DNS resolver, rather than their own. CGO has it's pros and cons for sure and I get why Go does this, but that's a whole tangential topic. Normally you shouldn't need to be this keen on this and it should just work. :)

[JustinyAhin]

@mbesemann for now, the fastest way to get a working pscale CLI is unfortunately to self-compile it from Git, with CGO enabled (which is the default).

Unfortunately it's not straightforward to get a cross-platform cross-OS build going with CGO enabled, which is what's stopping us from just releasing a new version that fixes this problem.

Hello @nickvanw,

What is the process to do the "self-compile"?

Thanks

[nickvanw]

Hi @JustinyAhin!

If you clone the repo on a machine with Go installed, you can run go build in the cmd/pscale directory. That should get you a self-compiled pscale binary on any platform.

[dbussink]

Alright, with https://github.com/planetscale/cli/releases/tag/v0.105.0 we should build the pscale CLI on MacOS with CGO (and thus the system resolver) enabled which should work around this issue.