Got timeout when remove buildx instance which related a remote docker daemon

[LinuxSuRen]

My command:
docker buildx rm arm

The output:
Cannot connect to the Docker daemon at tcp://121.36.64.187:2375. Is the docker daemon running?

Expected behaviors:
I can remove the buildx instance no matter if it's online.

[LinuxSuRen]

It's very slow when I execute the following command:
docker buildx ls

I guest that it comes from the same reason.

[tonistiigi]

It should still remove the builder, even if it is not online. It errors to show that the removal was not clean.

[theredcat]

Same problem here. I have a remote ARM builder that doesn't exists anymore and I cant remove it :

docker buildx rm -f arm64
ERROR: error during connect: Get "http://docker.example.com/_ping": command [ssh -l admin -- 10.11.12.164 docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh: connect to host 10.11.12.164 port 22: Connection timed out