New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

master.tmate.io lookup failure #32

Closed
hh opened this Issue Mar 3, 2014 · 35 comments

Comments

Projects
None yet
@hh
Copy link

hh commented Mar 3, 2014

$ tmate show-messages                                                                                                                                                                                                                                   
Tue Mar  4 09:18:32 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:18:42 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:18:52 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:19:02 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:19:12 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
@hh

This comment has been minimized.

Copy link

hh commented Mar 3, 2014

x11screensnapz001
Going through various iterations of lower/upcase master.tmate.io the response seems valid on the wire.

@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Mar 3, 2014

I just did a query:

pafy@bisou ~ % nslookup master.tmate.io
nslookupServer:         192.168.11.1
Address:        192.168.11.1#53

Non-authoritative answer:
Name:   master.tmate.io
Address: 128.199.246.194
Name:   master.tmate.io
Address: 82.196.12.126
Name:   master.tmate.io
Address: 192.241.193.106
Name:   master.tmate.io
Address: 192.241.130.94

And hitting the root DNS:


pafy@bisou ~ % dig @a.dns.gandi.net master.tmate.io
dig
; <<>> DiG 9.9.4 <<>> @a.dns.gandi.net master.tmate.io
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28665
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 3, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;master.tmate.io.               IN      A

;; ANSWER SECTION:
master.tmate.io.        10800   IN      A       192.241.193.106
master.tmate.io.        10800   IN      A       82.196.12.126
master.tmate.io.        10800   IN      A       128.199.246.194
master.tmate.io.        10800   IN      A       192.241.130.94
@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Mar 3, 2014

I'm sorry but I can't reproduce this bug.

The lookup code is dead simple:

https://github.com/nviennot/tmate/blob/master/tmate-session.c#L98-100
https://github.com/nviennot/tmate/blob/master/tmate-session.c#L34-L37

There is not much chance to screw up on two lines. Perhaps it would be interesting to see what happens if you upgrade libevent (which takes care of the dns lookup). Also you may do some stracing and see what's going on.

Sorry if I can't help more than that.

@hh

This comment has been minimized.

Copy link

hh commented Mar 3, 2014

Eventually it connected, and wireshark shows packets on the wire to be mostly normal

$ tmate show-messages
Tue Mar  4 09:36:23 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:36:33 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:36:43 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:36:53 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:37:03 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:37:13 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:37:23 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:37:33 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:37:43 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:37:53 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:38:03 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:38:13 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:38:23 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:38:33 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:38:43 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:38:53 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:39:03 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Tue Mar  4 09:39:14 2014 [tmate] Connecting to master.tmate.io...
Tue Mar  4 09:39:16 2014 [tmate] Remote session read only: ssh ro-XXXX@sf.tmate.io (clear your screen if you share this)
Tue Mar  4 09:39:16 2014 [tmate] Remote session: ssh XXXX@sf.tmate.io
@nicopace

This comment has been minimized.

Copy link

nicopace commented Sep 15, 2014

It is still happening:

$ tmate show-messages
Mon Sep 15 12:05:39 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:05:49 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:05:59 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:06:09 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:06:19 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:06:29 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:06:39 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:06:49 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:06:59 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:07:09 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:07:19 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:07:29 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:07:39 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:07:50 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:08:00 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:08:10 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:08:20 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:08:30 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:08:40 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)
Mon Sep 15 12:08:50 2014 [tmate] master.tmate.io lookup failure. Retrying in 10 seconds (non-recoverable failure in name resolution)

Some more info:

$ nmap master.tmate.io                                                                                                                                                                      

Starting Nmap 6.40 ( http://nmap.org ) at 2014-09-15 12:06 ART
Nmap scan report for master.tmate.io (128.199.246.194)
Host is up (0.39s latency).
Other addresses for master.tmate.io (not scanned): 192.241.130.94 192.241.193.106 82.196.12.126
rDNS record for 128.199.246.194: sg.tmate.io
Not shown: 997 filtered ports
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
222/tcp open  rsh-spx

$nslookup master.tmate.io
...

Non-authoritative answer:
Name:   master.tmate.io
Address: 128.199.246.194
Name:   master.tmate.io
Address: 192.241.130.94
Name:   master.tmate.io
Address: 192.241.193.106
Name:   master.tmate.io
Address: 82.196.12.126
@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Sep 15, 2014

Does the problem go away if you hardcode an IP address for master.tmate.io in /etc/hosts ?

@nicopace

This comment has been minimized.

Copy link

nicopace commented Sep 15, 2014

Yep... it works that way... what is happening?

@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Sep 15, 2014

I don't know, I can't reproduce that bug :(
Since I'm using the DNS resolver of libevent, maybe it has some quirks that I am unaware of.

@nicopace

This comment has been minimized.

Copy link

nicopace commented Sep 15, 2014

Can i help you reproduce the bug?
Do you want me to send you any info/

@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Sep 15, 2014

Will you be available at 8pm EST?

@nicopace

This comment has been minimized.

Copy link

nicopace commented Sep 15, 2014

nope... just 8-17 GMT-3.

On Mon, Sep 15, 2014 at 1:46 PM, Nicolas Viennot notifications@github.com
wrote:

Will you be available at 8pm EST?


Reply to this email directly or view it on GitHub
#32 (comment).

Ing. Nicolás Pace
http://www.linkedin.com/in/nickar/
http://www.unixono.com.ar/

@victorbrca

This comment has been minimized.

Copy link

victorbrca commented Oct 27, 2014

Same problem here on Ubuntu 14.04.

[tmate] master.tmate.io lookup failure. Retrying in 10 seconds (nodename nor servname provided, or not known)```

Resolution seemed ok and hard coding the IP in /etc/hosts helped:

$ nslookup master.tmate.io                                                                                                                                                             
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   master.tmate.io
Address: 192.241.130.94
Name:   master.tmate.io
Address: 192.241.193.106
Name:   master.tmate.io
Address: 82.196.12.126
Name:   master.tmate.io
Address: 128.199.246.194

However I was able to fix the issue by disconnecting from my VPN. Not sure if it's related, but it could be.

@nicopace

This comment has been minimized.

Copy link

nicopace commented Oct 27, 2014

I was directly connected to internet.
I'm still available @nviennot if you want to pair program this issue using tmate :)

@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Oct 28, 2014

Arh that sucks. I have negative time at the moment, so I'll deal with it in december. However, if a hero can debug that, that would be great :) :)

@job

This comment has been minimized.

Copy link

job commented Nov 9, 2014

pro-tip: nslookup is the worst dns debugging tool ever created. Please consider using dig or drill, or even ping if you must.

I've done testing from 200 servers distributed over the planet, could not discover issues. Therefor I suspect this is a local problem. Gandi's Authoritive DNS services are used by milions of domains, wide spread issues would have bigger impact.

@danhigham

This comment has been minimized.

Copy link

danhigham commented Jan 14, 2015

Same problem here.. MacOS 10.10.1

@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Jan 14, 2015

I'm going to look into it, but the sad news is that I might have to do a lot of work to replace the DNS resolver. I'll see what I can do.

@mgmoerman

This comment has been minimized.

Copy link

mgmoerman commented Jan 17, 2015

I have the same issue when using BIG IP VPN client, adding static entry to host file solves it.

@zhuangya

This comment has been minimized.

Copy link

zhuangya commented Feb 10, 2015

i have the same problem with a custom domain name(runningi a tmate slave server of couse)

tmate-slave via this docker

tmate client version: 1.8.10, installed from homebrew

@gatopan

This comment has been minimized.

Copy link

gatopan commented Feb 26, 2015

Same Issue, 10.9.5. Adding to 192.241.130.94 master.tmate.io to
/etc/hosts let me connect.

@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Feb 27, 2015

I'll fix the bug in 2 weeks (I have some deadline going on at the moment).

@jmangt

This comment has been minimized.

Copy link

jmangt commented Aug 12, 2015

Hi. I just started having this issue. Any help is appreciated

~  tmate show-messages
Wed Aug 12 12:13:44 2015 [tmate] Connecting to master.tmate.io...
Wed Aug 12 12:13:44 2015 [tmate] Error connecting: Connection refused
➜  ~  tmate -V
tmate 1.8.10
➜  ~  uname -a
Darwin Casper-Image 14.4.0 Darwin Kernel Version 14.4.0: Thu May 28 11:35:04 PDT 2015; root:xnu-2782.30.5~1/RELEASE_X86_64 x86_64
➜  ~  dig master.tmate.io

; <<>> DiG 9.8.3-P1 <<>> master.tmate.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30134
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;master.tmate.io.               IN      A

;; ANSWER SECTION:
master.tmate.io.        8313    IN      A       178.62.62.167
master.tmate.io.        8313    IN      A       46.101.184.80
master.tmate.io.        8313    IN      A       104.131.19.169
master.tmate.io.        8313    IN      A       107.170.226.142
master.tmate.io.        8313    IN      A       128.199.246.194
master.tmate.io.        8313    IN      A       128.199.33.62

;; Query time: 56 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Aug 12 12:16:22 2015
;; MSG SIZE  rcvd: 129
@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Aug 12, 2015

Hi :)

So Connection refused is different from a DNS lookup failure.
Try to force the IP of master.tmate.io in your /etc/hosts file with one of the IP that dig gives you.

I just tried all 6 of them, and none of giving me a connection refused.

@jmangt

This comment has been minimized.

Copy link

jmangt commented Aug 12, 2015

It is working now. I was working from a Stabucks and it just would not connect. Back at home and all is good. No changes required.

@falood

This comment has been minimized.

Copy link

falood commented Oct 6, 2015

Could you please offer config file for tmate client? Maybe decrease TMATE_DNS_RETRY_TIMEOUT to 1 is a temporary solution.

@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Oct 6, 2015

Will do when putting in the reconnection logic

@Gaolz

This comment has been minimized.

Copy link

Gaolz commented Jan 19, 2016

ubuntu 14.04, same issue. After adding 192.241.130.94 master.tmate.io to /etc/hosts, it works well. Thx @gatopan

@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Jan 19, 2016

@Gaolz that's surprising. This IP is no longer in use by tmate servers...

FYI: For Ubuntu, see https://tmate.io to use the up to date ppa.

@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Mar 29, 2016

Closing due to inactivity.

@nviennot nviennot closed this Mar 29, 2016

@foxx

This comment has been minimized.

Copy link

foxx commented Oct 4, 2016

I'm still getting this problem, adding a static entry to /etc/hosts works fine, but I have no idea why because host ssh.tmate.io comes back with results?! I'm running macOS Sierra.

@foxx

This comment has been minimized.

Copy link

foxx commented Oct 4, 2016

@nviennot I'm available quite late (EST) if you'd like to do a session to try and debug this? Seems quite a few people are having this issue.

@nviennot

This comment has been minimized.

Copy link
Member

nviennot commented Oct 4, 2016

@foxx Thursday evening (EST) would be better for me. Yes?

@foxx

This comment has been minimized.

Copy link

foxx commented Oct 4, 2016

@nviennot Sure that works for me, ping me your details and a proposed time, cal@iops.io

@robotmay

This comment has been minimized.

Copy link

robotmay commented Feb 28, 2017

Also receiving this. Possibly an issue with going through a VPN or a DNS cache?

@4n3w

This comment has been minimized.

Copy link

4n3w commented May 8, 2018

Check and make sure you don't have an ipv6 DNS server in your list of DNS servers. Looks like tmate doesn't support ipv6 addresses at least for name resolution. I removed the ipv6 entry from my resolver list, and tmate works again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment