have two working exitnodes in tunneldigger config

[paidforby]

We should remove extra active exit nodes from tunneldigger config, this way all home nodes will get directed to HE exit node. We can still include extra public IP addresses as emergency backups, but the tunnel brokers running on 45.34.140.42 (psychz) and 104.236.181.226 (someone else's tunnel broker?) could be causing confusion for home nodes. Since psychz exit node is being deprecated, we should comment it out. The other tunnel broker running on 104.236.181.226 appears to be broken since it does not supply mesh routes or an internet connection, so it makes sense to completely remove it.

[paidforby]

On an undisclosed node at 100.65.97.65 to which I applied the suggested fix, I'm seeing crazy up/down behavior on the tunnel, @bennlich or @jhpoelen any ideas what might be causing this,

Wed Sep 26 18:32:43 2018 daemon.info td-client: Performing broker selection...
Wed Sep 26 18:32:43 2018 daemon.info td-client: Not trying 64.71.176.94:8942 again as it broke last time we tried.
Wed Sep 26 18:32:43 2018 daemon.info td-client: Not trying 64.71.176.94:443 again as it broke last time we tried.
Wed Sep 26 18:32:55 2018 daemon.err td-client: No suitable brokers found. Retrying in 5 seconds
Wed Sep 26 18:33:00 2018 daemon.info td-client: Performing broker selection...
Wed Sep 26 18:33:01 2018 daemon.debug td-client: Broker usage of 64.71.176.94:443: 1471
Wed Sep 26 18:33:01 2018 daemon.info td-client: Selected 64.71.176.94:443 as the best broker.
Wed Sep 26 18:33:02 2018 daemon.info td-client: Tunnel successfully established.
Wed Sep 26 18:33:02 2018 daemon.info td-client: Requesting the broker to configure downstream bandwidth limit of 4096 kbps.
Wed Sep 26 18:35:03 2018 daemon.err td-client: Broker sent us a teardown request, closing tunnel with errorcode 3!
Wed Sep 26 18:35:03 2018 daemon.err td-client: Connection to 64.71.176.94 lost.
Wed Sep 26 18:35:03 2018 daemon.info td-client: Performing broker selection...
Wed Sep 26 18:35:03 2018 daemon.debug td-client: Broker usage of 64.71.176.94:8942: 1407
Wed Sep 26 18:35:03 2018 daemon.info td-client: Selected 64.71.176.94:8942 as the best broker.
Wed Sep 26 18:35:04 2018 daemon.err td-client: Unable to create local L2TP tunnel!
Wed Sep 26 18:35:04 2018 daemon.err td-client: Connection to 64.71.176.94 lost.
...
Wed Sep 26 18:35:13 2018 daemon.info td-client: Performing broker selection...
Wed Sep 26 18:35:14 2018 daemon.debug td-client: Broker usage of 64.71.176.94:8942: 1471
Wed Sep 26 18:35:14 2018 daemon.info td-client: Selected 64.71.176.94:8942 as the best broker.
Wed Sep 26 18:35:15 2018 daemon.info td-client: Tunnel successfully established.
Wed Sep 26 18:35:15 2018 daemon.info td-client: Requesting the broker to configure downstream bandwidth limit of 4096 kbps.
Wed Sep 26 18:35:35 2018 daemon.info td-client: Setting MTU to 1438
Wed Sep 26 18:35:41 2018 daemon.info td-client: Setting MTU to 1438

[bennlich]

Looking at the exitnode logs, the tunnel to your node was established and then closed 2 minutes later, and this happened several times in a row.

Looking at the broker code, the broker is programmed to kill the tunnel if it doesn't hear from the client for 2 minutes. It closes the connection with broker error code 3 (not to be confused with client error code 3), which is what shows up in your logs.

Why your node was unable to send a message to the broker for two whole minutes, I am not sure...

Weirdly, your node has been successfully connected for almost an hour now. Did anything about your network change?

[paidforby]

Interesting observations. Any idea what "client error code 3" corresponds to?
It's odd that it was able to reach the exit node and establish the tunnel and then it immediately times out.

Here's the a more complete log from the node in question, messages.txt. I'm still seeing the same up/down behavior from the the tunnel, though I am able to reach it through the exit node (note: I am debugging remotely, so it's hard to say what's changed other than the node being rebooted once or twice).

[jhpoelen]

As far as deprecating our only fail-over exit node goes, an ancient Dutch expression comes to mind: Don't toss your old shoes until the break in the new ones. Are you planning to setup a new exit node in addition to the HE one?

[paidforby]

The new HE exit node has proved fairly robust for the last six months or so. Additionally, there may be a standing offer for a donated hosting location that was discussed at a recent Tuesday meeting. Perhaps a loomio discussion is needed, but I don't see many people in favor of continuing to pay for the psychz exit node. You're correct that we should have two exit nodes, but I don't think psychz is a good place for a secondary exit node. To me, it makes sense to find a place in or near Oakland to put an exit node instead of having a virtual exit node in LA.

[bennlich]

I'm still seeing the same up/down behavior from the the tunnel

Actually, this looks to me like a different up/down behavior (or, the same behavior, but with a different root cause). @paidforby you are the master of finding a bunch of different bugs at the same time that look related but may actually be completely independent :-P

In this case, the logs you sent in messages.txt correspond to a bunch of setsockopt out of memory errors on the exitnode. This is sudomesh/bugs#24. Your node kept trying to connect and failing for 10 minutes, which is how long it took the exitnode monitor script to realize there was a problem and restart babeld.

[bennlich]

Any idea what "client error code 3" corresponds to?

client error 3 is here, but it doesn't look like it gets used anywhere. The exit code 3 you were seeing in your first log dump is actually broker error 3 (aka I haven't received any messages from the other end of the tunnel in 2 minutes error).

[jhpoelen]

@paidforby (related to the initial deprecation of the psychz exit node). Thanks for taking the initiative and I would also favor a more local exit node in addition to the HE node. Having gone through the labor intensive exercise of patching tunneldigger configurations myself (took months), I would highly recommend to add the working configuration for the psychz replacement before removing the current psychz ip . It is easy for us to turn off the psychz exit node by stopping tunnel digger on the server: the home nodes would try to connect and move on to the next candidate if it doesn't respond. However, turning on a new exit node IP currently involves patching all the existing home nodes. Perhaps we should consider some sort of seeding mechanism in which exit node ip addresses can be propagated to the home nodes without having to patch them manually. But for now, see above.

I am ok removing 104.236.181.226 . This ip was actively managed in the past, but doesn't seem to have gotten much tlc. Also, I don't think we have access to that exit node. If you'd like, I can introduce you to the person that used to manage that exit node. This might actually be a nice way to rekindle a relationship with a currently dormant contributor.

[paidforby]

Re-added 104.236.181.226 as a comment in tunneldigger config so knowledge of it's existence is not lost and as a reminder to contact exit node operator about the status of their tunnel broker.

[eenblam]

Fixed in 73fce58

Exit nodes are now specified by domain name, with builds.sudomesh.org still provided via IP in case DNS breaks.