sshttp with pppoe

[adrhc]

Hi, I'm trying to use sshttp with pppoe (mtu=1492, pppoe over eth0) e.g.:
sshttpd -n 4 -S 1022 -H 1443 -L 443 -l ${ppp0_ip} -U nobody -R /run/sshttpd
with my equivalent nf-setup using DEV=ppp0 instead of eth0 and 1022, 1443 ports I get the following behaviour:

• websites visited from browser hang
• ssh username@ppp0_ip hangs

When using this setup directly with eth0 (behind a router which connects to same ISP using same pppoe configuration + mtu=1500) everything works as expected (ssh username@eth0_ip, websites are accessible).

The exact mtu=1492 for ppp0 is mandatory otherwise no website works - found after many painful tests.
When using pppoe I bind sshd to ppp0_ip:1022 and nginx to ppp0_ip:1443 while when using eth0 directly I bind them to eth0_ip:1022 and eth0_ip:1443.

I guess sshttpd binary has nothing to do with the problem but only the iptables used by nf-setup; may be those iptables change/reset the mtu to 1500 -> I guess this might be because the result seems very similar to when used mtu 1500 with pppoe (websites hang).

So is this a bug or not?

[stealth]

Strange thing. What is tcpdump and stracing the sshttpd saying? Is at least
the initial handshake coming through? (I expect that the initial handshake is less than
1492)

If its really just the mtu, "ip link ... mtu 1492" for the ethernet device should also produce
the same behavior as with ppp. Can you try setting the same mtu for the ethernet
setup and test again if it fails? If it does, its likely some mtu thing. That however
would come to surprise as its a network stack thing and has nothing to do
with sshttp itself. Maybe the DIVERT targets disrepect mtu...

However, my guess would be that some of the iptables rules dont work properly
for the ppp device. Any Networkmanager in place that has a DROP policy for ppp?

25 #if it clashes with complex NATing rules, try this
26 #iptables -t mangle -F
27 #iptables -t nat -F
28 #iptables -t raw -F

try commenting out the flushing and maybe -P ACCEPT so you can be sure
to have no DROP policy

[adrhc]

My nf-setup is also slightly changed (though working when using with eth0 directly):
SSH_PORT=1022
HTTP_PORT=1080
DEV="ppp0"
modprobe nf_conntrack_ipv4 || true
iptables -t mangle -N DIVERTHTTP || true
iptables -t mangle -A OUTPUT -p tcp -o $DEV --sport $SSH_PORT -j DIVERTHTTP
iptables -t mangle -A OUTPUT -p tcp -o $DEV --sport $HTTP_PORT -j DIVERTHTTP
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERTHTTP
iptables -t mangle -A DIVERTHTTP -j MARK --set-mark 1
iptables -t mangle -A DIVERTHTTP -j ACCEPT
ip rule add fwmark 1 lookup 123 || true
ip route add local 0.0.0.0/0 dev lo table 123

You think might be a problem the using of DIVERTHTTP instead of your DIVERT word?

PS: I'll have to get home for further testing

[adrhc]

I'm pretty sure the problem it's not my existing iptables/routing setup; I'm using the default Ubuntu 16.04 desktop installation with ufw allowing 22, 80, 443 and DEFAULT_FORWARD_POLICY="ACCEPT".

Please also consider that the websites are accessible as long as I'm not running the nf-setup script -> this leads me to the conclusion that the problem is created by nf-setup script itself.

According to http://wiki.squid-cache.org/Features/Tproxy4:

• some systems require that lo is the interface TPROXY uses.
• some systems require that an ethN is the interface TPROXY uses.
• some systems require that each receiving interface have its own unique table.

When commenting the line:
ip route add local 0.0.0.0/0 dev lo table 123
I'm getting the same hanging result for ssh so maybe I'll have to use something else than lo (I'll try later). Still I can't imagine how that could be related to websites navigation.

[adrhc]

I have 1492 mtu well set for pppoe, no doubt here (many painful tests, too late now to write about them now).
The problem is this:
sshttpd[6438]: NS_Socket::bind_local::bind:Cannot assign requested address
but only when using pppoe.
sshttpd -n 4 -S 1022 -H 1443 -L 443 -l ${wan_ip} -U sshttp -R /run/sshttpd
while no 443 port is used:
Every 2.0s: netstat -eetlpnv | grep -P "^Proto|(:(443|1443|1080|80|444|1022))" Wed Apr 12 23:56:15 2017
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 86.127.134.140:1080 0.0.0.0:* LISTEN 0 28044 2450/nginx-ip.conf
tcp 0 0 0.0.0.0:1022 0.0.0.0:* LISTEN 0 73559 1708/sshd
tcp 0 0 86.127.134.140:1443 0.0.0.0:* LISTEN 0 28045 2450/nginx-ip.conf
tcp6 0 0 :::1022 :::* LISTEN 0 73561 1708/sshd

ufw status:
To Action From
-- ------ ----
[ 1] 22 LIMIT IN Anywhere
[ 2] 80 ALLOW IN Anywhere
[ 3] 443 ALLOW IN Anywhere
[ 5] Anywhere ALLOW IN 192.168.1.0/24

[stealth]

OK. I tracked it down a bit. As the process seems to exit on the failure (your netstat shows no running sshttp on port 443), I conclude that the bind() already fails during sshttp::init(). However, do you run a patched version? Usually you should see the error message straight on stderr, not in the syslog.
If you dont run your own patched fork, which git commit are you using?

This really looks like your ${wan_ip} is somewhat messed up and you try to bind
to a non-existing address. Try without the -l ${wan_ip} switch, so it binds to 0.0.0.0.
That should work.

[adrhc]

I analyzed the syslog and seems that the eth0 is going to sleep while used by ppp0 witch in turn disconnects ppp0. This correlates very well with your last post. So for the moment I see no issue with sshttp.

[adrhc]

I just solved the ppp0 sleeping problem by commenting these in NetworkManager-pppoe configuration:

[ppp]
# lcp-echo-failure=5
# lcp-echo-interval=30

Now my internet connection is stable and under control:
tailf /var/log/syslog | grep -ni -P "sleep|disconnect|NetworkManager|/etc/network/" -> shows nothing
When I start sshttpd with "-L 443 -l ${wan_ip}" I get the error:
NS_Socket::bind_local::bind:Cannot assign requested address
When I start sshttpd with only "-L 443" I get no error and I see the bindings:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      0          60090       7105/sshttpd

pretty clear ...
Then I try to use ssh over 443 and I get the error:
sshttp :: loop:: NS_Socket::bind_local::bind:Address already in use
extremly anoying ...
There's no other process using 443, I'm watch-ing this, really.

[adrhc]

... then I restarted the computer, started nginx on wan_ip:443 and wan_ip:1080 and it worked using the wan_ip:443 (https://wan_ip/my-web-site-url). I guess sshttp should work too.

[adrhc]

Besides eth0 I also have another card named eth1. When it is enabled nginx also complains about couldn't binding to 443. The test just above (done 2x) is accomplished with eth1 disabled at start. With the same environment setup then sshttp starts fine on wan_ip:443 (hooray); I then try ssh or https over 443 and I get:
sshttp:: loop::NS_Socket::bind_local::bind:Address already in use
So with wan_ip:443 nginx works while sshttp doesn't.

[adrhc]

When moving everything to 127.0.0.1 instead of wan_ip:
sshttpd -n 4 -S 1022 -H 1443 -L 443 -l 127.0.0.1 -U sshttp -R /run/sshttpd
and also doing an iptables redirect from wan_ip:443 to 127.0.0.1:443 I get the same result:
sshttp:: loop::NS_Socket::bind_local::bind:Address already in use

[adrhc]

Same for:
sshttpd -n 4 -S 1022 -H 1443 -L 444 -U sshttp -R /run/sshttpd
with:
wan_ip:443 -> 127.0.0.1:444

[adrhc]

I've done the same tests with sslh and I've got the same result (i'll ask them too about this):
sslh-select 1.17-2 started
bind:98:Address already in use
bind_peer:98:Address already in use

[adrhc]

I use ubuntu 16.04 with ufw with default configuration + ports 22, 80, 443 allowed.

[stealth]

Thats strange and sounds less of a sshttp problem itself, as reportedly sshl shows the
same behavior. From where are you doing these connect tests? I hope you are not
doing it from the local machine? The "Address already in use" sounds like sshttp
tries to bind to the same IP:port via IP_TRANSPARENT to an address thats apparently
already in use. So you either do the connect locally (which fails) or have some weird
SNAT rules on your external interface that clashes with the addresses of the sshttp box
and leads to "double binds".
Another (unlikely) reason is that some pppoe stack driver lacks IP_TRANSPARENT support
and somehow mangles it to an address thats already bound. I would check the pppoe source
once you made sure above mentioned config problems dont exist.

[adrhc]

Well, I'm using:

ssh username@ppp0_ip
or 
curl -k https://ppp0_ip/my-web-site

from localhost but this works well when not using pppoe.
I have no special iptables setup besides ufw and sshttp.
But why nginx works on 443 after closing sshttp while its + ufw iptables setup remains?
It's weird, I agree.

[stealth]

It may be different for ppp since its a point to point device. if the local ssh or curl
binds to wan_ip:port, sshttp as well as sshl try to bind to the same address again so
you would see the originator address in the logs. But that cannot happen if your
client is already using that address:port. I recommend doing the test from outside (it doesnt make much sense anyway to hide sshd behind nginx for localhost connections)

[adrhc]

First I'll clarify something very important (I guess I induced some confusion related to these):

• for me wan_ip = ppp0_ip when using pppoe
• for me wan_ip = eth0_ip when not using pppoe

... if the local ssh or curl binds to wan_ip:port, sshttp as well as sshl try to bind to the same address again so you would see the originator address in the logs. ...

Practically I understand that I might miss some correct configuration for the listening ports and/or ethernet interface but I have no idea about which.

I start sshttpd this way:
sshttpd -n 4 -S 1022 -H 1443 -L 443 -l ${ppp0_ip} -U sshttp -R /run/sshttpd
sshd_config contains only:

Port 22
Port 1022

nginx.conf contains only:
listen ppp0_ip:1443 ssl;

I stress again: stopping sshttp then using nginx with ppp0_ip:443 will make nginx work on 443.

This totally puzzles me:

... or curl binds to wan_ip:port ...

I'm pretty sure when using curl then curl itself won't bind a port but only open a connection on a port, is it?

[adrhc]

Ok, besides the comments above I anyway tried to do as you said and I connected from internet with my phone to ppp0_ip:443 and it works both for ssh:443 and https:443 (hooray).

There's a problem still! :(

E.g. wordpress (used by my blog) is hardcoding the base website url; so any link on it will be formed like https://my-internet-domain/my-website. Accessing https://my-internet-domain/ from localhost (host running sshttp) yields a series of these errors:

Apr 22 10:45:34 adr-desktop sshttpd[5302]: sshttp::loop::NS_Socket::bind_local::bind:Address already in use
Apr 22 10:45:34 adr-desktop sshttpd[5305]: sshttp::loop::NS_Socket::dstaddr::getsockopt:Operation not supported

and of course I'm writing my blog post from localhost ...

Not to mention that is better/easier/safer for me to test my other personal websites by accessing them using my-internet-domain instead of using ppp0_ip or other tricks.

Any idea about solving this?

[adrhc]

I'm feeling like a noob; anyway thanks to you I guess I solved the entire puzzle.

My claim that without pppoe it works is not true - I get the same error which, as you already said, it's natural. Every time I tested https and ssh I made a small but decisive mistake.
I'll explain just for the record for others to understand the whole issue.

I tested https://my-domain-name/my-website while my-domain-name was resolved to router's WAN ip (you know, the bookmark was easily accessible in browser's toolbar). Instead I should have test https://eth0_ip/my-website in order for the test to be equivalent to the one with ppp0_ip.

With ssh I used a shorter equivalent to ssh -p 443 my_username@my-domain-name with again my-domain-name being resolved to router's WAN ip (here I used ~/.ssh/config which was using my-domain-name). Instead I should have test ssh -p 443 my_username@eth0_ip in order for the test to be equivalent to the one with ppp0_ip.

Anyway the question remains:
is it possible to do something in order for e.g. https://ip-where-sshttp-is-bound-to/my-website to work?

[stealth]

Ok, so I conclude that everything is fine with sshttp and nothing needs to be fixed.
It was just an "issue" with connecting from localhost. Wordpress and CMS guidelines
are beyond my support :D
closing