Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPFS is doing port scanning, without indicating that in logs #5418

Closed
burdakovd opened this issue Sep 1, 2018 · 12 comments
Closed

IPFS is doing port scanning, without indicating that in logs #5418

burdakovd opened this issue Sep 1, 2018 · 12 comments

Comments

@burdakovd
Copy link

burdakovd commented Sep 1, 2018

Version information:

go-ipfs version: 0.4.15-
Repo version: 6
System version: amd64/linux
Golang version: go1.9.7

Type:

Bug

Description:

IPFS randomly connects to addresses in a way that resembles port scanning, without any indication of what is happening in the logs (level=debug).

A few days after installing IPFS daemon on my public server, I've got email from provider (AWS) with abuse report, indicating that sombeody received a port scan from my address (700 TCP ports scanned in total).

To be safer and not get banned, I've done the following steps:

  • Disable all NAT traversal, MDNS, and other "magic" technologies in IPFS server (as based on the searches, they may be responsible for things similar to port scanning)
  • To ensure I don't get blamed for port scanning, configured firewall to allow outgoing connections to only pot 4001
  • Configured IPFS to advertise only my public IP address (and not e.g. 127.0.0.1, or internal address of Docker container), to ensure none of the NAT "piercing" logic is needed.
  • Added logging on AWS to monitor all traffic going to and from machine
  • Added debug level logging on ipfs daemon

This is my full config: https://ipfs.io/ipfs/QmQUWSFqV9bDckgpzBaKBBZsPSwHCxVAjKRx31TD5zuLVV/
This is how I generate the config (start script of my Docker image): https://raw.githubusercontent.com/burdakovd/dapps.earth/master/ipfs/start.sh

Now, I started looking into rejected outgoing connections, and indeed I see behavior resembling port scan, despite all my settings. Luckily this time, my local firewall prevented another abuse report.

Here are logs indicating my node (internal IP is show in logs) connecting to 54.186.184.82. First connection is OK, and then 10 minutes later it starts connecting to random ports of 54.186.184.82.

Schema: [version, account, eni, source, destination, srcport, destport, protocol, packets, bytes, windowstart, windowend, action, flowlogstatus]
ACCEPT/REJECT means just whether the connection was accepted/rejected by my firewall, not necessarily means that the connection successfully went through, as it could've been rejected on the other side.
image

In the meantime, in ipfs logs (debug level of logging enabled), there is no mention of 54.186.184.82 whatsoever!

(note: 01:42:00 is the moment when I have restarted my node, after having updated log level to "debug")

Furthermore, I see that in third line of network logs 54.186.184.82 was attempting to connect to me at port 1024, despite me advertising only 4001.

@bonedaddy
Copy link
Contributor

enable the server profile during your init statement. ipfs init --profile=server

also I don't think it's doing port scanning I suspect the bitswap traffic might be detected as port scanning.

@burdakovd
Copy link
Author

@postables , I did enable server profile.

I did it with ipfs config profile apply server though instead of ipfs init --profile=server, I don't think it should make difference. I've included the full resulting config in the issue description.

Is it expected that bitswap would attempt to connect to multiple ports of a peer (without indicating that in logs)?

To start with, I'd be surprised if a peer advertises themselves listening on 8 different ports, why would they do so?
Furthermore, there was no mentions of peer IP address in debug-level logs whatsoever.

I'm not that familiar with internals of how bitswap works, but is it normal for it to attempt to connect to 700 different TCP ports of a given IP address (which is what I've received in initial abuse report)?

@burdakovd
Copy link
Author

burdakovd commented Sep 1, 2018

Another example:

image

In this case we've had an incoming connection from 171.214.222.88. After some period of exchange, it appears disconnect happened.

After that my node just started dialing random ports of 171.214.222.88, until it finally reestablished connection at 171.214.222.88:59512.

This time there are logs from ipfs for this session (intersting part at 16:54:51): https://ipfs.io/ipfs/QmRKhMz6vtftti5PFaCqQAGESBNjBWmEqaE5TCbXfgpn88/

Particularly worrying part is below:
image

It seems in this situation, after connection has been lost (maybe remote node was restarting, or temporarily went offline), we bombarded 171.214.222.88 with attempts to connect on many differenr ports. Those attempts were unsuccessful, and the connection got recovered after that node finally connected to us.

It appears that node is behind NAT, as this is what they are advertising: [/ip4/192.168.100.29/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/171.214.222.88/tcp/59606 /ip4/171.214.222.88/tcp/59512]. It advertises a bunch of local IP addresses, and a public IP with an ephemeral port generated by NAT.

For some reason I'm also getting TONS of ports of 171.214.222.88 from PEERS CLOSER messages, e.g., and this is how my node got an idea to attempt to connect to many ports of 171.214.222.88:

�[0;37m15:34:07.717 �[36mDEBUG �[0;34m dht: �[0mPEERS CLOSER -- worker for: <peer.ID fEwokY> added <peer.ID Z7jpNP> ([/ip4/171.214.222.88/tcp/62272 /ip4/171.214.222.88/tcp/60469 /ip4/171.214.222.88/tcp/62836 /ip6/::1/tcp/4001 /ip4/171.214.222.88/tcp/56383 /ip4/171.214.222.88/tcp/53663 /ip4/171.214.222.88/tcp/57890 /ip4/171.214.222.88/tcp/63478 /ip4/171.214.222.88/tcp/54038 /ip4/171.214.222.88/tcp/15271 /ip4/171.214.222.88/tcp/3811 /ip4/171.214.222.88/tcp/61617 /ip4/171.214.222.88/tcp/9751 /ip4/192.168.100.29/tcp/4001 /ip4/171.214.222.88/tcp/62421 /ip4/171.214.222.88/tcp/53483 /ip4/171.214.222.88/tcp/59495 /ip4/171.214.222.88/tcp/11817 /ip4/171.214.222.88/tcp/62354 /ip4/171.214.222.88/tcp/60000 /ip4/171.214.222.88/tcp/54804 /ip4/171.214.222.88/tcp/49942 /ip4/171.214.222.88/tcp/44293 /ip4/171.214.222.88/tcp/57158 /ip4/171.214.222.88/tcp/63682 /ip4/171.214.222.88/tcp/54838 /ip4/171.214.222.88/tcp/19213 /ip4/171.214.222.88/tcp/52111 /ip4/171.214.222.88/tcp/49093 /ip4/171.214.222.88/tcp/60562 /ip4/171.214.222.88/tcp/58653 /ip4/171.214.222.88/tcp/60472 /ip4/171.214.222.88/tcp/59542 /ip4/171.214.222.88/tcp/11749 /ip4/171.214.222.88/tcp/27011 /ip4/171.214.222.88/tcp/2191 /ip4/171.214.222.88/tcp/52826 /ip4/171.214.222.88/tcp/59978 /ip4/171.214.222.88/tcp/64238 /ip4/171.214.222.88/tcp/34921 /ip4/171.214.222.88/tcp/48367 /ip4/171.214.222.88/tcp/64888 /ip4/171.214.222.88/tcp/63596 /ip4/171.214.222.88/tcp/50054 /ip4/171.214.222.88/tcp/59972 /ip4/171.214.222.88/tcp/52387 /ip4/171.214.222.88/tcp/8553 /ip4/171.214.222.88/tcp/53991 /ip4/171.214.222.88/tcp/56400 /ip4/171.214.222.88/tcp/36687 /ip4/171.214.222.88/tcp/5375 /ip4/171.214.222.88/tcp/50710 /ip4/171.214.222.88/tcp/35497 /ip4/171.214.222.88/tcp/61058 /ip4/171.214.222.88/tcp/50270 /ip4/171.214.222.88/tcp/61527 /ip4/171.214.222.88/tcp/54526 /ip4/171.214.222.88/tcp/53606 /ip4/171.214.222.88/tcp/64457 /ip4/171.214.222.88/tcp/57888 /ip4/171.214.222.88/tcp/57645 /ip4/171.214.222.88/tcp/57587 /ip4/171.214.222.88/tcp/48227 /ip4/171.214.222.88/tcp/2007 /ip4/171.214.222.88/tcp/56086 /ip4/171.214.222.88/tcp/51588 /ip4/171.214.222.88/tcp/6379 /ip4/171.214.222.88/tcp/62904 /ip4/171.214.222.88/tcp/22604 /ip4/171.214.222.88/tcp/60315 /ip4/171.214.222.88/tcp/63470 /ip4/171.214.222.88/tcp/30586 /ip4/171.214.222.88/tcp/19287 /ip4/171.214.222.88/tcp/19443 /ip4/171.214.222.88/tcp/19123 /ip4/171.214.222.88/tcp/9129 /ip4/171.214.222.88/tcp/45425 /ip4/171.214.222.88/tcp/31301 /ip4/171.214.222.88/tcp/48393 /ip4/171.214.222.88/tcp/61744 /ip4/171.214.222.88/tcp/59020 /ip4/171.214.222.88/tcp/14110 /ip4/171.214.222.88/tcp/16426 /ip4/171.214.222.88/tcp/57084 /ip4/171.214.222.88/tcp/57928 /ip4/171.214.222.88/tcp/33319 /ip4/171.214.222.88/tcp/51117 /ip4/171.214.222.88/tcp/50408 /ip4/171.214.222.88/tcp/62295 /ip4/171.214.222.88/tcp/15373 /ip4/171.214.222.88/tcp/52290 /ip4/171.214.222.88/tcp/55704 /ip4/171.214.222.88/tcp/10987 /ip4/171.214.222.88/tcp/54847 /ip4/171.214.222.88/tcp/64208 /ip4/171.214.222.88/tcp/26163 /ip4/171.214.222.88/tcp/53140 /ip4/171.214.222.88/tcp/59628 /ip4/171.214.222.88/tcp/65120 /ip4/171.214.222.88/tcp/50560 /ip4/171.214.222.88/tcp/53366 /ip4/171.214.222.88/tcp/52089 /ip4/171.214.222.88/tcp/25153 /ip4/127.0.0.1/tcp/4001 /ip4/171.214.222.88/tcp/58684 /ip4/171.214.222.88/tcp/58884 /ip4/171.214.222.88/tcp/63299 /ip4/171.214.222.88/tcp/58365 /ip4/171.214.222.88/tcp/51140 /ip4/171.214.222.88/tcp/41977 /ip4/171.214.222.88/tcp/60675 /ip4/171.214.222.88/tcp/2977 /ip4/171.214.222.88/tcp/28746 /ip4/171.214.222.88/tcp/54140 /ip4/171.214.222.88/tcp/54874 /ip4/171.214.222.88/tcp/9634 /ip4/171.214.222.88/tcp/60342 /ip4/171.214.222.88/tcp/19743 /ip4/171.214.222.88/tcp/58104 /ip4/171.214.222.88/tcp/54546 /ip4/171.214.222.88/tcp/4607 /ip4/171.214.222.88/tcp/64964 /ip4/171.214.222.88/tcp/59606 /ip4/171.214.222.88/tcp/42267 /ip4/171.214.222.88/tcp/63234 /ip4/171.214.222.88/tcp/38997 /ip4/171.214.222.88/tcp/58352 /ip4/171.214.222.88/tcp/62192 /ip4/171.214.222.88/tcp/45225 /ip4/171.214.222.88/tcp/57315 /ip4/171.214.222.88/tcp/37197 /ip4/171.214.222.88/tcp/65230 /ip4/171.214.222.88/tcp/61424

It would appear that this giant list of ephemeral ports is just because node behind 171.214.222.88 was connecting to many peers, and they recorded from which port the connection happened, and passed it to each other. I don't think in that case it it reasonable to pass those ports to each other or attempt to connect to them.

Why are we connecting to so many ports? Can I configure IPFS daemon to only connect to port 4001 (which would work for nodes with public IP)? If some other node does not have public IP, and is behind the NAT, I won't connect to them, but they could dial me any time.

@bonedaddy
Copy link
Contributor

bonedaddy commented Sep 1, 2018

If you're node is advertising all locally connected interfaces then it doesn't appear that the server profile config change persisted as it is designed to stop this kind of advertisement

edit: it appears I misread your comment and that the node advertising all those addresses is a remote node

edit2: what I think might be happening is that the node you're dialing to on multiple ports is advertising content that your node needs so it's attempting to connect to to that peer on all known interfaces.

edit 3: the inverse could be true your node has content which that peer needs so that peer and your node are constantly trying to talk to each other

edit4: this issue may also be worth checking out #1226

edit5: this is another issue worth looking at #4343

@burdakovd
Copy link
Author

burdakovd commented Sep 1, 2018

edit2: what I think might be happening is that the node you're dialing to on multiple ports is advertising content that your node needs so it's attempting to connect to to that peer on all known interfaces.

edit 3: the inverse could be true your node has content which that peer needs so that peer and your node are constantly trying to talk to each other

My node doesn't need anything, and nobody can need stuff from it, since I'm not using it yet, it is a fresh new node. So this is purely a DHT traffic. I don't mind that, but I want to make sure that it only attempts to connect peers in reasonable way, and not by spamming them on dozens of ports.

I've read #1226 and #4343 and those are about node attempting to connect to local addresses. That is mitigated by server profile (which populates Swarm.AddrFilters), and isn't the case in my issue, as my node is attempting to connect to public addresses but on too many ports.

@bonedaddy
Copy link
Contributor

You might be able to mitigate this routing the dhtclient routing mode. If that doesn't work then the only thing I can think of is to use iptables or something similar to block connecting to the offending node.

@magik6k
Copy link
Member

magik6k commented Sep 6, 2018

The problem is likely that the node is advertising too many ports, like in #3780.

@gallizoltan
Copy link

Same issue here.
I started to run IPFS daemon on AWS, and I've got an "Abuse Report" email about "activity which resembles scanning hosts on the internet for security vulnerabilities".
I'm going to try ipfs init --profile=server now.

@alexanderkjeldaas
Copy link

Same here. Why is this not the default? Currently it's HIGHLY DANGEROUS (for business) to run IPFS.

@burdakovd
Copy link
Author

burdakovd commented Sep 22, 2018

To be clear, this particular issue is not about the following two things:

  1. Why ipfs is not enabling server profiler by default
  2. Why it is attempting to connect to local addresses (which has beend discussed in other issues before)

It is rather about why after enabling server profile, ipfs still attempts to connect to the same node (public IP) at dozens different ports, and how do we prevent this from triggering ISP abuse detection.

It might be a bug in the logic of how ports of that node are advertised and forwarded between peers (i.e. if node A is behind NAT and connects to dozens of peers, and all of them remember and forward the public host:port that they receieved connection from, at the end DHT will contain many records of the same public ip at different ports).

If we aren't able to identify why this is happening and fix it quickly, there are some remediations.

Remediation 1. On firewall allow outgoing connections to any port except 4001. This potentially hurts connectivity somewhat, but does help to prevent any activity that can resemble port scanning. This is what I've chosen.

Remediation 2. Change code of IPFS such that it only attempts to connect to new port of an IP address not more than once per day. Maybe also add limitation that it attempts to connect to an IP address not more than once per hour. Of course those limitations should be configurable. Once that is in place, even if some misbehaving node advertises thousands of closed ports on the same IP address, IPFS won't attempt to "scan" all of them in search of the one that is open. It wouldn't hurt connectivity with good peers, as they are likely advertising just one port.

@alexanderkjeldaas
Copy link

It might be a bug in the logic of how ports of that node are advertised and forwarded between peers (i.e. if node A is behind NAT and connects to dozens of peers, and all of them remember and forward the public host:port that they receieved connection from, at the end DT will contain many records of the same public ip at different ports).

Does IPFS use STUN server?

@Stebalien
Copy link
Member

Closing in favor of the meta issue #6932.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

No branches or pull requests

6 participants