Security Issue: if -ip not specified, servers bind to 0.0.0.0/0

[rotproofbits]

It appears that if -ip.bind is not specified, at least master and volume (and probably filer) bind to all interfaces, including non-local interfaces. Since there's no authentication or security, this presents a big risk by exposing servers to the internet by default (such as the big security problems that MongoDB and Redis had in the last few years that left thousands of servers across the internet vulnerable to takeover and data exfiltration).

It would be safer if -ip.bind was assumed to be 127.0.0.1/32, unless otherwise specified.

[rotproofbits]

It'd also be really helpful for documentation for me personally ;) if the difference between -ip and -ip.bind were explained a little more, since this is all it really says:

  -ip string
    	ip or server name (default "192.168.1.66")
  -ip.bind string
    	ip address to bind to (default "0.0.0.0")

I was attempting to specify the IP address of the server interface as -ip (since that is shown in much of the documentation), and was shocked to learn that it was publicly available.

However, -ip.bind appears to work great for that, but -ip appears to limit communications (?), or really unsure what -ip is used for.

[chrislusf]

Added more help message. It will be too limiting if -ip.bind is a required field.

[rotproofbits]

Cool, that was fast!

It should just bind to 127.0.0.1 instead then and be secure by default (https://en.wikipedia.org/wiki/Secure_by_default) to prevent MongoDB issues (such as https://medium.com/pixiters/find-open-mongodb-instances-on-the-internet-b8b83acc08ac , fixed in 3.6 https://docs.mongodb.com/manual/core/security-mongodb-configuration/ )

The user can always specify -ip.bind=0.0.0.0/0, no?

The default is extremely insecure, and it doesn't make things easier if you're just doing testing, since you can always just specify 0.0.0.0 instead.

[chrislusf]

SeaweedFS should not be put out in the internet. It's not good to just fix this and have a wrong sense of safety.

[rotproofbits]

Please consider where swfs' most likely users will be deploying it:

1. most likely people won't bother bringing up a seaweedfs cluster in a virtual private cloud (VPC) somewhere on expensive vm's when they could just use their cloud's own cheap S3 equivalent, so mostly not the cloud except in a few unusual cases
2. not corporate datacenters behind a firewall (most likely), because (as someone who has served as CISO or worked in CISO office in at least four companies), most corporations have a pretty healthy sense of security if they're still operating their own datacenter today, or the datacenter is just for legacy apps that haven't been brought over to the cloud, and most security types are allergic to M&M security (crunchy on the outside firewall, soft and delicious in the middle!)

So, who uses it? People who are interested in serving billions of files fast. ;) .. but, really, people looking for S3-like services, but on cheap hardware, and who (for whatever reason) can't choose Wasabi or B2. Most likely, people who are taking file uploads from people across the internet. Startups, etc, like my company.

These people have at least a sense of security, and even if a firewall will eventually be in place, they might be developing on a set of VM's that are exposed to the internet. And mistakes happen, especially if you assume that the dev of whatever project you're working on has taken adequate precautions.

Huge breaches have occurred because of insecure-by-default; how many S3 buckets have been left open, and now Amazon finally has tools (finally!) to help you close them down, and even emails you every month if you have public buckets, to warn you of the implications?

Please reconsider. Seaweedfs is way too cool to not have a strong security posture, and it costs you nothing to simply bind to localhost unless a user specifies otherwise. Thanks for reading.

[kmlebedev]

@rotproofbits Seaweedfs is a distributed storage about what localhost are you talking about, it simply will not work?
Need more security deploy in your private k8s cloud.

[rotproofbits]

@kmlebedev what won't work? Most examples show it as deploying on different ports on localhost, because most people will try it out that way. (Deploy in k8s introduces lots of extra complexity for a storage system, especially because this requires stateful containers.)

The very first examples in the README (https://github.com/chrislusf/seaweedfs#example-Using-Seaweed-Object-Store) only rely on localhost:

./weed master
# do you think this binds to localhost? or all interfaces? because look below:

./weed volume -dir="/tmp/data1" -max=5  -mserver="localhost:9333" -port=8080 &
./weed volume -dir="/tmp/data2" -max=10 -mserver="localhost:9333" -port=8081 &

Every data store that can leak data should bind by default to only localhost, and then users can choose to open it up further when they wish. Why is this even an argument? is it such a pain to specify -ip 0.0.0.0/0 when you really want to bind to the world?

[kmlebedev]

@kmlebedev what won't work? Most examples show it as deploying on different ports on localhost, because most people will try it out that way. (Deploy in k8s introduces lots of extra complexity for a storage system, especially because this requires stateful containers.)

The very first examples in the README (https://github.com/chrislusf/seaweedfs#example-Using-Seaweed-Object-Store) only rely on localhost:

./weed master
# do you think this binds to localhost? or all interfaces? because look below:

./weed volume -dir="/tmp/data1" -max=5  -mserver="localhost:9333" -port=8080 &
./weed volume -dir="/tmp/data2" -max=10 -mserver="localhost:9333" -port=8081 &

Every data store that can leak data should bind by default to only localhost, and then users can choose to open it up further when they wish. Why is this even an argument? is it such a pain to specify -ip 0.0.0.0/0 when you really want to bind to the world?

@rotproofbits
This is an example for a local demonstration of work, in a product story you don't want to do that.
And you will need at least 3 hosts for the master, and they can also be a filler and a volume.
Where communication takes place on gRPC everyone with everyone
Will not work on localhost in any way

[rotproofbits]

Exactly. This is "security 101". https://github.com/chrislusf/seaweedfs/wiki/Production-Setup shows how to specify -ip when you are going into production.

[kmlebedev]

shows how to specify -ip when you are going into production.

But why set a default so that nothing works?
I understand your concerns, but it seems that this needs to be addressed by other means.

[chrislusf]

@rotproofbits seems we can add a section to https://github.com/chrislusf/seaweedfs/wiki/Production-Setup about put it out on internet.

[rotproofbits]

But why set a default so that nothing works?

You set the default so that nothing works insecurely, and force people to explicitly enable the insecure binding to all IPs.

This is how nearly all other database and data store servers work, even though you will almost always be connecting to it from remote IPs when going into production.

"By default PostgreSQL is configured to be bound to 'localhost'". https://bigbinary.com/blog/configure-postgresql-to-allow-remote-connection

"The default MySQL server listen on localhost only. Which means no one can connect MySQL from the remote system. Which is also a good practice for the security purposes." https://tecadmin.net/mysql-allow-remote-connections/

"By default, Redis is only accessible from localhost. However, if you installed and configured Redis by following a different tutorial than this one, you might have updated the configuration file to allow connections from anywhere. This is not as secure as binding to localhost." https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-20-04

[kmlebedev]

@chrislusf I got the main point
I understood the main idea I need to separate the ip.bind for internal services(gRPC) and the main service(filler http port and s3 http port). Is it possible?

[chrislusf]

@rotproofbits there are many on-prem installations, and also deployments in k8s. Changing the default behavior for -ip.bind is disruptive.

[chrislusf]

I need to separate the ip.bind for internal services(gRPC) and the main service(filler http port and s3 http port). Is it possible?

what is the use case?

[dumblob]

Could this issue be reopened?

I've come to this just randomly by evaluating security concerns regarding Seaweedfs. We're kind of reluctant to deploy SW which is not mature enough (i.e. lacks the hard learned lessons about its security over the years of its existence) to realize the facts @rotproofbits laid out.

[chrislusf]

@dumblob you can use TLS. Relying on unencrypted http is not safe anyway.

[dumblob]

Nobody (incl. myself) wants to rely on unencrypted HTTP nor anyone advocated for that. Quite the opposite - @rotproofbits tried to explain in very detail why defaults matter and why the current defaults are really off.

That's one of the few main reasons why we could not choose seaweedfs. As it seems to be unwise to rely on the decisions of the upstream.

[chrislusf]

Added a fix. @rotproofbits @dumblob Thanks for suggestions and insisting on this!

[codewinch]

Thanks @chrislusf et al - this is a huge step forward and makes me want to try out seaweedfs!!

[imfantuan]

Maybe you should never set "systemctl disable firewalld" for your "Security" Server.

[imfantuan]

@kmlebedev what won't work? Most examples show it as deploying on different ports on localhost, because most people will try it out that way. (Deploy in k8s introduces lots of extra complexity for a storage system, especially because this requires stateful containers.)

The very first examples in the README (https://github.com/chrislusf/seaweedfs#example-Using-Seaweed-Object-Store) only rely on localhost:

./weed master
# do you think this binds to localhost? or all interfaces? because look below:

./weed volume -dir="/tmp/data1" -max=5  -mserver="localhost:9333" -port=8080 &
./weed volume -dir="/tmp/data2" -max=10 -mserver="localhost:9333" -port=8081 &

Every data store that can leak data should bind by default to only localhost, and then users can choose to open it up further when they wish. Why is this even an argument? is it such a pain to specify -ip 0.0.0.0/0 when you really want to bind to the world?

It is a "distributed" system, and most distributed system are not just for one "Personal computer".
So in my opinion, you are not the "most people".

[imfantuan]

酷，那太快了！

它应该只绑定到 127.0.0.1 默认情况下是安全的（https://en.wikipedia.org/wiki/Secure_by_default）以防止 MongoDB 问题（https://medium.com/pixiters/find-open -mongodb-instances-on-the-internet-b8b83acc08ac，在 3.6 中修复https://docs.mongodb.com/manual/core/security-mongodb-configuration/）

用户总是可以指定-ip.bind=0.0.0.0/0，不是吗？

值是_非常不安全_的，如果你只是在做测试，它不会让事情更容易，因为你总是可以指定 0.0.0.0 来代替。

I can show you how to follow this: https://en.wikipedia.org/wiki/Secure_by_default

1. enable firewalld
2. start firewalld
3. you are safe now