Exposing less ports

[wernight]

Related to issue #1554, is there a way to only expose something like 10 ports or less? I find it especially strange to require exposing so many port for a single-user system.

[garagatyi]

Currently we are using docker auto-publish feature for port exposure. That's why we can't control the number of ports that should be opened on a server.

[wernight]

Oh I see. One way would be not to expose that port directly to the internet, and proxy it via the che server (the one running on port 8080), possibly on another port. Or let users specify the port on which it'd get exposed.

[TylerJewell]

@wernight - that is correct. That is a much trickier solution. It's what we do in Codenvy and one of the ways that we are able to enforce user-based permissions. All traffic in Codenvy runs through a go-router that is installed as part of the system. And this way we can use tokens to enforce identity and permissions, including some port proxying, but not to the degree you are suggesting here.

We'd need to run some sort of in-memory port-port mapper. This could make the product more secure as you could have your docker daemon that is managing workspaces non-accessible by any IP external IP address, so that only the Che server is the sole point of contact.

This would be a major improvement to the underlying subsystems. Not sure how to label this, as we'd have to look at a generational major version for such a capability.

[wernight]

I'm not sure how Docker is used internally, but I expect something like this docker-compose.yml:

nginx:
  image: nginx
  ports:
    - 80:80
    - 8080:8080  # Probably needs one more, possibly one per container.
  links:
    - che-server
    - some-docker-container
che-server:
  image: ...
some-docker-container:
  image: ...

I do see the issue that some-docker-container is actually dynamic, hence requires a dynamic port mapping. I'm willing to help, but I don't know where to put the code. For example I don't know if I should add a hook that opens a port externally upon being called, or if all connection can go through port 80 and some kind of virtualhost can be applied to then detect where it should go.

The push model (hook being called by Che when a workspace is started), would allow also to simply expose the port, which would allow a Kubernetes deployment with some hacks. The pull model (when clients try to connect they tell which container they're trying to reach) allows a much cleaner model that would be simpler to deploy and safer.

The current solution requires a dedicated machine which is too costly, not really safe IMO, and doesn't support re-deploy on failure like Kubernetes allows.

Another idea, for the current single-user would be to allow only a single running instance of a workspace/container. This would mean that the docker-compose above could be used directly.

PS: What are port 8000 and 49152-49162 used for in the Docker image?

[TylerJewell]

The 8080 is the port of the Che server itself. The other ports you shouldn't be exposing - the docker run syntax was outdated vs. the latest syntax we support now. Each workspace starts by exposing ports at the beginning of the ephemeral port range, which starts at 32768, I believe. The ports are then opened in sequence.

I think @garagatyi believes the proper solution would be to enable some sort of port restrictions for each workspace when the workspace is launched. This would be modifying the syntax around our "docker run" command to pass in these parameters if they are defined in the che.properties file. So maybe it's restricted to a 10 port range, for example and then it passes in the publish attributes into the docker run.

We love pull requests - definitely want to improve here. Alex would have the sort of pointers necessary to outline the implementation plan for such a solution.

[wernight]

Yes simply cycling 10 or so ports sounds also pretty feasible, but still how is the network protected? Putting 8080 behind Nginx with basic authentication is easy, but what's the authentication that one can set up on the other ports?

[TylerJewell]

@wernight - we support permissions API and full authentication inside of Codenvy right now. Codenvy is a sytem that is locked down. There are many elements to the engineering that has been done. We will bring some of this into Che, though we are trying to figure out what are the elements that we can bring into the system without adding too much complexity to the architecture.

Let me elaborate what codenvy does:

1. There is a central user database.
2. There are multiple forms of authentication, which can generate user-specific tokens.
3. There is a broad-based permissions API that restricts access to resources within the Che server (can I create a new ws?), and also to resources within a ws (am I allowed to execute commands?).
4. We then have an enforcement of permissions that lives in a few places - access to the dashboard, within the che server, within the raw workspace (agent inside), and then also customizations to the web terminal running in each workspace so that you cannot randomly open a terminal if you know the IP:PORT, and then also within each ws agent that is running which exposes REST commands from the workspace to the outside world.
5. http/s support for all link points.

So authentication is a generic term - but because we have multiple touch points where software runs within a single che system, authentication / permissions have to be extended to each point on the system.

BTW - we do let orgs run Codenvy with a Fair Source 5 license - so you can operate it for small groups with certain restrictions for free.

But I think locking down the ports is absolutely essential within Che. We also know that Bitnami is adding some Apache filters to add in basic authentication that sits in front of a Che server.

[wernight]

That does seem pretty complicated. Jupyter uses WebSockets but through a single port. I have no idea about the protocol the agent/terminal/... are using through the exposed random Docker port. Supposing all those could use a HTTP protocol, it would simplify a lot of things while still allowing WebSocket performance. It'd allow only exposing a single port. It'd also allow basic AUTH to limit access to everything and HTTPs for everything.

I'll look again into Codenvy free tier (now that I found how to set up SSH key). Binatmi doesn't sound flexible and light enough.

[wernight]

This remains my No. 1 issue with Eclipse Che as I see no way to safely deploy it anywhere that is publicly accessible and I see no way to guard other opened ports. They may be opened only while a workspace is running but still. I it requires settings up an authenticated revert proxy with WebSocket support (not sure if nghttpx supports that).

Another fix that could make it a non-blocker for people like myself may be to have at least a token required for accessing those extra ports. This would allow to setup rate limits and https on the HTTP port and make it pretty safe. I'm not suggesting the whole multi-user, just a nonce provided on initial connection to HTTP (without authentication).

[garagatyi]

To protect ports with some sort of authentication we would need to change the way we allocate these ports.

@wernight Can you provide an example how opening ephemeral ports make system vulnerable?

[wernight]

I don't know what protocol you're using and I don't have the resources to investigate. Based on what I know, it seems possible that Eclipse Che exposes a port to access the terminal on a port in 32768+. That would mean that a port scanner may bypass any authentication set on port 80.

[garagatyi]

But port scanner can scan only 32768-... ports. And you probably don't have any other software listening on these ports. System services use ports lower then 10000 usually. Ephemeral ports are used for outgoing connections, so pretty safe in that case.

[wernight]

I have to open 32768-65535 ports for Che to work (else remains stuck during Starting workspace runtime after Successfully built ...). Also the network diagram shows that the server is listening on those port:
. Also the started Docker image opens ports:

• 0.0.0.0:32845->22/tcp
• 0.0.0.0:32844->4401/tcp
• 0.0.0.0:32843->4403/tcp
• 0.0.0.0:32842->4411/tcp
• 0.0.0.0:32841->8000/tcp
• 0.0.0.0:32840->8080/tcp
• 0.0.0.0:32839->9876/tcp

And I can just SSH in my public IP on port 32845 (it does seem to require a password but not sure if it's a nonce).

[garagatyi]

0.0.0.0:32845->22/tcp
0.0.0.0:32844->4401/tcp
0.0.0.0:32843->4403/tcp
0.0.0.0:32842->4411/tcp
0.0.0.0:32841->8000/tcp
0.0.0.0:32840->8080/tcp
0.0.0.0:32839->9876/tcp

It is a mapping of your public ports on host to ports inside of containers. So no one will be able to connect to your host system. Only inside containers. Because of that I don't think it is too risky to open 23768+ ports.
But if you afraid that someone gets into container then yes, authentication is required. But it doesn't have connection to range of ports you have to open.

[wernight]

About the risks:

• Docker containers do not contain. They limit but they are not a sandbox environment. That's why one should not run as root inside containers for example.
• I believe one of the most important and useful Che runtime, is one able to build and run docker images. This means basically root access from within the container, which is pretty close to a root access on the host machine.

[garagatyi]

It is possible to map root in container to non-root on host, but docker inside container is very dangerous.

[wernight]

Yes latest Docker adds some improvements, still not sure that's on by default. I imagine new users making a custom workspace running as root or other mistakes resulting in an unexpected attack vector.

Outside of having root access, there is access to the entire project source code even if user had set up Nginx authentication for HTTP on port 8080.

[garagatyi]

root mapping is disabled by default AFAIK.

Outside of having root access, there is access to the entire project source code even if user had set up > Nginx authentication for HTTP on port 8080.

Can you describe that situation more. I didn't understand what do you mean

[wernight]

Assumptions: Putting Che port 8080 behind Nginx reverse proxy for authentication isn't hard so let's assume that this is safe. Again I'm assuming here that one of the exposed ports in 32768-65535 does not require authentication nor a cryptographic nonce.

Risk: As you've imported your project, an SSH access inside the container means at least read access to /projects/my-project and allows to dump your entire source code.

[garagatyi]

So setting authentication for all exposed ports will be secure solution, right?

[wernight]

Ideas for fixes:

1. Require authentication or nonce on all exposed ports (which however doesn't allow deploying on Kubernetes I think).
2. HTTP reverse proxy through port 8080, or any single fixed port actually, instead of exposing ports 32768-65535 (adding authentication to an HTTP port is easy).
3. Don't require opening other ports than 8080 and use hole punshing to establish a connection (probably the hardest to implement and not the most secure here)
4. Require VPN, or using a proxy like SPDY or HTTP/2 (hack to achieve something similar to solution (2))

I believe solution (2) gives the most advantages, like allowing to deploy on Kubernetes and a lot more. It might even be the simplest to implement.

[wernight]

OMG! I just found out that it's actually the same on Codenvy. I didn't validate that it affects private repository but I think you should immediately block any SSH connection for users who didn't give a custom user/password and/or key.

The issue is that one can easily port scan the few codenvy IPs and SSH into any running workspace, private or public. This includes without limiting to copying the entire source code, using those machines to perform DDoS or other mischief, potentially access passwords or SSH keys to access more resources, inject code the project being developed, inject code into the browser of the user working on developing on web servers to potentially steal cookies...

[TylerJewell]

We agree on the reverse proxy sokution. We have to make some modifications to Che to support having all browser - workspace comms go through a single reverse proxy port. But we know what to do and will get it scheduled.

[TylerJewell]

@wernight - On the SSH of workspaces issue. It's possible to remove the SSHD from the workspaces when you create a custom workspace. We do make them necessarily simple for access by default. We have designs for a more advanced ipmlementation of the SSH stuff where we will do exactly what you suggested, where users can upload the appropriate key and / or a custom user name and password. We just haven't implemented that segment yet. It requires some coordination between the user dashboard and the IDE.

[skabashnyuk]

The issue is that one can easily port scan the few codenvy IPs and SSH into any running workspace, private or public.

@wernight Can you explain how it's possible without ssh keys?

[wernight]

ssh -p 34888 user@node1.beta.codenvy.com and the password is secret (scan here the port# and node# of course). I wouldn't put a full exploit here publicly, but I wouldn't mind to try it if you someone from Codenvy with authority allows me.

[skabashnyuk]

yes you right. @wernight do you think that randomising password will close this problem?

[TylerJewell]

@allquixotic - see this proposed improvement, which would help with port reduction on a system like OpenShift. #2004 (comment)

Also, please try codenvy. It has strict authentication and permissions enforcement. We give the software away for up to three users, so if it's just for personal use, it's going to meet your needs. It does not restrict the port range, but it does have strong forms of security to ensure that your workspaces are only accessible by people you give permission to. We are about to announce next week at CheConf that you can run Codenvy with the same CLI that you can run Che with.

@l0rd

[monaka]

IMO, it's a enough reasonable solution to use with a forward proxy server with authentications.
Che itself should be light-weight as possible as.

Probably your client browser have no WebSockets support also if it doesn't support PAC.
And some clients may be behind firewalls that open 80, 443 only.
I'm not against reducing ports. But it doesn't resolve all.

[TylerJewell]

@wernight - working to improve Che so that all traffic goes through a single port, including workspaces, is going to be taken as an issue in Q1. it's a challenging issue, but everyone agrees on the general direction of what to do.

[vinokurig]

Is this still an issue? If so please let us know otherwise in 1 week we will close this due to inactivity.

[TylerJewell]

This is still an issue - this should remain open until we have reverse proxy with port management.

[TylerJewell]

No. This is going to take many quarters before we see a solution.

[Metal-Mighty]

You mentioned something in the works by RedHat on one of my issues this year (#3603), has anything been released yet?

[TylerJewell]

you can see all of the development in the repo. There is no open branch of dev yet on this. Just early designs. It is a massive undertaking to build it and test it. Probably thinking at least six months of work.

[wernight]

It seems that Codenvy also requires ports tcp:32768-65535 to be open. It states on the documentation that "This range can be limited." however it doesn't state how. Like one should inform Codenvy about there currently allowed range. Like, I tried just reducing the allowed range in the firewall and then info --network command fails to connect to the agent (so it'd fail to start).

This seems common to Che/Codenvy.

[ghost]

@wernight I think info --network uses the same port all the time. I can't find that function quickly. @TylerJewell is it correct? Does fakeagent container uses the same port all the time?

Also, I think reducing allowed range is firewall isn't the right way. Does it work if you do so and run a random container with -P. Does Docker use the beginning of your range or start picking ports from the beginning of the ephemeral port range?

In other words, you do not have to inform Codenvy that you have got a new ephemeral port range, but you should inform Docker. Codenvy just lets Docker published all ports exposed in a container.

[wernight]

So it sounds there is a possible bug on Codenvy, not sure for Che.

It's possible tell Docker to use a specific port (instead of using its auto-port selection) and manually select it within a limited range provided via some configuration flag. The reason limiting the ports to let's say 10, especially for Che:

• Can manually list them one by one (for example Kubernetes requires that, also reverse proxy require that)
• Who would need more than 10 on a single user Che instance? If they do, they could increase the range, or have multiple Che.

[garagatyi]

We use ports auto publishing implemented in docker. So it is docker who controls range. And docker gets this range from system ephemeral port range.
I agree that it would be better to configure ports range in Che/Codenvy but it is not implemented.

[wernight]

Looks like there is no way to limit that just for some containers in Docker right now, see moby/moby#13322

[ghost]

@wernight have you tried limiting this range on the OS level?

[wernight]

One could change it in /proc/sys/net/ipv4/ip_local_port_range, and I'll make a note of that in Che on k8s, but that seems very wrong generally speaking.

[TylerJewell]

Given improvements in the recent versions of Che and Codenvy, those limitations will not achieve the goal you are seeking. The only solution here is for us to provide a new server resolution strategy that channels all traffic through a reverse proxy under a certain port. It will be done some time this calendar year, but we cannot offer any specific timeframe.

[TylerJewell]

We now have a prototype for how we will channel everything through a single port. This does not yet channel SSH traffic, but all primary workspace traffic. #4440 (review)

I should note that this code may or may not get committed. We haven't been able to figure out a clean way yet to solve the SSH issue and we have to evaluate the impact to custom assemblies of Che like Codenvy, still.

[wernight]

I think SSH can be left separate for now as it's also an optional thing for a cloud IDE with a terminal. That could also allow to rate limit or block SSH if so desired, and it's a fixed port as far as I recall.

[davidwindell]

Can this issue be closed now we have #4361

[ghost]

Not only that. With Che on OpenShift all internal networking stuff is handled by OpenShift HaProxy, services and routes.

[davidwindell]

Awesome, but don't forget about those of us who don't use OpenShift ;)