DRBD config leak

[Ganeti-Issues-Migrator]

Originally reported of Google Code with ID 1135.

This issue is for discussion about the DRBD config leak.

Originally added on 2015-10-15 16:27:55 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi - glad to communicate with you directly!

Let me shortly sum up things that have been / will be done.

riseup.net has been contacted, and we have verified that they are safe from any issues identified in your report.

We have examined the DoS issue, and unfortunately concluded that further changes to the Python OpenSSL wrapper that Ganeti uses are needed to prevent client-initiated protocol renegotiation. Given that DDoS vs DoS is not that different these days, we might opt for other means of limiting the resource consumption of the RAPI daemon.

The openness of the RAPI has been determined to be a problem, and we are in agreement with the package maintainer that future versions should be locked down by default, and offer better authentication and authorization mechanisms. It will take a while until this gets implemented though, as a forced push of a lockdown with no simple alternative for authentication would break a lot of tools relying on the interface.

The DRBD secret will be purged from any output that can be shown to an unauthorized RAPI user, in earlier versions by omitting it, in later versions by introducing a type-based constraint that we use for other kinds of sensitive data in Ganeti.

RBD comes with its own set of authentication and verification mechanisms that are enabled by default and not controlled by Ganeti. Should anyone disable those, we cannot prevent them from shooting themselves in the foot.

The security releases we'll do later on in the month will involve just the DRBD fix, with other changes arriving in the course of the next few months. The exact date will depend on CVE, package maintainers, etc. More updates will be posted here.

Thanks for taking a look at Ganeti!
Riba

Originally added on 2015-10-15 20:42:20 +0000 UTC.

[Ganeti-Issues-Migrator]

-- Empty comment --

Originally added on 2015-10-15 21:22:00 +0000 UTC.

[Ganeti-Issues-Migrator]

Hello,

thank you for your message.

As google Security said they didn't contact RiseUp, I contacted them 2 weeks ago.
They were vulnerable if the attacker had a control over the DNS servers, but the situation seems to be under control now (iptables everywhere).

1/ About the client-initiated protocol renegotiation, your solution seems to be a good approach.

2/ I agree with you about the openness of the RAPI as a design problem and the fact it is not easy to patch without breaking the compatibility with other tools.

For information, I asked Google Security Team to contact MITRE in order to get a CVE. And I will send you a pre-version of the security advisory before publishing it along with the security releases of Ganeti so you can check it.

Regards,

Originally added on 2015-10-21 20:39:37 +0000 UTC.

[Ganeti-Issues-Migrator]

Thanks Pierre

I contacted MITRE a couple times, and haven't heard back.

Originally added on 2015-10-21 20:40:56 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi Pierre,

In order to make sure this is properly fixed, could you help me understand how the DNS issue interacts with the DRBD secret leak you discovered? When it comes to setting up DRBD connections, Ganeti always uses the IP and not the hostname, so hijacking an existing connection via DNS poisoning would not really work.

Thanks!

Originally added on 2015-10-23 10:22:22 +0000 UTC.

[Ganeti-Issues-Migrator]

MITRE came back! :)

The CVEs are:

> DoS

Use CVE-2015-7944.

>  Info Leak

Use CVE-2015-7945.

Originally added on 2015-10-23 12:48:13 +0000 UTC.

[Ganeti-Issues-Migrator]

Awesome, thanks!

Originally added on 2015-10-23 13:12:06 +0000 UTC.

[Ganeti-Issues-Migrator]

Hello Riba,

You are right, I just checked and I only see IPs in the DRBD configurations.
The perimeter of the problem is less important, then.


For information, I sent an email to security@google.com on Aug 28:

> [...]
> (1)
> As for the Information Disclosure, I consider the infrastructure could
> be vulnerable depending on the configuration of DRBD and Ganeti:
> 
> - If the setup is using DRBDv8, one attacker has to mitm the network
> using ARP or BGP in the worst case.
> Note it's possible to abuse other weak situations, like hacking local     <- this was wrong by checking the DRBD configuration
> DNS servers to force one node to connect to a remote rogue node.          <- this was wrong by checking the DRBD configuration
> 
> I was thinking about a local security problem with a local user in a
> ganeti node connecting directly to the second node using FUSE,
> but there is no user-land implementation of DRBD and I gave up as I
> considered it was too time consuming to implement.
> 
> - If the setup is using DRBDv9, there is a possiblity to directly
> connect to the nodes depending on the configuration:
> 
> http://drbd.linbit.com/users-guide-9.0/s-drbd-client.html
> 
>   # DRBD "client"
>   floating 10.1.11.6:8006 {
>     disk      /dev/does-not-exist;
>     node-id   4;
>  }
> 
> I'm not sure Ganeti currently supports DRBDv9.
> 
> - I still consider Ganeti has a fundamental problem of its design - if
> my assumption holds: Ganeti offers the possibility of using a CEPH
> back-end instead of DRBD. Information about CEPH/RBD is likely to be
> retrieved using the PoC. However, it still remains an "assumption" as
> I haven't entered to the detail because it was still on a Beta status
> and I also did not have my infrastructure good enough to analyze.
> 
> Then concerning the official documentation for Ganeti:
> (http://docs.ganeti.org/ganeti/current/html/install.html ),
> it doesn't specify anything about network segmentation using (V)LAN
> for the DRBD replication. It would be better if they can add something
> on it.


Auditing Ganeti is not easy, as the infrastructure is not easy to setup
and to correctly configure (with all the different options).


evn@: Thank you!


Regards,

Originally added on 2015-10-23 20:31:35 +0000 UTC.

[Ganeti-Issues-Migrator]

It's my fault that I did not explicitly say that this cannot be done with DNS poisoning - my answers instead just said that you had to be capable of ARP spoofing or routing the traffic out of the cluster to retrieve data. Sorry about that - we moved the communication here exactly so that we could converse directly and clarify misunderstandings. And RiseUp is probably better off with additional protection against DNS poisoning, so no harm done there :)

While I am aware that setting up Ganeti is not a walk in the park, just out of curiosity, did you ever try setting up OpenStack? :D

Cheers,
Riba

Originally added on 2015-10-27 14:35:30 +0000 UTC.

[Ganeti-Issues-Migrator]

Hello,

thank you for your message.

From my tests, it is possible to send an unauthenticated request to the RAPI daemon, resulting in a creation of a job:

  - "https://ip:5080/instances/info"

A Ganeti job will be created for each request. I think this request can be used to flood Ganeti with useless jobs and may disrupt Ganeti.

About OpenStack, I will audit it in the future :D

Regards,

Originally added on 2015-11-03 15:21:36 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi,

It is true that allowing job submits might cause inconvenience for users because it allows the DOS to "leak" to other daemons, but luckily we limit the number of jobs that can concurrently be running. The limit is 20 right now, but we can deploy an additional per-RAPI limit easily, barely touching the code.

Regarding the fixes: as we've been swamped by unexpected and pretty important work recently, I did not manage to have the patches ready by start of November. Will have the time for them very soon though!

Best,
Riba

Originally added on 2015-11-05 13:20:46 +0000 UTC.

[Ganeti-Issues-Migrator]

Hello,

FYI: I didn't find new vulnerabilities in Ganeti and I just stopped the research.

Do you have information about upcoming releases of Ganeti ?

Regards,

Originally added on 2015-11-12 16:53:59 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi!

Expect the set of releases to happen early next week.

Cheers,
Riba

Originally added on 2015-11-17 17:38:08 +0000 UTC.

[Ganeti-Issues-Migrator]

Hello,

I'm going to send you the security advisory tomorrow (to riba@), so you can review it (it's going to be a big copy/paste from the email I sent concerning the vulnerabilities..).

Regards,

Originally added on 2015-11-19 14:24:55 +0000 UTC.

[Ganeti-Issues-Migrator]

Sure, but also CC evn@ please, or I can do that for you. Thanks!

Originally added on 2015-11-20 13:52:25 +0000 UTC.

[Ganeti-Issues-Migrator]

I just sent a draft of the advisory to riba@ and evn@.

Originally added on 2015-11-24 14:08:48 +0000 UTC.

[Ganeti-Issues-Migrator]

Thanks for the advisory!

I have to unfortunately inform you that the releases will not be done this week - the long time this went undetected means that six separate releases are needed, and that's simply taking a while.

Will follow up Monday!

Originally added on 2015-11-26 16:58:24 +0000 UTC.

[Ganeti-Issues-Migrator]

Hello,

have you the date of the releases ?

Regards,

Originally added on 2015-12-01 17:49:06 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi,

No, but I'm hoping to be able to follow up with a date soon.

The DoS vulnerability is the blocker. It cannot be addressed easily, given that the Python OpenSSL library we use, even if fixed, cannot be updated in all supported OS's containing versions 2.9 through 2.15. The same holds for the cgroup-based mitigation (systemd vs non-systemd), and Ganeti filter-based mitigation (2.14 onwards).

So I have been talking to the Debian maintainer, and the option of offloading the SSL aspect of RAPI to nginx / haproxy came up - that's what I'm exploring right now.

As all the necessary options are already present in Ganeti, it will be done on the packaging level, so the release code is being prepared in parallel right now.

More updates soon :)

Cheers,
Riba

Originally added on 2015-12-01 19:38:45 +0000 UTC.

[Ganeti-Issues-Migrator]

Hello,

what are the status of the 2 vulnerabilities (fixed?) ? Do you have information about upcoming releases of Ganeti ?

Regards,

Originally added on 2015-12-28 15:09:52 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi Pierre,

Sorry for not announcing this earlier, but we just announced the vulnerabilities today.

You have been credited both in the release mail (https://groups.google.com/forum/#!topic/ganeti/9bLyzwmmvdg) and the oCERT advisory (http://www.ocert.org/advisories/ocert-2015-012.html).

The RAPI is getting locked down in packages coming out today, and the DRBD secret leak has been mitigated. We will pursue a better solution for the SSL renegotiation issue, but we have at least published instructions for how to make Ganeti cooperate with a proxy.

Thanks a lot for all the work you have put into this from the Ganeti team!

Cheers,
Riba

Originally added on 2015-12-30 13:52:01 +0000 UTC.

Changed State: Released

[Ganeti-Issues-Migrator]

Hello,

thank you for your message.

I will release a complete advisory today (with technical details).

Regards,

Originally added on 2016-01-04 11:57:53 +0000 UTC.

[Ganeti-Issues-Migrator]

Excellent, thanks!

Originally added on 2016-01-04 12:01:13 +0000 UTC.