distribute private SSH-key only to master / candidates

[Ganeti-Issues-Migrator]

Originally reported of Google Code with ID 377.

Hi,

since all SSH actions are initiated from the master node (and potentially the candidates), it makes sense to copy the private part of the SSH key only to master/candidates.

This would enhance security, especially in following example setup:

A Cluster of two groups. One group holding instances for an internal network, the other holding instances i.e. for a DMZ. Now someone takes care, that master candidates are only nodes in the internal group (BTW can this be set at group level?).

Now think of a HV-escape to root on a DMZ-node. Without the private SSH key on this DMZ-Node, no other node will be compromised. right?

Ganeti already handles candidate pro-/demotion and therefore copies/removes the cluster config. Perhaps handling private SSH keys could also be done in this workflow?

Thanks, Sascha.

Originally added on 2013-02-19 10:31:13 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi Sascha, thanks for filling this issue.

Note that there are two issues here:

- the SSH private key, which allows full node compromise; this we can stop copying, indeed
- the cluster RPC private key, which allows full VM control, and if Ganeti has bugs, it could also allow full node compromise

Restricting the latter is also possible, but involves much more work. Just a heads-up that even if we restrict the SSH key, there is still the possibility of escalation.

Originally added on 2013-02-19 11:15:11 +0000 UTC.

[Ganeti-Issues-Migrator]

Hi Iustin,

thanks for pointing out the cluster RPC private key issue too. And indeed, full VM control is an impact I can't neglect. However, starting with restricting SSH is from my point of view a (big) step into a more secure direction.

Thanks, Sascha.

Originally added on 2013-02-20 07:39:37 +0000 UTC.

[Ganeti-Issues-Migrator]

-- Empty comment --

Originally added on 2013-11-29 09:21:28 +0000 UTC.

Added Labels: Type-Enhancement Priority-Medium
Changed State: Accepted
Added to Milestone: Release2.12

[Ganeti-Issues-Migrator]

FYI, there is a design doc out for review: https://groups.google.com/forum/#!topic/ganeti-devel/O1HjIs0janY

Originally added on 2013-12-04 15:02:37 +0000 UTC.

[Ganeti-Issues-Migrator]

The SSL part is fixed with the following commits (in 2.11). The SSH part is still open.

commit 221146777be8a6e2ad2543065b56597385c14f27
Author: Helga Velroyen <helgav@google.com>
Date:   Fri Feb 28 08:48:45 2014 +0100

    Setting correct permissions of client cert (split-user)
    
    This patch makes sure that the client certificate gets
    the right permissions and owner when created. Additionally
    it enhances the 'ensure_dirs' script to correct the
    permissions in case they are broken for whatever reason.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Jose Lopes <jabolopes@google.com>

commit 46ae85de9ca72de3b42a50c6bb2af9a17b04966e
Author: Helga Velroyen <helgav@google.com>
Date:   Thu Feb 27 15:33:28 2014 +0100

    Add some whitespace to fix formatting
    
    Some error messages were lacking some spaces between lines
    to make it more readable.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Klaus Aehlig <aehlig@google.com>

commit da27bc7dd3830bd9a435cbd0fbcad6a6829dd6aa
Author: Helga Velroyen <helgav@google.com>
Date:   Thu Feb 27 14:38:56 2014 +0100

    Consider old client cert only when available
    
    This fixes a bug which occurred only after upgrading
    from 2.10 to 2.11. During the cluster renew-crypto
    operation, Ganeti tries to include the old certificate
    in the candidate map while it is providing new
    certificates. This failed when there was no certificate
    file existing before (which is the case after an
    upgrade). This patch tries to include the old certificate
    only if it is available.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Klaus Aehlig <aehlig@google.com>


commit 992fd37df59dd792f57cc68f163b1ab70f2da1bc
Author: Helga Velroyen <helgav@google.com>
Date:   Wed Feb 19 14:48:19 2014 +0100

    Updating security doc wrt to SSL security
    
    This patch updates the security document with respect to
    the recent changes in RPC security. For details see
    design-node-security.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Klaus Aehlig <aehlig@google.com>

commit 3e8a6f39aeb7a31319dfbd5e65a3ee02ecae2baa
Author: Helga Velroyen <helgav@google.com>
Date:   Wed Feb 19 15:51:17 2014 +0100

    Smooth renewal of client certificates
    
    This patch fixes another chicken-and-egg problem which
    occurred when the node certificates get renewed. When
    renewing a node certificate, the previous certificate
    has to be used to update the configuration. To address
    this, we keep the digest of the previos certificate
    around till the new one is written to all nodes in
    the configuration.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Klaus Aehlig <aehlig@google.com>

commit 0565f8623dd81d3888a10b5451cb38ba98c80569
Author: Helga Velroyen <helgav@google.com>
Date:   Thu Feb 13 19:27:18 2014 +0100

    Update design doc wrt to improved SSL design
    
    This patch updates the design document of Ganeti's node
    security to make it consistent with the implementation
    and to extend it with a couple of suggestions to improve
    the SSL security even more.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Klaus Aehlig <aehlig@google.com>

commit e593c9c8fcd98a049017a65917a6e45aac93d3b1
Author: Helga Velroyen <helgav@google.com>
Date:   Thu Feb 13 19:31:20 2014 +0100

    Test node certificate renewal in QA
    
    This extends the QA by explicitely testing the renewal
    of SSL client certificates.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Klaus Aehlig <aehlig@google.com>

commit ab4b1cf20b3b86f02ef932327a60a6529cdac6bf
Author: Helga Velroyen <helgav@google.com>
Date:   Thu Feb 13 13:53:08 2014 +0000

    Use node UUID as client certificate serial number
    
    It turns out, that some implementations of OpenSSL are more
    pedantic in checking the certficates than others. In this
    particular case, the SSL connection could not be
    established when the serial number of the certificates
    was not unique.
    
    To avoid this problem, this patch extends Ganeti's X509
    infrastructure to set the certificate's serial
    number. In case of client certificates, we now use the
    node's UUID as serial number, because the UUIDs are
    assumed to be unique in a cluster. This is however still
    not complying to how SSL was designed to be used, but at
    least it is a lot better than setting every serial number
    to 1, which was used before and is still used for other
    certificates than the client certificate.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Klaus Aehlig <aehlig@google.com>

commit c14100486de77f68a6535a8e461630f98a86533e
Author: Helga Velroyen <helgav@google.com>
Date:   Fri Jan 10 14:12:23 2014 +0100

    Add certificate of auto-promoted master candidates to map
    
    When a normal node is auto-promoted to be a master
    candidate, its SSL client certificate digest needs
    to be added to the map of candidate certificates
    as well.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Klaus Aehlig <aehlig@google.com>

commit ed7487711f0e39907443e9404a368b1ea87fec3f
Author: Helga Velroyen <helgav@google.com>
Date:   Wed Jan 8 15:14:12 2014 +0100

    Correct exception when ssconf file does not exist
    
    After an upgrade to 2.11, the ssconf file for the master
    certificates might not exist. Based on the non-existance,
    noded falls back to a compatibility mode regarding dealing
    with SSL certificates. The check for the ssconf file
    caught the wrong exception, which resulted the fall-back
    mechanism to fail.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Klaus Aehlig <aehlig@google.com>

commit fc6ccde470f68734741962f16debb0b8beb22784
Author: Helga Velroyen <helgav@google.com>
Date:   Wed Jan 8 12:56:21 2014 +0100

    Create client certificate for normal nodes
    
    The vcluster QA revealed a bug in the SSL certificate
    handling code, where certificates were only created
    when the node is a master-candidate. However, every node
    should have a certificate, but only the digests of the
    certificates of the master candidates are added to the
    certificate map.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Klaus Aehlig <aehlig@google.com>

commit 575b31bfc4bd64938c8ac0890cdff9a2c65f24d7
Author: Helga Velroyen <helgav@google.com>
Date:   Wed Dec 18 14:31:45 2013 +0100

    Update design doc to match implementation
    
    This patch contains some minor changes in the design doc
    to make sure the details match the implementation.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit 898fd9e1350a246f59c08b9f6f74dda812c1b08b
Author: Helga Velroyen <helgav@google.com>
Date:   Wed Dec 18 14:20:11 2013 +0100

    Update UPGRADE nodes
    
    Adds to the upgrade nodes that a renewal of the node
    certificates is necessary.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit 224c4204b83c123048e8f1508d0dddfdd5344f0e
Author: Helga Velroyen <helgav@google.com>
Date:   Thu Dec 12 10:28:36 2013 +0100

    Update NEWS wrt to client RPC certificates
    
    This updates the NEWS file regarding the changes in
    RPC communication.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>


commit a6c43c0278ba4081c3775d687a31400495e7ba27
Author: Helga Velroyen <helgav@google.com>
Date:   Tue Dec 17 14:15:23 2013 +0100

    Verify client certificates
    
    This patch adds a step to 'gnt-cluster verify' to verify
    the existence and validity of the nodes' client
    certificates. Since this is a crucial point of the
    security concept, the verification is very detailed with
    expressive error messages and well tested by unit tests.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit b3cc16469014cd3c9e5e6b3dfb63f412234ecc1a
Author: Helga Velroyen <helgav@google.com>
Date:   Wed Dec 11 13:15:37 2013 +0100

    Verify incoming RPCs against candidate map
    
    From this patch on, incoming RPC calls are checked against
    the map of valid master candidate certificates. If no map
    is present, the cluster is assumed to be in
    bootstrap/upgrade mode and compares the incoming call
    against the server certificate. This is necessary, because
    neither at cluster initialization nor at upgrades from
    pre-2.11 versions a candidate map is established yet.
    
    After an upgrade, the cluster RPC communication continues
    to use the server certificate until the client certificates
    are created and the candidate map is populated using
    'gnt-cluster renew-crypto --new-node-certificates'.
    
    Note that for updating the master's certificate, a trick
    was necessary. The new certificate is first created under
    a temporary name, then it's digest is updated and
    distributed using the old certificate, because otherwise
    distribution will fail since the nodes don't know the
    new digest yet. Then the certificate is moved to its
    proper location.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit 28756f80f3dee3677ea312b51868a9ab8d73e703
Author: Helga Velroyen <helgav@google.com>
Date:   Wed Dec 11 16:55:39 2013 +0100

    Handle promoting/demoting nodes wrt to client certificates
    
    This patch makes Ganeti correctly handle the client
    certificates when nodes get promoted to master candidates
    or demoted to normal nodes.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit d722af8b6febd0d0ec1dd94c04041754f042cb0f
Author: Helga Velroyen <helgav@google.com>
Date:   Wed Dec 11 11:05:54 2013 +0100

    Extend RPC call to create SSL certificates
    
    So far the RPC call 'node_crypto_tokens' did only retrieve
    the certificate digest of an existing certificate. This
    call is now enhanced to also create a new certificate and
    return the respective digest. This will be used in various
    operations, among those cluster init and renew-crypto.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit 60cc531d0e6f5c7484df6beb44b5afaacc77c529
Author: Helga Velroyen <helgav@google.com>
Date:   Wed Dec 11 11:07:32 2013 +0100

    Create client SSL certificates on cluster init
    
    This patch makes Ganeti create a client SSL certificate for
    the master node on cluster initialization. Note that some of
    the code in this patch is later moved into an LU to serve
    requirements for crypto renewal and updates, but for this
    point in the patch series it makes sense to add it here.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit 1059337d25a1a43021d1b281f39c9a4f1b0ba650
Author: Helga Velroyen <helgav@google.com>
Date:   Tue Dec 10 11:21:20 2013 +0100

    Store candidate certificates in ssconf
    
    This patch enables Ganeti to store the candidate
    certificate map in ssconf. A utility function to
    read it is provided as well.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit 840ad2abb9cccf7876a2145b5328dbde2b908643
Author: Helga Velroyen <helgav@google.com>
Date:   Fri Dec 6 15:30:03 2013 +0100

    Handle client certificates on node add/remove
    
    This patch adds the certificate of a newly added or
    readded master candidate node to the map of master candidate
    certificates. It removes a master candidate node's certificate
    digest from the candidate certificate map if the node is
    removed from the cluster.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit 5b6f9e3599dc60595c0cfd917ab78840c3f32d71
Author: Helga Velroyen <helgav@google.com>
Date:   Fri Dec 6 11:39:18 2013 +0100

    Add certificate for master node
    
    On cluster initialization, the master node's
    SSL certificate digest is added to the list of master
    candidate certificates.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit 3bcf2140cc25507a8c4a0ca1fca860c0c6a69569
Author: Helga Velroyen <helgav@google.com>
Date:   Thu Dec 5 16:43:04 2013 +0100

    Add candiate certificate map to configuration
    
    At the end of this patch series, incoming RPC calls are
    legitimized against a map of master candidate nodes'
    SSL certificate digests. This patch adds the map itself
    to the cluster's configuration.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit b544a3c2c2975636f469b945242bc851bb20ed18
Author: Helga Velroyen <helgav@google.com>
Date:   Thu Dec 5 16:05:09 2013 +0100

    Retrieve a node's certificate digest
    
    In various cluster operations, the master node needs to
    retrieve the digest of a node's SSL certificate. For this
    purpose, we add an RPC call to retrieve the digest. The
    function is designed in a general way to make it possible
    to retrieve other (public) cryptographic tokens of a node
    in the future as well (for example an SSH key).
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

commit 3338a9ce2670331e409c4a076866c80fe9ac4356
Author: Helga Velroyen <helgav@google.com>
Date:   Thu Dec 5 14:13:51 2013 +0100

    Utility functions to manipulate the candidate map
    
    This patch adds a couple of utility functions to manipulate
    the map of master candidate SSL certificate digests.
    
    Signed-off-by: Helga Velroyen <helgav@google.com>
    Reviewed-by: Hrvoje Ribicic <riba@google.com>

Originally added on 2014-02-28 09:34:51 +0000 UTC.

[Ganeti-Issues-Migrator]

Fixed in 2.12

Originally added on 2015-03-03 15:26:44 +0000 UTC.

Changed State: Fixed

[Ganeti-Issues-Migrator]

-- Empty comment --

Originally added on 2015-03-25 16:06:08 +0000 UTC.

Changed State: Released