Skip to content

Commit

Permalink
docs: remove 1.7 upgrade guide and add upgradeCompatibility for 1.9
Browse files Browse the repository at this point in the history 
Signed-off-by: André Martins <andre@cilium.io>
  • Loading branch information
@aanm @twpayne
aanm authored and twpayne committed May 26, 2021
1 parent 91387ae commit 076f475
Show file tree
Hide file tree
Showing 2 changed files with 1 addition and 245 deletions.
2 changes: 0 additions & 2 deletions Documentation/gettingstarted/kubeproxy-free.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1111,8 +1111,6 @@ This section elaborates on the various ``kubeProxyReplacement`` options:
In Cilium's Helm chart, the default mode is ``kubeProxyReplacement=probe`` for
new deployments.

For existing Cilium deployments in version v1.6 or prior, please consult the :ref:`1.7_upgrade_notes`.

The current Cilium kube-proxy replacement mode can also be introspected through the
``cilium status`` CLI command:

Expand Down
244 changes: 1 addition & 243 deletions Documentation/operations/upgrade.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ version which was installed in this cluster. Valid options are:

* ``1.7`` if the initial install was Cilium 1.7.x or earlier.
* ``1.8`` if the initial install was Cilium 1.8.x.
* ``1.9`` if the initial install was Cilium 1.9.x.

.. tabs::
.. group-tab:: kubectl
Expand Down Expand Up @@ -292,10 +293,6 @@ latest ``1.1.y`` release before subsequently upgrading to ``1.2.z``.
+-----------------------+-----------------------+-------------------------+---------------------------+
| Current version | Target version | L3 impact | L7 impact |
+=======================+=======================+=========================+===========================+
| ``1.7.0`` | ``1.7.1`` | Minimal to None | Clients must reconnect[1] |
+-----------------------+-----------------------+-------------------------+---------------------------+
| ``>=1.7.1`` | ``1.7.y`` | Minimal to None | Clients must reconnect[1] |
+-----------------------+-----------------------+-------------------------+---------------------------+
| ``>=1.7.1`` | ``1.8.y`` | Minimal to None | Clients must reconnect[1] |
+-----------------------+-----------------------+-------------------------+---------------------------+
| ``1.8.x`` | ``1.9.y`` | Minimal to None | Clients must reconnect[1] |
Expand Down Expand Up @@ -1260,245 +1257,6 @@ Removed resource fields
``CiliumEndpoint.Status.Spec``, and ``EndpointIdentity.LabelsSHA256``,
deprecated in 1.4, have been removed.

.. _1.7_upgrade_notes:

1.7 Upgrade Notes
-----------------

.. _1.7_required_changes:

IMPORTANT: Changes required before upgrading to 1.7.x
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. warning::

Do not upgrade to 1.7.x before reading the following section and completing
the required steps.

In particular, if you are using network policy and upgrading from 1.6.x or earlier
to 1.7.x or later, you MUST read the 1.7 :ref:`configmap_remote_node_identity`
section about the
``enable-remote-node-identity`` flag to avoid potential disruption
to connectivity between host networking pods and Cilium-managed pods.

* Cilium has bumped the minimal Kubernetes version supported to v1.11.0.

* The ``kubernetes.io/cluster-service`` label has been removed from the Cilium
`DaemonSet` selector. Existing users must either choose to keep this label in
`DaemonSet` specification to safely upgrade or re-create the Cilium `DaemonSet`
without the deprecated label. It is advisable to keep the label when doing
an upgrade from ``v1.6.x`` to ``v1.7.x`` in the event of having to do a
downgrade. The removal of this label should be done after a successful
upgrade.

The Helm option ``agent.keepDeprecatedLabels=true`` will keep the
``kubernetes.io/cluster-service`` label in the new `DaemonSet`:

.. tabs::
.. group-tab:: kubectl

.. parsed-literal::
helm template cilium \
--namespace=kube-system \
...
--set agent.keepDeprecatedLabels=true \
...
> cilium.yaml
kubectl apply -f cilium.yaml
.. group-tab:: Helm

.. parsed-literal::
helm upgrade cilium --namespace=kube-system \
--set agent.keepDeprecatedLabels=true
Trying to upgrade Cilium without this option might result in the following
error: ``The DaemonSet "cilium" is invalid: spec.selector: Invalid value: ...: field is immutable``


* If ``kvstore`` is setup with ``etcd`` **and** TLS is enabled, the field name
``ca-file`` will have its usage deprecated and will be removed in Cilium v1.8.0.
The new field name, ``trusted-ca-file``, can be used since Cilium v1.1.0.

*Required action:*

This field name should be changed from ``ca-file`` to ``trusted-ca-file``.

Example of an old etcd configuration, with the ``ca-file`` field name:

.. code:: yaml
---
endpoints:
- https://192.168.0.1:2379
- https://192.168.0.2:2379
ca-file: '/var/lib/cilium/etcd-ca.pem'
# In case you want client to server authentication
key-file: '/var/lib/cilium/etcd-client.key'
cert-file: '/var/lib/cilium/etcd-client.crt'
Example of new etcd configuration, with the ``trusted-ca-file`` field name:

.. code:: yaml
---
endpoints:
- https://192.168.0.1:2379
- https://192.168.0.2:2379
trusted-ca-file: '/var/lib/cilium/etcd-ca.pem'
# In case you want client to server authentication
key-file: '/var/lib/cilium/etcd-client.key'
cert-file: '/var/lib/cilium/etcd-client.crt'
* Due to the removal of external libraries to connect to container runtimes,
Cilium no longer supports the option ``flannel-manage-existing-containers``.
Cilium will still support integration with Flannel for new containers
provisioned but not for containers already running in Flannel. The options
``container-runtime`` and ``container-runtime-endpoint`` will not have any
effect and the flag removal is scheduled for v1.8.0

* The default ``--tofqdns-min-ttl`` value has been reduced to 1 hour. Specific
IPs in DNS entries are no longer expired when in-use by existing connections
that are allowed by policy. Prior deployments that used the default value may
now experience denied new connections if endpoints reuse DNS data more than 1
hour after the initial lookup without making new lookups. Long lived
connections that previously outlived DNS entries are now better supported,
and will not be disconnected when the corresponding DNS entry expires.

.. _configmap_remote_node_identity:

New ConfigMap Options
~~~~~~~~~~~~~~~~~~~~~

* ``enable-remote-node-identity`` has been added to enable a new identity
for remote cluster nodes and to associate all IPs of a node with that new
identity. This allows for network policies that distinguish between
connections from host networking pods or other processes on the local
Kubernetes worker node from those on remote worker nodes.

After enabling this option, all communication to and from non-local
Kubernetes nodes must be whitelisted with a ``toEntity`` or ``fromEntity``
rule listing the entity ``remote-node``. The existing entity ``cluster``
continues to work and now includes the entity ``remote-node``. Existing
policy rules whitelisting ``host`` will only affect the local node going
forward. Existing CIDR-based rules to whitelist node IPs other than the
Cilium internal IP (IP assigned to the ``cilium_host`` interface), will no
longer take effect.

This is important because Kubernetes Network Policy dictates that network
connectivity from the local host must always be allowed, even for pods that
have a default deny rule for ingress connectivity. This is so that
network liveness and readiness probes from kubelet will not be dropped by
network policy. Prior to 1.7.x, Cilium achieved this by always allowing
ingress host network connectivity from any host in the cluster. With 1.7
and ``enable-remote-node-identity=true``, Cilium will only automatically
allow connectivity from the local node, thereby providing a better default
security posture.

The option is enabled by default for new deployments when generated via
Helm, in order to gain the benefits of improved security. The Helm option
is ``--set global.remoteNodeIdentity``. This option can be disabled in
order to maintain full compatibility with Cilium 1.6.x policy enforcement.
**Be aware** that upgrading a cluster to 1.7.x by using Helm to generate a
new Cilium config that leaves ``enable-remote-node-identity`` set as the
default value of ``true`` **can break network connectivity.**

The reason for this is that
with Cilium 1.6.x, the source identity of ANY connection from a host-networking pod or from
other processes on a Kubernetes worker node would be the ``host`` identity. Thus, a
Cilium 1.6.x or earlier environment with network policy enforced may be implicitly
relying on the ``allow everything from host identity`` behavior to
whitelist traffic from host networking to other Cilium-managed pods.
With the shift to 1.7.x, if ``enable-remote-node-identity=true``
these connections will be denied by policy if they are coming from
a host-networking pod or process on another Kubernetes worker node, since the source
will be given the ``remote-node`` identity (which is not automatically
allowed) rather than the ``host`` identity (which is automatically allowed).

An indicator that this is happening would be drops visible in Hubble or
Cilium monitor with a source identity equal to 6 (the numeric value for the
new ``remote-node`` identity. For example:

::

xx drop (Policy denied) flow 0x6d7b6dd0 to endpoint 1657, identity 6->51566: 172.16.9.243:47278 -> 172.16.8.21:9093 tcp SYN

There are two ways to address this. One can set
``enable-remote-node-identity=false`` in the `ConfigMap` to retain the
Cilium 1.6.x behavior. However, this is not ideal, as it means there is no
way to prevent communication between host-networking pods and
Cilium-managed pods, since all such connectivity is allowed automatically
because it is from the ``host`` identity.

The other option is to keep ``enable-remote-node-identity=true`` and
create policy rules that explicitly whitelist connections between
the ``remote-host`` identity and pods that should be reachable from host-networking pods
or other processes that may be running on a remote Kubernetes worker node. An example of
such a rule is:


::

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "allow-from-remote-nodes"
spec:
endpointSelector:
matchLabels:
app: myapp
ingress:
- fromEntities:
- remote-node

See :ref:`policy-remote-node` for more examples of remote-node policies.


* ``enable-well-known-identities`` has been added to control the
initialization of the well-known identities. Well-known identities have
initially been added to support the managed etcd concept to allow the etcd
operator to bootstrap etcd while Cilium still waited on etcd to become
available. Cilium now uses CRDs by default which limits the use of
well-known identities to the managed etcd mode. With the addition of this
option, well-known identities are disabled by default in all new deployment
and only enabled if the Helm option ``global.etcd.managed=true`` is set. Consider
disabling this option if you are not using the etcd operator respectively
managed etcd mode to reduce the number of policy identities whitelisted for
each endpoint.

* ``kube-proxy-replacement`` has been added to control which features should
be enabled for the kube-proxy eBPF replacement. The option is set to
``probe`` by default for new deployments when generated via Helm. This
makes cilium-agent to probe for each feature support in a kernel, and
to enable only supported features. When the option is not set via Helm,
cilium-agent defaults to ``partial``. This makes ``cilium-agent`` to
enable only those features which user has explicitly enabled in their
ConfigMap. See :ref:`kubeproxy-free` for more option values.

For users who previously were running with ``global.nodePort.enabled=true`` it is
recommended to set the option to ``strict`` before upgrading.

* ``enable-auto-protect-node-port-range`` has been added to enable
auto-appending of a NodePort port range to
``net.ipv4.ip_local_reserved_ports`` if it overlaps with an ephemeral port
range from ``net.ipv4.ip_local_port_range``. The option is enabled by
default. See :ref:`kubeproxy-free` for the explanation why the overlap can
be harmful.

Removed options
~~~~~~~~~~~~~~~~~~

* ``lb``: The ``--lb`` feature has been removed. If you need load-balancing on
a particular device, consider using :ref:`kubeproxy-free`.

* ``docker`` and ``e``: This flags has been removed as Cilium no longer requires
container runtime integrations to manage containers' networks.

* All code associated with ``monitor v1.0`` socket handling has been removed.

Advanced
========

Expand Down

0 comments on commit 076f475

Please sign in to comment.