v1.9 backports 2020-11-17

.github/workflows/bpf-checks.yaml

@@ -16,7 +16,7 @@ jobs:
         with:
           fetch-depth: 0
       - name: Run checkpatch.pl
-        uses: docker://cilium/cilium-checkpatch:5b099019bf0db775b33b3f32cd5ecea55dd15f21
+        uses: docker://cilium/cilium-checkpatch:f5443bb156d5ac4110ede4b501d6a2cf356d3202
   coccicheck:
     name: coccicheck
     runs-on: ubuntu-latest

.github/workflows/go-check.yaml

@@ -87,7 +87,7 @@ jobs:
           sudo chmod +x bin/protoc && sudo cp bin/protoc /usr/local/bin
       - name: Set GOPATH
         run: |
-          echo "##[set-env name=GOPATH;]$GITHUB_WORKSPACE"
+          echo "GOPATH=$GITHUB_WORKSPACE" >> $GITHUB_ENV
         shell: bash
       - name: Check k8s generated files
         run: |

Documentation/contributing/release/backports.rst

@@ -66,8 +66,10 @@ One-time setup
       $ git config --global user.email johndoe@example.com
 
 #. Make sure you have a GitHub developer access token with the ``public_repos``
-   scope available. For details, see `contrib/backporting/README.md
-   <https://github.com/cilium/cilium/blob/master/contrib/backporting/README.md>`_
+   scope available. You can do this directly from
+   https://github.com/settings/tokens or by opening GitHub and then navigating
+   to: User Profile -> Settings -> Developer Settings -> Personal access token
+   -> Generate new token.
 
 #. This guide makes use of several tools to automate the backporting process.
    The basics require ``bash`` and ``git``, but to automate interactions with
@@ -213,7 +215,7 @@ Via GitHub web interface
    backport PRs later.
 
 #. Mark all PRs you backported with the backport pending label
-   ``backport-pending/X.Y`` and clear the ``needs-backport/vX.Y`` label. This
+   ``backport-pending/X.Y`` and clear the ``needs-backport/X.Y`` label. This
    can be done with the command printed out at the bottom of the output from
    the ``start-backport`` script above (``GITHUB_TOKEN`` needs to be set for
    this to work).

Documentation/hubble.rst

@@ -18,11 +18,11 @@ achieve all of this at large scale.
 Hubble's server component is embedded into the Cilium agent in order to achieve
 high performance with low-overhead. The gRPC services offered by Hubble server
 may be consumed locally via a Unix domain socket or, more typically, through
-Hubble Relay. Hubble-relay is a standalone component which is aware of all
-running Hubble instances and offers full cluster visibility by connecting to
-their respective gRPC APIs. This capability is usually referred to as
-multi-node. Hubble Relay's main goal is to offer a rich API that can be safely
-exposed and consumed by the Hubble UI and CLI.
+Hubble Relay. Hubble Relay is a standalone component which is aware of all
+Hubble instances and offers full cluster visibility by connecting to their
+respective gRPC APIs. This capability is usually referred to as multi-node.
+Hubble Relay's main goal is to offer a rich API that can be safely exposed and
+consumed by the Hubble UI and CLI.
 
 .. note:: This guide does not cover Hubble in standalone mode, which is
           deprecated with the release of Cilium v1.8.
@@ -44,31 +44,33 @@ local Unix domain socket.
 The Observer service
 ^^^^^^^^^^^^^^^^^^^^
 
-The Observer service is the principal service. It makes two methods available:
-``GetFlows`` and ``ServerStatus``.  While the ``ServerStatus`` method is pretty
-straightforward (it provides metrics related to the running server), the
-``GetFlows`` one is far more sophisticated and the more important one.
+The Observer service is the principal service. It provides three RPC endpoints:
+``GetFlows``, ``GetNodes`` and ``ServerStatus``.  While ``ServerStatus`` and
+``GetNodes`` endpoints are pretty straightforward (they provides metrics and
+other information related to the running instance(s)), ``GetFlows`` is far more
+sophisticated and the more important one.
 
-Using ``GetFlows``, callers can get a stream of payloads. Request parameters
-allow callers to specify filters in the form of blacklists and whitelists to
-allow for fine-grained filtering of data.
+Using ``GetFlows``, callers get a stream of payloads. Request parameters allow
+callers to specify filters in the form of blacklists and whitelists to allow
+for fine-grained filtering of data.
 
 In order to answer ``GetFlows`` requests, Hubble stores monitoring events from
-Cilium's event monitor into a ring buffer structure.  Monitoring events are
-obtained by registering a new listener to Cilium's monitor.  The ring buffer is
-capable of storing a configurable amount of events in memory. Events are
-continuously consumed, overriding older ones once the ring buffer is full.
+Cilium's event monitor into a user-space ring buffer structure. Monitoring
+events are obtained by registering a new listener on Cilium monitor. The
+ring buffer is capable of storing a configurable amount of events in memory.
+Events are continuously consumed, overriding older ones once the ring buffer is
+full.
 
 .. image:: ./images/hubble_getflows.png
 
 For efficiency, the internal buffer length is a bit mask of ones + 1. The most
 significant bit of this bit mask is the same position of the most significant
 bit position of 'n'. In other terms, the internal buffer size is always a power
-of 2. As the ring buffer is a hot code path, it has been designed to not employ
-any locking mechanisms and uses atomic operations instead. While this approach
-has performance benefits, it also has the downsides of being a complex
-component and that reading the very last event written to the buffer is not
-possible as it cannot be guaranteed that it has been fully written.
+of 2 with 1 slot reserved for the writer. In effect, from a user perspective,
+the ring buffer capacity is one less than a power of 2. As the ring buffer is a
+hot code path, it has been designed to not employ any locking mechanisms and
+uses atomic operations instead. While this approach has performance benefits,
+it also has the downsides of being a complex component.
 
 Due to its complex nature, the ring buffer is typically accessed via a ring
 reader that abstracts the complexity of this data structure for reading. The
@@ -81,8 +83,8 @@ The Peer service
 
 The Peer service sends information about Hubble peers in the cluster in a
 stream. When the ``Notify`` method is called, it reports information about all
-the peers in the cluster and subsequently sends information about peers that are
-updated, added or removed from the cluster. Thus , it allows the caller to
+the peers in the cluster and subsequently sends information about peers that
+are updated, added or removed from the cluster. Thus, it allows the caller to
 keep track of all Hubble instances and query their respective gRPC services.
 
 This service is typically only exposed on a local Unix domain socket and is
@@ -98,11 +100,22 @@ Cilium's datapath node handler interface.
 Hubble Relay
 ------------
 
-.. note:: At the time of this writing, Hubble Relay component is still
-          work in progress and may undergo major changes. For this reason,
-          internal documentation about Hubble Relay is limited.
-
-Hubble Relay is a component that was introduced in the context of multi-node
-support. It leverages the Peer service to obtain information about Hubble
-instances and consume their gRPC API in order to provide a more rich API that
-covers events from across the entire cluster.
+Hubble Relay is the Hubble component that brings multi-node support. It
+leverages the Peer service to obtain information about Hubble instances and
+consume their gRPC API in order to provide a more rich API that covers events
+from across the entire cluster (or even multiple clusters in a ClusterMesh
+scenario).
+
+Hubble Relay was first introduced as a technology preview with the release of
+Cilium v1.8. It is declared stable with the release of Cilium v1.9.
+
+Hubble Relay implements the Observer service for multi-node. To that end, it
+maintains a persistent connection with every Hubble peer in a cluster with a
+peer manager. This component provides callers with the list of peers. Callers
+may report when a peer is unreachable, in which case the peer manager will
+attempt to reconnect.
+
+As Hubble Relay connects to every node in a cluster, the Hubble server
+instances must make their API available (by default on port 4244). By default,
+Hubble server endpoints are secured using mutual TLS (mTLS) when exposed on a
+TCP port in order to limit access to Hubble Relay only.

Documentation/operations/upgrade.rst

@@ -357,6 +357,14 @@ Annotations:
   install --set ipv4.enabled=true``. The following section calls out specific
   values where the prefix cannot be simply dropped followed by a full
   table of old and new Helm values.
+* On Linux kernel v5.10 and above, running the agent with BPF kube-proxy replacement
+  under direct routing operation as well as BPF-based masquerading will bypass
+  subsystems like netfilter/iptables in the host namespace in order to significantly
+  improve throughput and latency for the BPF datapath given routing is not performed
+  in the host stack but directly in BPF instead. To opt-out from this behavior,
+  the Helm option ``bpf.hostRouting=true`` can be used. If the underlying kernel
+  does not implement the needed BPF features, then the agent will fallback and rely
+  on host routing automatically.
 
 .. _1.9_helm_options:
 

api/v1/openapi.yaml

@@ -1630,6 +1630,9 @@ definitions:
       host-routing:
         description: Status of host routing
         "$ref": "#/definitions/HostRouting"
+      clock-source:
+        description: Status of clock source
+        "$ref": "#/definitions/ClockSource"
       stale:
         description: List of stale information in the status
         type: object
@@ -2500,6 +2503,22 @@ definitions:
       snat-exclusion-cidr:
         description: Any packet sent to IP addr belonging to CIDR will not be SNAT'd
         type: string
+  ClockSource:
+    description: |-
+      Status of BPF clock source
+
+      +k8s:deepcopy-gen=true
+    type: object
+    properties:
+      mode:
+        description: Datapath clock source
+        type: string
+        enum:
+        - ktime
+        - jiffies
+      hertz:
+        description: Kernel Hz
+        type: integer
   HostRouting:
     description: |-
       Status of host routing