Adapt docs for v 3.4.5.1 (#141)

* Add release notes.
 * Update docs to be fleetspeak-first.
 * Add information on the MSI intaller.

conf.py

@@ -171,8 +171,8 @@ class ProcessLink(transforms.Transform):
     default_priority = 1000
 
     text_replacements = {
-        "__GRR_VERSION__": "3.4.3.1",
-        "__GRR_DEB_VERSION__": "3.4.3-1"
+        "__GRR_VERSION__": "3.4.5.1",
+        "__GRR_DEB_VERSION__": "3.4.5-1"
     }
 
     def find_replace(self, node):

deploying-grr-clients/on-windows.md

@@ -1,5 +1,20 @@
 # Installing GRR clients on Windows
 
+## MSI installer
+
+Since version 3.4.5.1, GRR provides a new MSI based installer for the client on
+Windows.
+
+The client can be installed either by double-clicking the installer or by using
+the
+[`msiexec`](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/msiexec)
+command.
+
+The client can be removed via the "Apps and features" setting in the control
+panel or by using `msiexec`.
+
+## Legacy self-extracting .exe installer
+
 For Windows you will see a 64 bit installer. Run the installer as admin (it
 should load the UAC prompt if you are not admin). It should run silently and
 install the client to `c:\windows\system32\grr\%version%\`. It will also
@@ -86,7 +101,7 @@ Anonymous share.
 The best way to verify whether the whole installation process has worked
 is to search for the client in the GUI.
 
-## Uninstalling on Windows
+### Uninstalling
 On Windows the client does not have a standard uninstaller. It is
 designed to have minimal impact on the system and leave limited traces
 of itself such that it can be hidden reasonably easily. Thus it was

developing-grr/setting-up-dev-env.md

@@ -133,13 +133,15 @@ grr_config_updater initialize
 
 You can use one of the commands below to run specific GRR server components.
 
+Fleetspeak server: `fleetspeak_server`
+
 Admin UI: `grr_admin_ui`
 
 Frontend: `grr_frontend`
 
 Worker: `grr_worker`
 
-You can also run a local GRR client by executing `grr_client` command.
+You can also run a local GRR client by executing `fleetspeak_client` command.
 
 Adding `--verbose` flag to any of these commands would force GRR components to output debug information to stderr.
 

installing-grr-server/from-release-deb.md

@@ -37,6 +37,12 @@ community edition of MySQL from Ubuntu repositories:
     mysql> CREATE DATABASE grr;
 
     mysql> GRANT ALL ON grr.* TO 'grr'@'localhost';
+
+    mysql> CREATE USER 'fleetspeak'@'localhost' IDENTIFIED BY 'password';
+
+    mysql> CREATE DATABASE fleetspeak;
+
+    mysql> GRANT ALL ON fleetspeak.* TO 'fleetspeak'@'localhost';
     ```
     Please note: GRR is senstive to the MySQL's `max_allowed_packet` setting.
     Make sure it's not lower than 20971520. Creation of a new user is optional
@@ -56,7 +62,7 @@ community edition of MySQL from Ubuntu repositories:
     ```
 
     The installer will prompt for a few pieces of information to get things set up.
-    After successful installation, the `grr-server` service should be running:
+    After successful installation, the `grr-server` and `fleetspeak-server` services should be running:
 
     ```bash
     root@grruser-bionic:/home/grruser# systemctl status grr-server
@@ -73,6 +79,18 @@ community edition of MySQL from Ubuntu repositories:
 
     Nov 22 10:16:39 grruser-bionic systemd[1]: Starting GRR Service...
     Nov 22 10:16:39 grruser-bionic systemd[1]: Started GRR Service.
+
+		root@grruser-bionic:/home/grruser# systemctl status fleetspeak-server
+		● fleetspeak-server.service - Fleetspeak Server Service
+			 Loaded: loaded (/lib/systemd/system/fleetspeak-server.service; disabled; vendor preset: enabled)
+			 Active: active (running) since Thu 2021-08-12 18:35:13 UTC; 4s ago
+				 Docs: https://github.com/google/fleetspeak
+		 Main PID: 3555 (fleetspeak-serv)
+				Tasks: 7 (limit: 4666)
+			 CGroup: /system.slice/fleetspeak-server.service
+							 └─3555 /usr/bin/fleetspeak-server --services_config /etc/fleetspeak-server/server.services.config --components_config /etc/fleetspeak-server/server.components.config
+
+		Aug 12 18:35:13 grruser-bionic systemd[1]: Started Fleetspeak Server Service.
     ```
 
     In addition, administrative commands for GRR, e.g `grr_console` and

installing-grr-server/from-released-pip.md

@@ -62,6 +62,8 @@ commands such as as `grr_server` and `grr_config_updater` will be added to
 the virtualenv. You can then launch the server components as follows:
 
 ```bash
+# To run fleetspeak server.
+grr_server --component fleetspeak_server --verbose
 # To run AdminUI.
 grr_server --component admin_ui --verbose
 # To run the worker.
@@ -70,6 +72,6 @@ grr_server --component worker --verbose
 grr_server --component frontend --verbose
 ```
 
-Note that GRR requires Python 2.7+, so for platforms with older default Python
+Note that GRR requires Python 3.6+, so for platforms with older default Python
 versions (e.g Centos 6), you need to build a newer version of Python from source
 and use that for creating the virtualenv.

installing-grr-server/overview.md

@@ -22,9 +22,9 @@ GRR components are:
 
 #### Client
 
-The GRR client **is not a server component** but it comes bundled with GRR server. GRR client is deployed on corporate assets using the usual mechanism for software distribution and updates (e.g. SMS, apt). The client communicates with the front-end server using a HTTP POST requests. GRR client sends and receives GRR messages from the server. All communication with the front end servers is encrypted.
+The GRR client **is not a server component** but it comes bundled with GRR server. GRR client is deployed on corporate assets using the usual mechanism for software distribution and updates (e.g. SMS, apt). The client consists of 2 processes: the fleetspeak client responsible for implementig the communication protocol based on a streaming HTTPS connection. The GRR client process sends and receives GRR messages from the server and implements the business logic.
 
-#### Datastore
+#### GRR Datastore
 The data store acts both as a central storage component for data, and as a communication mechanism for all GRR server components.
 
 ***Note on the AFF4 datastore deprecation***
@@ -33,12 +33,24 @@ The data store acts both as a central storage component for data, and as a commu
 
 *Use of AFF4-based deployments is now discouraged. REL_DB is expected to be much more stable and performant. Please see [these docs](../maintaining-and-tuning/grr-datastore.md) if you're upgrading an older GRR version and would like to try out the new datastore.*
 
-#### Front End Servers
-The front end servers' main task is to decrypt POST requests from the client, un-bundle the contained messages and queue these on the data store. The front end also fetches any messages queued for the client and sends them to the client.
+#### Fleetspeak server
 
-#### Worker
+This component terminates the streaming HTTPS connections from the fleetspeak clients and implements the communication protocol. It receives messages from clients and delivers queued messages to the clients.
+
+#### Fleetspeak admin server
+
+This component provides an interface for the GRR server to send messages to clients.
+
+#### Fleetspeak database
+
+The fleetspeak system uses a MySQL database for queing messages for clients.
+
+#### GRR Front End Servers
+The front end servers' main task is to receive GRR messages from the fleetspeak server, un-bundle the contained messages and queue these on the data store. The front end also fetches any messages queued for the client and sends them to the client.
+
+#### GRR Worker
 In order to remain scalable, the front end does not do any processing of data, preferring to offload processing to special worker components. The number of workers can be tuned in response to increased workload. Workers typically check queues in the data stores for responses from the client, process those and re-queue new requests for the clients (See Flows and Queues).
 
-#### Web UI
+#### GRR Web UI
 GRR Web UI is the central application which enables the incident responder or forensic analyst to interact with the system. It allows for analysis tasks to be queued for the clients, and results of previous stored analysis to be examined. It also acts as an API endpoint: GRR API can be used for automation and integration with other systems.
 

maintaining-and-tuning/key-management/rotating.md

@@ -2,7 +2,7 @@
 
 ## Rotating Client Keys ##
 
-The keys used by the clients to talk to the server do not have to be / cannot be rotated since the client id depends directly on the private key. Just removing the private key results in the client generating a new one but the client will also get a new client id.
+The keys used by the fleetspeak client to talk to the server do not have to be / cannot be rotated since the client id depends directly on the private key. Just removing the fleetspeak state file results in the client generating a new key pair but the client will also get a new client id.
 
 Client keys used for signing executables and drivers can be changed in the config as described in the [documentation about keys](which-keys-and-how.md) but
 

maintaining-and-tuning/key-management/which-keys-and-how.md

@@ -2,23 +2,19 @@
 
 ## Communication Security.
 
-GRR communication happens using signed and encrypted protobuf messages.
-We use 2048 bit RSA keys to protect symmetric AES128 encryption. The
-security of the system does not rely on SSL transport for communication
-security. This enables easy replacement of the comms protocol with
-non-http mechanisms such as UDP packets.
-
-The communications use a CA and server public key pair generated on
-server install. The CA public key is deployed to the client so that it
+GRR uses fleetspeak for communication, which in turn uses TLS.
+
+The fleetspeak server uses a CA and server public key pair generated on
+server install. The CA certificate is deployed to the client so that it
 can ensure it is communicating with the correct server. If these keys
 are not kept secure, anyone with MITM capability can intercept
 communications and take control of your clients. Additionally, if you
 lose these keys, you lose the ability to communicate with your clients.
 
-**Note**: The fact that the client uses a CA key to verify the server
+**Note**: The fact that the client uses a CA certificate to verify the server
 ensures only that the client doesn't accidentally connect to the wrong
 server and protects for example from privilege escalation attacks using
-a malicious GRR server. This feature does not stop clients that do not
+a malicious fleetspeak server. This feature does not stop clients that do not
 have the CA certificate from connecting to your server - it's for
 example possible to use a modified client that just doesn't do this
 check. Such clients would the be able to see for example IOCs that you

maintaining-and-tuning/monitoring.md

@@ -15,20 +15,19 @@ and [Grafana](https://grafana.com/) can plot the parsed data.
 This example will walk you through a basic Prometheus setup.
 For this example, the GRR Frontend, Worker, and Admin UI will be
 launched on your local machine. You can also choose to monitor
-Fleetspeak servers, if you have a GRR + Fleetspeak setup; otherwise
-feel free to skip the relevant steps, which are marked as **FS**.
+Fleetspeak servers; otherwise feel free to skip the relevant steps, which are
+marked as **FS**.
 
 1. Install GRR, for example from
 [pip](../installing-grr-server/from-released-pip.html).
 
-1. **FS:** Follow
-[the instructions](../fleetspeak.html) to install Fleetspeak
-configured with GRR.
-
 1. Run the GRR components locally. Execute each of the
 three commands in a separate terminal:
 
     ```bash
+    # Fleetspeak monitoring is configured in a separate config file.
+    grr_server --component fleetspeak_server
+
     grr_server --component admin_ui -p Monitoring.http_port=44451
 
     grr_server --component frontend -p Monitoring.http_port=44452
@@ -41,10 +40,13 @@ three commands in a separate terminal:
     Prometheus requires to know which type of component listens on which
     ports. If you use `Monitoring.http_port_max`, make sure that only one
     type of GRR components (e.g. only workers) listen on a given
-    range of ports. 
+    range of ports.
+
+1. **FS:** Edit the file `server.components.config`. It is located in
+`/etc/fleetspeak-server` on a setup installed from a DEB and in the virtualenv,
+in `fleetspeak-server-bin/etc/fleetspeak-server` when running from a
+virtualenv.  Add the following to the end of the file:
 
-1. **FS:** Go to `~/.config/fleetspeak-server/components.textproto` and
-add the following to the end of the file:
     ```bash
     stats_config: <
       address: "localhost:54451"

release-notes.md

@@ -12,6 +12,16 @@ new version.
 
 ## Server
 
+### 3.4.5.1 (August 19 2021)
+
+Mid-quarter release for Q3 2021.
+
+ - Sandboxing in the GRR client: TSK, libfsntfs and YARA libraries are now run in a separate, unprivileged process. This uses an unprivileged user and network/IPC namespaces on Linux, an unprivileged user in combination with `sandbox_init` on macOS, and [AppContainer Isolation](https://docs.microsoft.com/en-us/windows/win32/secauthz/appcontainer-isolation) on Windows.
+ - Fleetspeak, the next generation communication framework, is now enabled by default. It's still possible to enable the deprecated, legacy communication framework via `grr_config_updater`.
+ - New MSI installer for Windows clients. The old, self-extracting ZIP exe is now deprecated.
+ - `ListNamedPipes`: New flow for named pipe collection on Windows.
+ - Ongoing progress on the UIv2: early version of the next generation UI can be checked via the `<admin ui address>/v2` URL.
+
 ### 3.4.3.1 (May 19 2021)
 
 Regular release for Q2 2021.

what-is-grr.md

@@ -8,7 +8,7 @@ GRR consists of 2 parts: client and server.
 
 **GRR client** is deployed on systems that one might want to investigate. On every such system, once deployed, GRR client periodically polls GRR frontend servers for work. "Work" means running a specific action: downloading file, listing a directory, etc.
 
-**GRR server** infrastructure consists of several components (frontends, workers, UI servers) and provides web-based graphical user interface and an API endpoint that allows analysts to schedule actions on clients and view and process collected data.
+**GRR server** infrastructure consists of several components (frontends, workers, UI servers, fleetspeak) and provides web-based graphical user interface and an API endpoint that allows analysts to schedule actions on clients and view and process collected data.
 
 ## Remote forensics at scale