doc: documentation update for inlet/outlet separation

Author: vincentbernat

console/data/docs/00-intro.md

@@ -1,9 +1,9 @@
 # Introduction
 
-*Akvorado*[^name] receives flows (currently Netflow/IPFIX and sFlow), enriches
-them with interface names (using SNMP), geo information (using
-[IPinfo](https://ipinfo.io/) or MaxMind), and exports them to Kafka, then
-ClickHouse. It also exposes a web interface to browse the result.
+*Akvorado*[^name] receives network flows (currently Netflow/IPFIX and sFlow), enriches
+them with interface names (using SNMP), geographic information (using
+[IPinfo](https://ipinfo.io/) or MaxMind), and exports them to ClickHouse via
+Kafka. It also provides a web interface to browse the results.
 
 [^name]: [Akvorado][] means "water wheel" in Esperanto.
 
@@ -21,77 +21,76 @@ install the `docker-compose-v2` package. On macOS, you can use the
 # mkdir akvorado
 # cd akvorado
 # curl -sL https://github.com/akvorado/akvorado/releases/latest/download/docker-compose-quickstart.tar.gz | tar zxvf -
-# docker compose up -d
+# docker compose up
 ```
 
 Monitor the output of `docker compose ps`. Once `akvorado-console` service is
-"healthy", *Akvorado* web interface should be running on port 8081.
-
-A few synthetic flows are generated in the background. To disable them:
-
-1. Remove the reference for `docker-compose-demo.yml` from `.env`,
-2. Comment the last line of `akvorado.yaml`, and
-3. Run `docker compose up -d --remove-orphans`.
-
-If you want to send you own flows, the inlet is accepting both NetFlow
-(port 2055) and sFlow (port 6343). You should also customize some
-settings in `akvorado.yaml`. They are described in details in the
-[“configuration” section](02-configuration.md) section of the
-documentation.
-
-- `clickhouse` → `asns` to give names to your internal AS numbers (optional)
-- `clickhouse` → `networks` to attach attributes to your networks (optional)
-- `inlet` → `metadata` → `provider` → `communities` to set the communities to
-  use for SNMP queries (mandatory)
-- `inlet` → `core` → `exporter-classifiers` to define rules to attach
-  attributes to your exporters (optional)
-- `inlet` → `core` → `interface-classifiers` to define rules to attach
-  attributes to your interfaces, including the "boundary" attribute
-  which is used by default by the web interface (mandatory)
-
-The last point is very important: without classification, the homepage widgets
-won't work and nothing will be displayed by default in the “Visualize” tab. The
-provided configuration expects your interface to be named `Transit: Cogent`,
-`PNI: Akamai`, `PPNI: Fastly`, or `IX: DECIX`.
+present "healthy", *Akvorado* web interface should be running on port 8081. It
+can take a few minutes. The demo includes synthetic flow data to help you
+explore the features.
+
+### Next steps
+
+To connect your own network devices:
+
+1. **Disable demo data**:
+   - Remove the reference to `docker-compose-demo.yml` from `.env`
+   - Comment out the last line in `akvorado.yaml`
+
+1. **Customize the configuration** in `akvorado.yaml`:
+   - Set SNMP communities for your devices in `outlet` → `metadata` → `provider` → `communities`
+   - Configure interface classification rules in `outlet` → `core` → `interface-classifiers`
+
+1. **Configure your routers/switches** to send flows to *Akvorado*:
+   - NetFlow/IPFIX: port 2055
+   - sFlow: port 6343
+   
+1. **Restart all containers**
+   - `docker compose down --volumes`
+   - `docker compose up -d`
+
+> [!IMPORTANT]
+> Interface classification is essential for the web interface to work properly.
+> Without it, you won't see data in the dashboard widgets or visualization tab.
+> See the [configuration guide](02-configuration.md) for details.
+
+### Need help?
+
+- Check the [installation guide](01-install.md) for other deployment options
+- Read the [configuration guide](02-configuration.md) for detailed setup instructions
+- Review the [operations guide](04-operations.md) for router configuration examples
+- Check the [troubleshooting guide](05-troubleshooting.md) if you run into an issue
 
 You can get all the expanded configuration (with default values) with
 `docker compose exec akvorado-orchestrator akvorado orchestrator
 --check --dump /etc/akvorado/akvorado.yaml`.
 
-Once you are ready, you can run everything in the background with
-`docker compose up -d`.
-
 ## Big picture
 
 ![General design](design.svg)
 
-*Akvorado* is split into three components:
+*Akvorado* is split into four components:
 
-- The **inlet service** receives flows from exporters. It poll each
-  exporter using SNMP to get the *system name*, the *interface names*,
-  *descriptions* and *speeds*. It applies rules to add attributes to
-  exporters. Interface rules attach to each interface a *boundary*
-  (external or internal), a *network provider* and a *connectivity
-  type* (PNI, IX, transit). Optionally, it may also receive BGP routes
-  through the BMP protocol to get the *AS number*, the *AS path*, and
-  the communities. The flow is exported to *Kafka*, serialized using
-  *Protobuf*.
+- The **inlet service** receives flows from exporters and forwards them unparsed
+  to Kafka.
+
+- The **outlet service** consumes flows from Kafka, parses them, and enriches
+  them with metadata. It polls each exporter using SNMP to get the *system
+  name*, the *interface names*, *descriptions* and *speeds*. It applies rules to
+  add attributes to exporters. Interface rules attach to each interface a
+  *boundary* (external or internal), a *network provider* and a *connectivity
+  type* (PNI, IX, transit). Optionally, it may also receive BGP routes through
+  the BMP protocol to get the *AS number*, the *AS path*, and the communities.
+  The enriched flows are then exported to ClickHouse.
 
 - The **orchestrator service** configures the internal and external components.
   It creates the *Kafka topic* and configures *ClickHouse* to receive the flows
-  from Kafka. It exposes configuration settings for the other services to use.
-  It provides to ClickHouse additional data, notably *GeoIP* data.
-
-- The **console service** exposes a web interface to look and
-  manipulate the flows stored inside the ClickHouse database.
-
-## Serialized flow schemas
+  from the outlet service. It exposes configuration settings for the other
+  services to use. It provides to ClickHouse additional data, notably *GeoIP*
+  data.
 
-Flows sent to Kafka are encoded with a versioned schema. When the schema
-changes, a different Kafka topic is used. For example, the
-`flows-ZUYGDTE3EBIXX352XPM3YEEFV4` topic receive serialized flows using a
-specific version of the schema. The inlet service exports the schema with its
-HTTP service, via the `/api/v0/inlet/flow.proto` endpoint.
+- The **console service** provides a web interface to view and manipulate the
+  flows stored in the ClickHouse database.
 
 ## ClickHouse database schemas
 

console/data/docs/01-install.md

@@ -1,11 +1,10 @@
 # Installation
 
-*Akvorado* is written in Go. It provides its 3 components into a
-single binary or Docker image. It also requires an installation of
+*Akvorado* is written in Go. It provides its 4 components in a single binary or
+Docker image. It also requires an installation of
 [Kafka](https://kafka.apache.org/quickstart) and
-[ClickHouse](https://clickhouse.com/docs/en/getting-started/install/).
-They have to be installed separately. For ClickHouse, the minimal
-version is 22.4 (due to the use of the `INTERPOLATE` directive).
+[ClickHouse](https://clickhouse.com/docs/en/getting-started/install/). These
+must be installed separately. For ClickHouse, the minimum version is 22.4.
 
 ## Docker image
 
@@ -22,7 +21,7 @@ Check the `docker/docker-compose.yml` file for an example on how to deploy
 the [quick start procedure](00-intro.md#quick-start). This documentation assumes
 you are running the `docker compose` setup.
 
-If you want to compile the Docker image yourself, you can use `make docker`.
+If you want to compile the Docker image yourself, use `make docker`.
 
 ## Pre-built binary
 
@@ -32,9 +31,9 @@ Currently, only a pre-built binary for Linux x86-64 is provided.
 
 ## Compilation from source
 
-You need a proper installation of [Go](https://go.dev/doc/install) (1.24+),
-[NodeJS](https://nodejs.org/en/download/) (20+) with NPM (6+), and
-[protoc](https://protobuf.dev/installation/). For example, on Debian:
+You need a proper installation of [Go](https://go.dev/doc/install) (1.24+) and
+[NodeJS](https://nodejs.org/en/download/) (20+) with NPM (6+). For example, on
+Debian:
 
 ```console
 # apt install golang nodejs npm protobuf-compiler
@@ -94,7 +93,7 @@ The following `make` targets are available:
 
 Be sure to read the [changelog](99-changelog.md) before attempting an upgrade.
 Upgrade the orchestrator first. This will update the ClickHouse database if
-needed. Then, upgrade all inlets. Then the console.
+needed. Then, upgrade all inlets and outlets. Then the console.
 
 When using `docker compose`, use the following commands to fetch an updated
 `docker-compose.yml` file and update your installation.
@@ -109,6 +108,5 @@ When using `docker compose`, use the following commands to fetch an updated
 
 The `docker-compose-upgrade.tar.gz` tarball ships `.env.dist` instead of `.env`.
 You may want to check the differences with your setup (most of the time, there
-shouldn't have any). Note that if Zookeeper or Kakfa gets upgraded in the
-process, this can be disruptive. You may want to review the changes in
-`docker/versions.yml`.
+shouldn't be any). Note that if Kafka gets upgraded in the process, this can
+be disruptive. You may want to review the changes in `docker/versions.yml`.