Docs for multi-tenancy.

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

README.md

@@ -56,8 +56,11 @@ remote storage endpoints. For more information about prom-migrator, visit
   * [Data Model](docs/sql_schema.md#data-model-schema)
   * [Filtering Series](docs/sql_schema.md#filtering-series)
   * [Data Retention](docs/sql_schema.md#data-retention)
-  * [Roles and Permissions](docs/sql_permissions.md)
+  * [Roles and Permissions](docs/sql_permissions.sql)
 * **[Tutorial: Getting Started with Promscale](https://docs.timescale.com/timescaledb/latest/tutorials/promscale/)** 
+* **[High Availability](docs/high-availability/prometheus-HA.md)**
+* **[Multi-tenancy](docs/multi_tenancy.md)**
+* **[Multi-Node TimescaleDB](docs/multinode.md)**
 * **[Writing Data to Promscale](docs/writing_to_promscale.md)**
 * **[Alerting & Recording Rules](docs/alerting-recording.md)**
 * **[Deleting Data](docs/metric_deletion_and_retention.md)**
@@ -106,11 +109,15 @@ Click the video below for an overview of Promscale:
   SQL for deeper analytics and compatibility with a huge ecosystem of data visualization, analysis, and AI/ML tools.
 * **Rock-solid stability** due to being built on top of PostgreSQL, with 30+ years of development work.
 * **Support for backfilling** to ingest data from the past.
+* **Native support for Multi-Tenancy**. Ingest data from multiple tenants and write queries across [more than one tenant](docs/multi_tenancy.md)
+  easily, using PromQL or SQL.
 * **High-Availability** support for [Prometheus HA deployments](docs/high-availability/prometheus-HA.md) as well as
   high-availability deployments of [TimescaleDB itself](https://blog.timescale.com/blog/high-availability-timescaledb-postgresql-patroni-a4572264a831/).
 * **Simple architecture**. Unlike some other long-term stores, our architecture consists of only three components: Prometheus, Promscale, and TimescaleDB.
 * **ACID compliance** to ensure consistency of your data.
 * **Horizontal scalability** using [multinode support](https://blog.timescale.com/blog/timescaledb-2-0-a-multi-node-petabyte-scale-completely-free-relational-database-for-time-series/) with TimescaleDB version 2.0.
+* **Operationally mature**. Built on top of Postgres, data written via Promscale is safe, reliable and offers
+  all of the advantages of an operationally mature platform.
 
 ## 🔧 Choose your own (installation) adventure
 

docs/multi_tenancy.md

@@ -0,0 +1,121 @@
+# Multi-tenancy
+
+Promscale supports multi-tenancy by having different Prometheus servers
+corresponding to different tenants all write data to the same set of Promscale
+servers. Each Prometheus server will indicate the tenant it is writing using
+either custom headers or external labels.
+
+When executing PromQL queries, a Promscale server can be configured to either
+return data for all tenants or to restrict its answers to a particular tenant or set or tenants.
+In this way, a tenant can be given access to a particular Promscale PromQL endpoint for
+querying its own data while being sure that it could not access other tenant's data.
+
+Multi-tenancy in Promscale features:
+1. Simple configuration for multi-tenancy using headers or external labels
+2. Cross-tenant queries in PromQL and SQL
+3. Allow data with no tenant information to be written or queried when multi-tenant mode is enabled
+4. Restricting the set of valid tenants a Promscale instance can ingest or query
+5. Authorization of tenants using bearer_tokens
+
+## Configuring Promscale for multi-tenancy
+
+By default, Promscale ingests data without multi-tenancy. However, configuring Promscale for multi-tenancy is simple.
+For Promscale to ingest tenant data, apply the `-multi-tenancy` flag during the start. This will allow Promscale to
+ingest data from all tenants by default. If you want Promscale to ingest or query data from a restricted list of tenants, mention the
+tenant names in `-multi-tenancy-valid-tenants=<tenant_name_seperated_by_commas>`.
+
+When multi-tenancy is enabled, accepting data without a tenant name being specified is disabled by default.
+If you want to accept data without the tenant information, apply the `-multi-tenancy-allow-non-tenants` flag.
+
+Remember to set the appropriate CLI flags on all of your Promscale instances.
+
+Example:
+
+If we want to ingest data from `tenant-A` and `tenant-B` Prometheus instances and also allow data to
+be ingested from non-tenant instances of Prometheus, then the required configuration flags will be:
+
+```shell
+-multi-tenancy -multi-tenancy-valid-tenants=tenant-A,tenant-B -multi-tenancy-allow-non-tenants
+```
+
+If we want to ingest all incoming tenant data (i.e., irrespective of tenant validation) from
+Prometheus instances along with non-tenant Prometheus instances
+
+```shell
+-multi-tenancy -multi-tenancy-allow-non-tenants
+```
+
+Note: `-multi-tenancy-valid-tenants` has a default value as `allow-all`.
+
+## Configuring Prometheus for writing multi-tenant data
+
+Promscale happily accepts tenant information either via `headers` or using `external_labels`.
+You can choose either of the ways depending on your needs.
+
+Promscale keeps tenant name as `__tenant__: <tenant_name>` label pair attached to the series that is
+ingested in that tenant's write-request.
+
+### Using headers
+
+Tenant name can be supplied with a header with the key `TENANT` and the value corresponding the tenant name.
+
+Example:
+
+`TENANT: tenant-A`
+
+In Prometheus remote-write config
+
+```yaml
+remote_write:
+  - url: http://localhost:9201/write
+    headers:
+      TENANT: A
+```
+
+### Using external_labels
+
+Promscale reserves `__tenant__` as label_key for storing tenant name for the incoming series. You can
+push data directly with this label as
+
+```yaml
+global:
+  external_labels:
+    __tenant__: A
+```
+
+## Querying multi-tenant data
+
+Since data from any tenant is kept with a `__tenant__` label key, it can be used to query any tenant data.
+Please note only those tenants are allowed in the query, that is authorized under `-multi-tenancy-valid-tenants`
+flag. Moreover, data without a tenant label can be queried only if `-multi-tenancy-allow-non-tenants` is applied.
+
+If a query aims to be evaluated across multiple tenants (say tenant-A and tenant-B), it can be done
+by `metric_name{__tenant__=~"tenant-A|tenant-B"}` respectively.
+
+Example:
+
+Imagine if Promscale contains data from `tenant-A`, `tenant-B` and `tenant-C`. If Promscale is configured with
+`-multi-tenancy-valid-tenants=tenant-A,tenant-B` then,
+
+If the PromQL query is
+1. `metric_name{__tenant__=tenant-A}` then data corresponding to `metric_name` from `tenant-A` will
+    only be returned.
+2. `metric_name{__tenant__=tenant-C}` then no data will be returned.
+3. `metric_name{__tenant__=~tenant-A|tenant-B}` then data from `metric_name` from tenants `tenant-A` and `tenant-B`
+    will be returned.
+4. `metric_name{__tenant__=~tenant-A|tenant-C}` then data from `metric_name` will be only from `tenant-A` and
+    not from `tenant-C`. This is because the Promscale instance is configured to return data from only `tenant-A` and `tenant-B`.
+5. `metric_name` then the data from `metric_name` will be from `tenant-A` and `tenant-B` only.
+
+Now, if the same database also contains samples without any tenant label and `-multi-tenancy-allow-non-tenants` is applied then
+
+If the PromQL query is
+1. `metric_name{__tenant__=tenant-A}` then data corresponding to `metric_name` from `tenant-A` will
+   only be returned.
+2. `metric_name{__tenant__=tenant-A|}` then data corresponding to `metric_name` from non-tenant and from `tenant-A`
+   will be returned.
+3. `metric_name` then data corresponding to `metric_name` will be from the non-tenants, `tenant-A` and `tenant-B` respectively.
+4. `metric_name{__tenant__=""}` then data will be only from non-tenants.
+
+Note: If you are querying from multiple Promscales, you can **also** configure individual Promscale instances (differently) to selectively
+authorize any valid tenant for queries, based on your requirement.