From d09520152e4e4fd90cd474853e933943adf45423 Mon Sep 17 00:00:00 2001 From: tbedford Date: Tue, 12 Mar 2024 10:08:23 +0000 Subject: [PATCH 1/2] [add] - migration guide to cross-ref tutorial --- .../databases/influxdb/migrating-v2-v3.md | 30 +++++++++++++++++++ .../databases/influxdb/overview.md | 4 ++- mkdocs.yml | 1 + 3 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 docs/integrations/databases/influxdb/migrating-v2-v3.md diff --git a/docs/integrations/databases/influxdb/migrating-v2-v3.md b/docs/integrations/databases/influxdb/migrating-v2-v3.md new file mode 100644 index 00000000..1bc146af --- /dev/null +++ b/docs/integrations/databases/influxdb/migrating-v2-v3.md @@ -0,0 +1,30 @@ +# Migrating InfluxDB from v2 to v3 + +If you have data in a v2 InfluxDB database, and you want to migrate it to InfluxDB v3, then Quix can help. + +Quix provides the following InfluxDB [connectors](../../../connectors/index.md): + +* InfluxDB v2 source +* InfluxDB v3 source +* InfluxDB v3 destination + +A summary of the procedure is: + +1. Add the Quix InfluxDB v2 connector to Quix, to connect your InfluxDB v2 database to Quix. +2. Add the Quix InfluxDB v3 connector to Quix, on the output of your InfluxDB v2 connector. + +You'll have the following pipeline: + +![Migration pipeline](../../../tutorials/influxdb-migration/images/influxdb-migration-pipeline.png) + +Data is read from InfluxDB v2 and published to InfluxDB v3. + +No coding is required, unless you also want to make changes to data as it is migrated. If you want to do this you can add a transform (or multiple transforms) between the two connectors. + +## See also + +If you are new to Quix you could try our [Quickstart](../../../get-started/quickstart.md) and then complete the [Quix Tour](../../../get-started/quixtour/overview.md). This gives you a good overview of how to use Quix, for a minimal investment in your time. + +There is an [in-depth step-by-step tutorial](../../../tutorials/influxdb-migration/overview.md) available on migrating InfluxDB v2 data to InfluxDB v3. + +A tutorial featuring a more complex pipeline is also available. See the [InfluxDB alerting tutorial featuring PagerDuty](../../../tutorials/influxdb-alerting/overview.md). diff --git a/docs/integrations/databases/influxdb/overview.md b/docs/integrations/databases/influxdb/overview.md index 52b5700f..715ebe59 100644 --- a/docs/integrations/databases/influxdb/overview.md +++ b/docs/integrations/databases/influxdb/overview.md @@ -17,7 +17,9 @@ Quix currently supports the following connectors for InfluxDB: | [InfluxDB 3.0](https://github.com/quixio/quix-samples/tree/main/python/sources/InfluxDB){target=_blank} | Source | Enables you to publish data from an InfluxDB v3 bucket into a Quix topic. | | [InfluxDB 3.0](https://github.com/quixio/quix-samples/tree/main/python/destinations/InfluxDB){target=_blank} | Destination | Enables you to publish data from a Quix topic into an InfluxDB v3 bucket. | -Note if you want to move data from InfluxDB v2 to v3 you could use the v2 source and v3 destination to do this. +!!! tip + + If you want to [migrate data from InfluxDB v2 to v3](./migrating-v2-v3.md) you could use the Quix InfluxDB v2 source and v3 destination connectors to do this.
Try the Quickstart diff --git a/mkdocs.yml b/mkdocs.yml index a1abb6e9..d2c3a8eb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -132,6 +132,7 @@ nav: - 'Using Telegraf': 'integrations/databases/influxdb/telegraf.md' - 'Replacing Flux': 'integrations/databases/influxdb/replacing-flux.md' - 'Alerting': 'integrations/databases/influxdb/alerting.md' + - 'Migrating from v2 to v3': 'integrations/databases/influxdb/migrating-v2-v3.md' - 'Tutorials': - 'Overview': 'tutorials/overview.md' - 'Migrating InfluxDB v2 to v3': From d8d2cdf4aced3e07ccd4a521fd271250fdab81ca Mon Sep 17 00:00:00 2001 From: tbedford Date: Wed, 13 Mar 2024 16:02:31 +0000 Subject: [PATCH 2/2] [add] - details of required config --- docs/develop/integrate-data/read-csv.md | 2 +- .../databases/influxdb/migrating-v2-v3.md | 35 +++++++++++++++++++ 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/docs/develop/integrate-data/read-csv.md b/docs/develop/integrate-data/read-csv.md index f329ad08..746afa65 100644 --- a/docs/develop/integrate-data/read-csv.md +++ b/docs/develop/integrate-data/read-csv.md @@ -14,7 +14,7 @@ You'll need a streaming token to authenticate with Quix if you're going to run a In Quix Cloud, the streaming token is automatically set in the project environment. -Once you [get your streaming token]((../../develop/authentication/streaming-token.md)), you can subsequently load it from a suitable `.env` file: +Once you [get your streaming token](../../develop/authentication/streaming-token.md), you can subsequently load it from a suitable `.env` file: ``` Quix__Sdk__Token="" diff --git a/docs/integrations/databases/influxdb/migrating-v2-v3.md b/docs/integrations/databases/influxdb/migrating-v2-v3.md index 1bc146af..0000e4d2 100644 --- a/docs/integrations/databases/influxdb/migrating-v2-v3.md +++ b/docs/integrations/databases/influxdb/migrating-v2-v3.md @@ -21,6 +21,41 @@ Data is read from InfluxDB v2 and published to InfluxDB v3. No coding is required, unless you also want to make changes to data as it is migrated. If you want to do this you can add a transform (or multiple transforms) between the two connectors. +## Configuring the source connector + +You'll need to configure the following variables for your source connector: + +| Environment Variable | Description| +|---|---| +| `output` | Output topic. Add a new topic called `influxdb-v2-data`. | +| `INFLUXDB_DATABASE` | Database name in InfluxDB where the data is stored. In InfluxDB this is also called a bucket. | +| `INFLUXDB_HOST` | Host address for the InfluxDB instance. Default: `eu-central-1-1.aws.cloud2.influxdata.com`. | +| `INFLUXDB_ORG` | Organization name in InfluxDB. | +| `INFLUXDB_TOKEN` | Authentication token to access InfluxDB. Configure as a secret type, so that your token is never revealed. | +| `task_interval` | The polling period for queries of the database. Set this to `1s` (one second) to ensure you see any new data promptly (this make testing a little easier as you don't need to wait too long for updates). | + +!!! tip + + To see more documentation on these variables, and examples, you can refer to the [detailed README](https://github.com/quixio/template-influxv2-to-v3?tab=readme-ov-file#influxdb-v2-source){target=_blank}. + +## Configuring the destination connector + +You'll need to configure the following variables for your destination connector: + +| Environment Variable | Description| +|---|---| +| `input` | Input topic. Example `influxdb-v2-data`. | +| `INFLUXDB_DATABASE` | Database name in InfluxDB where the data is stored. In InfluxDB this is also called a bucket. | +| `INFLUXDB_HOST` | Host address for the InfluxDB instance. Default: `eu-central-1-1.aws.cloud2.influxdata.com`. | +| `INFLUXDB_ORG` | Organization name in InfluxDB. | +| `INFLUXDB_TOKEN` | Authentication token to access InfluxDB. Configure as a secret type, so that your token is never revealed. | +| `INFLUXDB_FIELD_KEYS` | These are the columns of rows of data that you want to write to the InfluxDB v3 database. | +| `INFLUXDB_TAG_KEYS` | The metadata that you want to write to the InfluxDB v3 database. | + +!!! tip + + To see more documentation on these variables, and examples, you can refer to the [detailed README](https://github.com/quixio/template-influxv2-to-v3?tab=readme-ov-file#influxdb-v3-sink){target=_blank}. + ## See also If you are new to Quix you could try our [Quickstart](../../../get-started/quickstart.md) and then complete the [Quix Tour](../../../get-started/quixtour/overview.md). This gives you a good overview of how to use Quix, for a minimal investment in your time.