diff --git a/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md b/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md index 48d28545cf..71c3fab461 100644 --- a/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md +++ b/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md @@ -28,8 +28,8 @@ However, you can also provide your own custom transformation [jobs](#job-files) for each source table, using your own data mapping and key pattern. You specify these jobs declaratively with YAML configuration files that require no coding. -The data tranformation involves two separate stages. First, the data ingested by -[Debezium](https://debezium.io/) is automatically transformed to a JSON format. Then, +The data tranformation involves two separate stages. First, the data ingested +during CDC is automatically transformed to a JSON format. Then, this JSON data gets passed on to your custom transformation for further processing. You can provide a job file for each source table you want to transform, but you @@ -48,7 +48,7 @@ data in Redis as The diagram below shows the flow of data through the pipeline: -{{< image filename="/images/rdi/data-transformation-pipeline.png" >}} +{{< image filename="/images/rdi/RDIPipeDataflow.drawio.svg" >}} ## Pipeline configuration diff --git a/content/integrate/redis-data-integration/observability.md b/content/integrate/redis-data-integration/observability.md index 9d13764814..d9fe5c2125 100644 --- a/content/integrate/redis-data-integration/observability.md +++ b/content/integrate/redis-data-integration/observability.md @@ -39,7 +39,7 @@ command from the CLI.{{< /note >}} ## Collector metrics -The endpoint for the collector metrics is `https://:9121/metrics/collector-source` +The endpoint for the collector metrics is `https:///metrics/collector-source` These metrics are divided into three groups: @@ -49,7 +49,7 @@ These metrics are divided into three groups: ## Stream processor metrics -The endpoint for the stream processor metrics is `https://:9121/metrics/rdi` +The endpoint for the stream processor metrics is `https:///metrics/rdi` RDI reports metrics during the two main phases of the ingest pipeline, the *snapshot* phase and the *change data capture (CDC)* phase. (See the diff --git a/static/images/rdi/RDIPipeDataflow.drawio.svg b/static/images/rdi/RDIPipeDataflow.drawio.svg new file mode 100644 index 0000000000..58e9019de5 --- /dev/null +++ b/static/images/rdi/RDIPipeDataflow.drawio.svg @@ -0,0 +1,4 @@ + + + +
RDI stream processor
rename
field
calc
field
calc
Redis key
RDI
job
(YAML)
Transformation steps
Declarative transformation
Jobs described in YAML using SQL or JMESPath expressions.
Redis key
Hash
JSON
Set
Sorted set
Stream
String
Single table/stream job
One or more output objects
Table
Name
Date_of_birth
Job_title
...
Relational source DB
Change
data
{/}
{/}
...
{/}
JSON
change
event
Redis stream
RDI collector
(CDC)
\ No newline at end of file