From cca9f7e2dd63ac2356bf25d2ab49cb68661f5b9b Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 12 Sep 2024 12:11:19 +0100 Subject: [PATCH 1/4] DOC-4275 updated dataflow diagram --- .../redis-data-integration/data-pipelines/data-pipelines.md | 2 +- static/images/rdi/RDIPipeDataflow.drawio.svg | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 static/images/rdi/RDIPipeDataflow.drawio.svg diff --git a/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md b/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md index 48d28545cf..b243f3667e 100644 --- a/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md +++ b/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md @@ -48,7 +48,7 @@ data in Redis as The diagram below shows the flow of data through the pipeline: -{{< image filename="/images/rdi/data-transformation-pipeline.png" >}} +{{< image filename="/images/rdi/RDIPipeDataflow.drawio.svg" >}} ## Pipeline configuration diff --git a/static/images/rdi/RDIPipeDataflow.drawio.svg b/static/images/rdi/RDIPipeDataflow.drawio.svg new file mode 100644 index 0000000000..92b636b3f5 --- /dev/null +++ b/static/images/rdi/RDIPipeDataflow.drawio.svg @@ -0,0 +1,4 @@ + + + +
{/}
RDI
rename
field
calc
field
calc
Redis key
YAML
Transformation steps
Declarative transformation
Jobs described in YAML using SQL or JMESPath expressions.
Redis key
Hash
JSON
Set
Sorted set
Stream
String
Change event JSON
Single table/stream job
One or more output objects
 \ No newline at end of file From 9f2226d8892dd4cafce7b7576086942a3eb96974 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 19 Sep 2024 10:46:30 +0100 Subject: [PATCH 2/4] DOC-4275 updates to diagram based on feedback --- .../redis-data-integration/data-pipelines/data-pipelines.md | 4 ++-- static/images/rdi/RDIPipeDataflow.drawio.svg | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md b/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md index b243f3667e..71c3fab461 100644 --- a/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md +++ b/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md @@ -28,8 +28,8 @@ However, you can also provide your own custom transformation [jobs](#job-files) for each source table, using your own data mapping and key pattern. You specify these jobs declaratively with YAML configuration files that require no coding. -The data tranformation involves two separate stages. First, the data ingested by -[Debezium](https://debezium.io/) is automatically transformed to a JSON format. Then, +The data tranformation involves two separate stages. First, the data ingested +during CDC is automatically transformed to a JSON format. Then, this JSON data gets passed on to your custom transformation for further processing. You can provide a job file for each source table you want to transform, but you diff --git a/static/images/rdi/RDIPipeDataflow.drawio.svg b/static/images/rdi/RDIPipeDataflow.drawio.svg index 92b636b3f5..2405eaf57c 100644 --- a/static/images/rdi/RDIPipeDataflow.drawio.svg +++ b/static/images/rdi/RDIPipeDataflow.drawio.svg @@ -1,4 +1,4 @@ -
{/}
RDI
rename
field
calc
field
calc
Redis key
YAML
Transformation steps
Declarative transformation
Jobs described in YAML using SQL or JMESPath expressions.
Redis key
Hash
JSON
Set
Sorted set
Stream
String
Change event JSON
Single table/stream job
One or more output objects
 \ No newline at end of file +
{/}
RDI stream processor
rename
field
calc
field
calc
Redis key
RDI
job
(YAML)
Transformation steps
Declarative transformation
Jobs described in YAML using SQL or JMESPath expressions.
Redis key
Hash
JSON
Set
Sorted set
Stream
String
Single table/stream job
One or more output objects
Table
Name
Date_of_birth
Job_title
...
Relational source DB
Change
data
{/}
{/}
...
{/}
JSON
change
event
CDC
Redis stream
 \ No newline at end of file From e82cb6b2ff555f26836aca99b7e4144579643274 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 19 Sep 2024 11:01:07 +0100 Subject: [PATCH 3/4] DOC-4275 tagged on update to metrics ports --- content/integrate/redis-data-integration/observability.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/integrate/redis-data-integration/observability.md b/content/integrate/redis-data-integration/observability.md index 9d13764814..d9fe5c2125 100644 --- a/content/integrate/redis-data-integration/observability.md +++ b/content/integrate/redis-data-integration/observability.md @@ -39,7 +39,7 @@ command from the CLI.{{< /note >}} ## Collector metrics -The endpoint for the collector metrics is `https://:9121/metrics/collector-source` +The endpoint for the collector metrics is `https:///metrics/collector-source` These metrics are divided into three groups: @@ -49,7 +49,7 @@ These metrics are divided into three groups: ## Stream processor metrics -The endpoint for the stream processor metrics is `https://:9121/metrics/rdi` +The endpoint for the stream processor metrics is `https:///metrics/rdi` RDI reports metrics during the two main phases of the ingest pipeline, the *snapshot* phase and the *change data capture (CDC)* phase. (See the From eb960074280763b4d0a4f22ff73d62802dbf00bf Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 19 Sep 2024 12:46:01 +0100 Subject: [PATCH 4/4] DOC-4275 further updates to diagram --- static/images/rdi/RDIPipeDataflow.drawio.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/static/images/rdi/RDIPipeDataflow.drawio.svg b/static/images/rdi/RDIPipeDataflow.drawio.svg index 2405eaf57c..58e9019de5 100644 --- a/static/images/rdi/RDIPipeDataflow.drawio.svg +++ b/static/images/rdi/RDIPipeDataflow.drawio.svg @@ -1,4 +1,4 @@ -
{/}
RDI stream processor
rename
field
calc
field
calc
Redis key
RDI
job
(YAML)
Transformation steps
Declarative transformation
Jobs described in YAML using SQL or JMESPath expressions.
Redis key
Hash
JSON
Set
Sorted set
Stream
String
Single table/stream job
One or more output objects
Table
Name
Date_of_birth
Job_title
...
Relational source DB
Change
data
{/}
{/}
...
{/}
JSON
change
event
CDC
Redis stream
 \ No newline at end of file +
RDI stream processor
rename
field
calc
field
calc
Redis key
RDI
job
(YAML)
Transformation steps
Declarative transformation
Jobs described in YAML using SQL or JMESPath expressions.
Redis key
Hash
JSON
Set
Sorted set
Stream
String
Single table/stream job
One or more output objects
Table
Name
Date_of_birth
Job_title
...
Relational source DB
Change
data
{/}
{/}
...
{/}
JSON
change
event
Redis stream
RDI collector
(CDC)
 \ No newline at end of file