From 833d50533a3c7b5f24b8ae5030498af1041c4a09 Mon Sep 17 00:00:00 2001
From: jonavellecuerdo <cuerdojonavelle@gmail.com>
Date: Wed, 20 Sep 2023 13:21:28 -0400
Subject: [PATCH 1/2] IN-930 Create data flow diagram

Why these changes are being introduced:
* Provide a visual that encapsulates the flow of data from source
to destination.

How this addresses that need:
* Create a diagram showing data flow for a Carbon run
* Add link to Confluence document in the README

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/IN-930
---
 README.md | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/README.md b/README.md
index 11ad1ab..c62fabe 100644
--- a/README.md
+++ b/README.md
@@ -7,6 +7,43 @@ Carbon is a tool for loading data into [Symplectic Elements](https://support.sym
 
 Please refer to the [mitlib-tf-workloads-carbon](https://github.com/mitlibraries/mitlib-tf-workloads-carbon) for the deployment configuration.
 
+For more information on the Carbon application, please refer to our [internal documentation on Confluence](https://mitlibraries.atlassian.net/l/cp/1E6cMvuT).
+
+## Data flow
+
+This flowchart depicts the data flow from MIT's Data Warehouse to the application's in-memory buffered streams and to the final location of the output file, an XML file on the Elements FTP server.
+
+```mermaid
+---
+title:
+---
+
+flowchart TB
+   subgraph ext-source[Database]
+      mit-dwrhs[(MIT Data Warehouse)]
+   end
+   
+   subgraph in-memory[Application In-memory]
+      direction TB
+      rec-generator([Record generator])
+      buffered-writer([Buffered Writer])
+      buffered-reader([Buffered Reader])
+   end
+
+   subgraph elements-ftp[Elements FTP server]
+      direction TB
+      xml-file([Feed XML file])
+   end
+
+   mit-dwrhs -->|Fetch query results | rec-generator
+   rec-generator-->|Yielding records one at a time, <br> transform record into normalized XML strings <br> and write to a file| buffered-writer
+   buffered-writer -->|Stream contents from file to another read end of pipe| buffered-reader
+   buffered-reader -->|Stream contents from read end of pipe to an XML file on FTP server|xml-file
+
+
+```
+
+
 ## Development
 
 * To install with dev dependencies: `make install`

From bbb2f9a442ae7ed692eef2354387c051179733ec Mon Sep 17 00:00:00 2001
From: jonavellecuerdo <cuerdojonavelle@gmail.com>
Date: Wed, 18 Oct 2023 10:19:30 -0400
Subject: [PATCH 2/2] Update diagram

---
 README.md | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/README.md b/README.md
index c62fabe..5afaf52 100644
--- a/README.md
+++ b/README.md
@@ -25,9 +25,12 @@ flowchart TB
    
    subgraph in-memory[Application In-memory]
       direction TB
-      rec-generator([Record generator])
-      buffered-writer([Buffered Writer])
-      buffered-reader([Buffered Reader])
+      rec-generator([Query Results Generator])
+      subgraph piped[Piped Read-Write Buffer]
+         buffered-writer([Buffered Writer])
+         buffered-reader([Buffered Reader])
+      end
+      ftps-client((FTPS Client))
    end
 
    subgraph elements-ftp[Elements FTP server]
@@ -36,11 +39,10 @@ flowchart TB
    end
 
    mit-dwrhs -->|Fetch query results | rec-generator
-   rec-generator-->|Yielding records one at a time, <br> transform record into normalized XML strings <br> and write to a file| buffered-writer
-   buffered-writer -->|Stream contents from file to another read end of pipe| buffered-reader
-   buffered-reader -->|Stream contents from read end of pipe to an XML file on FTP server|xml-file
-
-
+   rec-generator-->|Yielding records one at a time, <br> transform record into normalized XML strings <br> and pass to write buffer| piped
+   buffered-writer -.->|Pipe contents to read buffer| buffered-reader
+   buffered-reader -->|Read buffer acts as data feed for an XML file on FTP server <br>| ftps-client
+   ftps-client -->|Stream contents from read buffer to an XML file on FTP server|xml-file
 ```