v0.2.0 (#121)

* Add Releases Feed to Docs Release page (#69) * fixes Add Releases Feed to Docs page #68 Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Mkdocs test fixes Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add unit tests Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Fix Code Smell Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> --------- Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Update Packages and Doc Logos (#76) * Upgrade Packages Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Upgrade Functions Packages Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add logos Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add logo to pypi readme Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Remove pydantic models Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Remove GraphQL Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Remove strawberry type Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Remove GraphQL Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Upgrade GraphQL Package Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Logo Size Changes Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> --------- Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Fix Logo Alignment (#78) * Upgrade Packages Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Upgrade Functions Packages Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add logos Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add logo to pypi readme Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Remove pydantic models Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Remove GraphQL Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Remove strawberry type Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Remove GraphQL Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Upgrade GraphQL Package Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Logo Size Changes Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Logo fixes Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> --------- Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * fixes #77 (#79) Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * add spark session (#80) * set up spark sessions Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * adding unit tests for spark session Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * remove tests for spark session Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * fix code smells Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * add constants in pipeline utils Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> --------- Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * Add Pipelines as Extras libraries (#82) Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Spark pipeline source Event hub and destination Event hub (#83) * add source eventhub to pipelines Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * updates for documentation Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * updates to mkdocs Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * added unit tests Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * documentation for eventhub source Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * fix spelling error Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * add source eventhub destination Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * added destination source documentation Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * remove unit tests to revisit later Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * fix mkdoc errors Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * fix code smells Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> --------- Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * Spark pipeline source delta and destination delta (#84) * Spark pipeline source delta and destination delta Signed-off-by: rodalynbarce <Rodalyn.Barce@shell.com> * fix mkdocs errors Signed-off-by: rodalynbarce <Rodalyn.Barce@shell.com> * docstring amendments Signed-off-by: rodalynbarce <Rodalyn.Barce@shell.com> * Added to docstrings on delta source & destination Signed-off-by: JamesKnBr <James.Broady@Shell.com> * added query type for eventhubs Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> --------- Signed-off-by: rodalynbarce <Rodalyn.Barce@shell.com> Signed-off-by: JamesKnBr <James.Broady@Shell.com> Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> Co-authored-by: JamesKnBr <James.Broady@Shell.com> Co-authored-by: cching95 <73163191+cching95@users.noreply.github.com> * Added delta sharing with docs (#85) Signed-off-by: JamesKnBr <James.Broady@Shell.com> * Unit Tests for sources (#88) * add pipeline utils unit tests Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * add unit test for eventhub and delta Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * Unit Test Fixes Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * remove test spark Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * add unit test for destination delta Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * fix eventhub spell errors Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> --------- Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> Co-authored-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add pipeline eventhub body transformer (#89) * add transform eventhub Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * transformer docs set up Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * documentation for transformer eventhub body Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * add transformer eventhub unit tests Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> --------- Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * Added pipeline source autoloader and unit test (#93) Signed-off-by: rodalynbarce <Rodalyn.Barce@shell.com> * Pipeline Executor (#95) * Dependency Injector Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add Pipeline Executor Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add Utilities Interface Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add Pipeline Executor Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add Pipeline Execute Test Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * fixes #90 Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Fixes for tests Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Step Order fix Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Sonarqube fixes Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> --------- Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Delta Table Create Utility (#97) * Add to docs site Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Update Delta Table Name Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add Delta Table Create Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add Delta Table Create Test Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> --------- Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * remove print statement in twa (#99) Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> Co-authored-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> Co-authored-by: Amber-Rigg <44523299+Amber-Rigg@users.noreply.github.com> Co-authored-by: NooraKubati <55835429+NooraKubati@users.noreply.github.com> * Pipeline Deployment (#102) * Add DBX Pipeline Deployment Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Updates for Pipeline Deployment Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> --------- Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Pipeline Deployment Updates (#103) * Pipeline Deployment Updates Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Update to run logic of Pipeline Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> --------- Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Updates for Pipeline Deployment (#104) Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Unit test for spark pipeline eventhub destination (#105) * eventhub destination unit test Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * add unit test for spark eventhub destination Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * add spark eventhub write stream unit test Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * update exception handling for eventhub and delta Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * update exception in utilities Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> --------- Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * Fix for Dependency Injection (#106) Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Unit tests for source and destination streams (#108) * streaming unit test for source eventhub Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * add delta read stream unit test Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * add unit test for delta destination streaming Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * remove extra test Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * remove code smell Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * add previous line of code back Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> --------- Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * Pipeline Deployments and Pipeline Secrets (#111) * Fix for Dependency Injection Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Pipeline Deployment And Secrets Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Add System Type Docs Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Updates for Secrets Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Code Smell fix Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Refactor Pipeline Secret Conversion Logic Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * import required objects Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Code smell fix Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Code Smell Fix Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Fix Code Smells Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> --------- Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Added fails tests to pipeline sources (#112) * Added fails tests to pipeline sources Signed-off-by: JamesKnBr <James.Broady@Shell.com> * Added fails tests for pipeline destinations Signed-off-by: JamesKnBr <James.Broady@Shell.com> * Fixed code smell Signed-off-by: JamesKnBr <James.Broady@Shell.com> --------- Signed-off-by: JamesKnBr <James.Broady@Shell.com> * Fix for dockerbuild error (#114) Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Fix for docker build (#115) Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Cleanup Dockerfile (#116) Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Pipeline Secrets, Converters and Deployment Updates (#119) * Updates for Pipelines Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Update DBX Documentation Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Updates for Pipeline Deployment Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Remove package.json Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * Fix code smells Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> --------- Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> * time weighted average bug fix (#120) * fix twa group by dataframe return Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> * bug fixes for groupby and apply method Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> --------- Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> --------- Signed-off-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> Signed-off-by: cching95 <73163191+cching95@users.noreply.github.com> Signed-off-by: rodalynbarce <Rodalyn.Barce@shell.com> Signed-off-by: JamesKnBr <James.Broady@Shell.com> Co-authored-by: GBBBAS <42962356+GBBBAS@users.noreply.github.com> Co-authored-by: rodalynbarce <121169437+rodalynbarce@users.noreply.github.com> Co-authored-by: JamesKnBr <James.Broady@Shell.com> Co-authored-by: JamesKnBr <125024404+JamesKnBr@users.noreply.github.com> Co-authored-by: Amber-Rigg <44523299+Amber-Rigg@users.noreply.github.com> Co-authored-by: NooraKubati <55835429+NooraKubati@users.noreply.github.com>
rtdip · Mar 29, 2023 · 07e25d5 · 07e25d5
1 parent 874a098
commit 07e25d5
Show file tree

Hide file tree

Showing 114 changed files with 3,787 additions and 115 deletions.
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
@@ -28,12 +28,15 @@
 				"python.linting.pydocstylePath": "/opt/conda/bin/pydocstyle",
 				"python.linting.pylintPath": "/opt/conda/bin/pylint",
 				"python.testing.pytestArgs": [
+					"--cov=.",
+					"--cov-report=xml:cov.xml",					
 					"tests"
 				],
 				"python.testing.unittestEnabled": false,
 				"python.testing.pytestEnabled": true,
+				"python.testing.cwd": "${workspaceFolder}",
 				"python.analysis.extraPaths": ["${workspaceFolder}"],
-				"python.defaultInterpreterPath": "/opt/conda/envs/rtdip-sdk/bin/python",
+				"python.defaultInterpreterPath": "~/micromamba/envs/rtdip-sdk/bin/python",
 				"terminal.integrated.env.linux":{
 					"PYTHONPATH": "${workspaceFolder}:${env:PYTHONPATH}"        
 				},				

diff --git a/.gitignore b/.gitignore
@@ -43,6 +43,7 @@ htmlcov/
 .nox/
 .coverage
 .coverage.*
+cov.xml
 .cache
 nosetests.xml
 coverage.xml
@@ -135,4 +136,7 @@ spark-warehouse/
 spark-checkpoints/
 
 # Delta Sharing
-config.share
+config.share
+
+# DBX package
+src/sdk/python/rtdip_sdk/pipelines/deploy/dbx/.dbx/project.json
diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -7,6 +7,18 @@
       "request": "attach",
       "port": 9091,
       "preLaunchTask": "func: host start",
-    }
+    },
+    {
+      "name": "Python: Debug Tests",
+      "type": "python",
+      "request": "launch",
+      "program": "${file}",
+      "purpose": ["debug-test"],
+      "console": "internalConsole",
+      "env": {
+          "PYTEST_ADDOPTS": "--no-cov"
+      },
+      "justMyCode": false
+  }    
   ]
 }
diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -12,12 +12,15 @@
     "python.linting.pydocstylePath": "/opt/conda/bin/pydocstyle",
     "python.linting.pylintPath": "/opt/conda/bin/pylint",
     "python.testing.pytestArgs": [
+        "--cov=.",
+        "--cov-report=xml:cov.xml",
         "tests"
     ],
     "python.testing.unittestEnabled": false,
     "python.testing.pytestEnabled": true,
+    "python.testing.cwd": "${workspaceFolder}",
     "python.analysis.extraPaths": ["${workspaceFolder}"],
-    "python.defaultInterpreterPath": "/opt/conda/envs/rtdip-sdk/bin/python",
+    "python.defaultInterpreterPath": "~/micromamba/envs/rtdip-sdk/bin/python",
     "terminal.integrated.env.osx":{
         "PYTHONPATH": "${workspaceFolder}:${env:PYTHONPATH}"        
     },    

diff --git a/PYPI-README.md b/PYPI-README.md
@@ -1,4 +1,6 @@
-# What is RTDIP SDK?
+<p align="center"><img src=https://raw.githubusercontent.com/rtdip/core/develop/docs/getting-started/images/rtdip-horizontal-color.png alt="rtdip" width=50% height=50%/></p>
+
+# What is the RTDIP SDK?
 
 **Real Time Data Ingestion Platform (RTDIP) SDK** is a software development kit built to easily access some of RTDIP's transformation functions.
 

diff --git a/docs/__init__.py b/docs/__init__.py
@@ -0,0 +1,14 @@
+# Copyright 2022 RTDIP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
diff --git a/docs/api/overview.md b/docs/api/overview.md
@@ -2,4 +2,4 @@
 
 # RTDIP REST APIs
 
-RTDIP provides REST API endpoints for querying data in the platform. The APIs are a wrapper to the python [RTDIP SDK](../sdk/overview.md) and provide similar functionality for users and applications that are unable to leverage the python RTDIP SDK. It is recommended to read the [RTDIP SDK documentation](../sdk/overview.md) and in particular the [Functions](../sdk/code-reference/resample.md) section for more information about the options and logic behind each API.
+RTDIP provides REST API endpoints for querying data in the platform. The APIs are a wrapper to the python [RTDIP SDK](../sdk/overview.md) and provide similar functionality for users and applications that are unable to leverage the python RTDIP SDK. It is recommended to read the [RTDIP SDK documentation](../sdk/overview.md) and in particular the [Functions](../sdk/code-reference/query/resample.md) section for more information about the options and logic behind each API.
diff --git a/docs/blog/rtdip_ingestion_pipelines.md b/docs/blog/rtdip_ingestion_pipelines.md
@@ -158,7 +158,7 @@ Destinations are components that connect to sink/destination systems and write d
 |---------------------------|----------------------|--------------------|----------------------|----------------------|---------|---------|
 | Delta Append|*:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|Q1 2023|
 | Eventhub|*:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:||Q1 2023|
-| Delta Merge||:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|Q1 2023|
+| Delta Merge||:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|Q2 2023|
 | Kafka|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|Q2 2023|
 | Kinesis||:heavy_check_mark:|:heavy_check_mark:||:heavy_check_mark:|Q2 2023|
 | Rest API|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|Q2 2023|
@@ -193,4 +193,4 @@ Edge components are designed to provide a lightweight, low latency, low resource
 
 ## Conclusion
 
-This is a very high level overview of the framework and the components that will be developed. As the framework is open source, the lists defined above and timelines can change depending on circumstances and resource availability. Its an exciting year for 2023 for the Real Time Data Ingestion Platform. Check back in regularly for updates and new features! If you would like to contribute, please visit our repository onn Github and connect with us on our Slack channel on the LF Energy Foundation Slack workspace.
+This is a very high level overview of the framework and the components that will be developed. As the framework is open source, the lists defined above and timelines can change depending on circumstances and resource availability. Its an exciting year for 2023 for the Real Time Data Ingestion Platform. Check back in regularly for updates and new features! If you would like to contribute, please visit our repository on Github and connect with us on our Slack channel on the LF Energy Foundation Slack workspace.
diff --git a/docs/macros.py b/docs/macros.py
@@ -0,0 +1,33 @@
+# Copyright 2022 RTDIP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from github import Github
+
+def define_env(env):
+    @env.macro
+    def github_releases(owner, repo):
+        github_client = Github()
+        repo = github_client.get_repo("{}/{}".format(owner, repo))
+        output = "----\r\n"
+        for release in repo.get_releases():
+            title_markdown = "##[{}]({})".format(release.title, release.html_url)
+
+            subtitle_markdown = ":octicons-tag-24:[{}]({}) ".format(release.tag_name, release.html_url)
+            subtitle_markdown += ":octicons-calendar-24: Published {} ".format(release.published_at)
+            if release.draft:
+                subtitle_markdown += ":octicons-file-diff-24: Draft "
+            if release.prerelease:
+                subtitle_markdown += ":octicons-git-branch-24: Pre-release "
+            output += "{}\r\n{}\r\n{}\r\n----\r\n".format(title_markdown, subtitle_markdown, github_client.render_markdown(release.body))
+        return output
diff --git a/docs/releases/core.md b/docs/releases/core.md
@@ -0,0 +1,3 @@
+# Releases 
+
+{{ github_releases("rtdip", "core") }}
diff --git a/docs/releases/images/cicd.png b/docs/releases/images/cicd.png
diff --git a/docs/releases/releases.md b/docs/releases/releases.md
diff --git a/docs/sdk/code-reference/pipelines/converters/pipeline_job_json.md b/docs/sdk/code-reference/pipelines/converters/pipeline_job_json.md
@@ -0,0 +1 @@
+::: src.sdk.python.rtdip_sdk.pipelines.converters.pipeline_job_json
diff --git a/docs/sdk/code-reference/pipelines/deploy/databricks_dbx.md b/docs/sdk/code-reference/pipelines/deploy/databricks_dbx.md
@@ -0,0 +1 @@
+::: src.sdk.python.rtdip_sdk.pipelines.deploy.databricks
diff --git a/docs/sdk/code-reference/pipelines/destinations/spark/delta.md b/docs/sdk/code-reference/pipelines/destinations/spark/delta.md
@@ -0,0 +1,2 @@
+# Write to Delta
+::: src.sdk.python.rtdip_sdk.pipelines.destinations.spark.delta
diff --git a/docs/sdk/code-reference/pipelines/destinations/spark/eventhub.md b/docs/sdk/code-reference/pipelines/destinations/spark/eventhub.md
@@ -0,0 +1,2 @@
+# Write to Eventhub 
+::: src.sdk.python.rtdip_sdk.pipelines.destinations.spark.eventhub
diff --git a/docs/sdk/code-reference/pipelines/secrets/databricks.md b/docs/sdk/code-reference/pipelines/secrets/databricks.md
@@ -0,0 +1,2 @@
+# Databricks Secret Scope
+::: src.sdk.python.rtdip_sdk.pipelines.secrets.databricks
diff --git a/docs/sdk/code-reference/pipelines/sources/spark/autoloader.md b/docs/sdk/code-reference/pipelines/sources/spark/autoloader.md
@@ -0,0 +1,2 @@
+# Read from Autoloader
+::: src.sdk.python.rtdip_sdk.pipelines.sources.spark.autoloader
diff --git a/docs/sdk/code-reference/pipelines/sources/spark/delta.md b/docs/sdk/code-reference/pipelines/sources/spark/delta.md
@@ -0,0 +1,2 @@
+# Read from Delta
+::: src.sdk.python.rtdip_sdk.pipelines.sources.spark.delta
diff --git a/docs/sdk/code-reference/pipelines/sources/spark/delta_sharing.md b/docs/sdk/code-reference/pipelines/sources/spark/delta_sharing.md
@@ -0,0 +1,2 @@
+# Read from Delta sharing
+::: src.sdk.python.rtdip_sdk.pipelines.sources.spark.delta_sharing
diff --git a/docs/sdk/code-reference/pipelines/sources/spark/eventhub.md b/docs/sdk/code-reference/pipelines/sources/spark/eventhub.md
@@ -0,0 +1,2 @@
+# Read from an Eventhub
+::: src.sdk.python.rtdip_sdk.pipelines.sources.spark.eventhub
diff --git a/docs/sdk/code-reference/pipelines/transformers/spark/eventhub.md b/docs/sdk/code-reference/pipelines/transformers/spark/eventhub.md
@@ -0,0 +1,2 @@
+# Read from an Eventhub
+::: src.sdk.python.rtdip_sdk.pipelines.transformers.spark.eventhub
diff --git a/docs/sdk/code-reference/pipelines/utilities/spark/delta_table_create.md b/docs/sdk/code-reference/pipelines/utilities/spark/delta_table_create.md
@@ -0,0 +1 @@
+::: src.sdk.python.rtdip_sdk.pipelines.utilities.spark.delta_table_create
diff --git a/docs/sdk/code-reference/authenticate.md → .../sdk/code-reference/query/authenticate.md b/docs/sdk/code-reference/authenticate.md → .../sdk/code-reference/query/authenticate.md
diff --git a/docs/sdk/code-reference/db-sql-connector.md → .../code-reference/query/db-sql-connector.md b/docs/sdk/code-reference/db-sql-connector.md → .../code-reference/query/db-sql-connector.md
diff --git a/docs/sdk/code-reference/interpolate.md → docs/sdk/code-reference/query/interpolate.md b/docs/sdk/code-reference/interpolate.md → docs/sdk/code-reference/query/interpolate.md
@@ -33,4 +33,4 @@ print(x)
 This example is using [```DefaultAuth()```](authenticate.md) and [```DatabricksSQLConnection()```](db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](authenticate.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](pyodbc-sql-connector.md) or [```TURBODBCSQLConnection()```](turbodbc-sql-connector.md).
 
 !!! note "Note"
-    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../sqlwarehouses/sql-warehouses.md). <br />
+    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../../sqlwarehouses/sql-warehouses.md). <br />
diff --git a/docs/sdk/code-reference/metadata.md → docs/sdk/code-reference/query/metadata.md b/docs/sdk/code-reference/metadata.md → docs/sdk/code-reference/query/metadata.md
@@ -25,4 +25,4 @@ print(x)
 This example is using [```DefaultAuth()```](authenticate.md) and [```DatabricksSQLConnection()```](db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](authenticate.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](pyodbc-sql-connector.md) or [```TURBODBCSQLConnection()```](turbodbc-sql-connector.md).
 
 !!! note "Note"
-    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../sqlwarehouses/sql-warehouses.md). <br />
+    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../../sqlwarehouses/sql-warehouses.md). <br />
diff --git a/...dk/code-reference/pyodbc-sql-connector.md → ...e-reference/query/pyodbc-sql-connector.md b/...dk/code-reference/pyodbc-sql-connector.md → ...e-reference/query/pyodbc-sql-connector.md
diff --git a/docs/sdk/code-reference/raw.md → docs/sdk/code-reference/query/raw.md b/docs/sdk/code-reference/raw.md → docs/sdk/code-reference/query/raw.md
@@ -29,4 +29,4 @@ print(x)
 This example is using [```DefaultAuth()```](authenticate.md) and [```DatabricksSQLConnection()```](db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](authenticate.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](pyodbc-sql-connector.md) or [```TURBODBCSQLConnection()```](turbodbc-sql-connector.md). 
 
 !!! note "Note"
-    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../sqlwarehouses/sql-warehouses.md). <br />
+    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../../sqlwarehouses/sql-warehouses.md). <br />
diff --git a/docs/sdk/code-reference/resample.md → docs/sdk/code-reference/query/resample.md b/docs/sdk/code-reference/resample.md → docs/sdk/code-reference/query/resample.md
@@ -32,4 +32,4 @@ print(x)
 This example is using [```DefaultAuth()```](authenticate.md) and [```DatabricksSQLConnection()```](db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](authenticate.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](pyodbc-sql-connector.md) or [```TURBODBCSQLConnection()```](turbodbc-sql-connector.md).
 
 !!! note "Note"
-    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../sqlwarehouses/sql-warehouses.md). <br />
+    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../../sqlwarehouses/sql-warehouses.md). <br />
diff --git a/...k/code-reference/time-weighted-average.md → ...-reference/query/time-weighted-average.md b/...k/code-reference/time-weighted-average.md → ...-reference/query/time-weighted-average.md
@@ -33,4 +33,4 @@ print(x)
 This example is using [```DefaultAuth()```](authenticate.md) and [```DatabricksSQLConnection()```](db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](authenticate.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](pyodbc-sql-connector.md) or [```TURBODBCSQLConnection()```](turbodbc-sql-connector.md).
 
 !!! note "Note"
-    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../sqlwarehouses/sql-warehouses.md). <br />
+    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../../sqlwarehouses/sql-warehouses.md). <br />
diff --git a/.../code-reference/turbodbc-sql-connector.md → ...reference/query/turbodbc-sql-connector.md b/.../code-reference/turbodbc-sql-connector.md → ...reference/query/turbodbc-sql-connector.md
diff --git a/docs/sdk/overview.md b/docs/sdk/overview.md
@@ -13,15 +13,15 @@ See [RTDIP SDK Usage](rtdip-sdk-usage.md) for more information on how to use the
 
 The RTDIP SDK includes several authentication methods to cater to the preference of the user. See below:
 
-* [Default Authentication](code-reference/authenticate.md) - For authenticating users with Azure AD using the [azure-identity](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) package. Note the order that Default Authentication uses to sign in a user and how it does it in this [documentation](https://azuresdkdocs.blob.core.windows.net/$web/python/azure-identity/1.10.0b1/index.html). From experience, the Visual Studio Code login is the easiest to setup, but the azure cli option is the most reliable option. This [page](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/identity/azure-identity/TROUBLESHOOTING.md#troubleshooting-default-azure-credential-authentication-issues) is useful for troubleshooting issues with this option to authenticate.
+* [Default Authentication](code-reference/query/authenticate.md) - For authenticating users with Azure AD using the [azure-identity](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) package. Note the order that Default Authentication uses to sign in a user and how it does it in this [documentation](https://azuresdkdocs.blob.core.windows.net/$web/python/azure-identity/1.10.0b1/index.html). From experience, the Visual Studio Code login is the easiest to setup, but the azure cli option is the most reliable option. This [page](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/identity/azure-identity/TROUBLESHOOTING.md#troubleshooting-default-azure-credential-authentication-issues) is useful for troubleshooting issues with this option to authenticate.
 
 !!! note "Visual Studio Code"
     As per the guidance in the documentation - **To authenticate in Visual Studio Code, ensure version 0.9.11 or earlier of the Azure Account extension is installed. To track progress toward supporting newer extension versions, see [this GitHub issue](https://github.com/Azure/azure-sdk-for-net/issues/27263). Once installed, open the Command Palette and run the Azure: Sign In command**.
 
 
-* [Certificate Authentication](code-reference/authenticate.md) - Service Principal authentication using a certificate
+* [Certificate Authentication](code-reference/query/authenticate.md) - Service Principal authentication using a certificate
 
-* [Client Secret Authentication](code-reference/authenticate.md) - Service Principal authentication using a client id and client secret
+* [Client Secret Authentication](code-reference/query/authenticate.md) - Service Principal authentication using a client id and client secret
 
 !!! note "Note"
     </b>RTDIP are continuously adding more to this list so check back regularly!<br />
@@ -36,25 +36,25 @@ The RTDIP SDK uses Databricks SQL Warehouses to provide access to RTDIP data. Se
 
 Once the user is authenticated, they can connect to Databricks SQL warehouses using one of the following methods:
 
-* [Databricks SQL Connector](code-reference/db-sql-connector.md) 
+* [Databricks SQL Connector](code-reference/query/db-sql-connector.md) 
 
-* [PYODBC SQL Connector](code-reference/pyodbc-sql-connector.md)
+* [PYODBC SQL Connector](code-reference/query/pyodbc-sql-connector.md)
 
-* [TURBODBC SQL Connector](code-reference/turbodbc-sql-connector.md) 
+* [TURBODBC SQL Connector](code-reference/query/turbodbc-sql-connector.md) 
 
 ## Functions
 
 Finally, once the user is authenticated and connected to Databricks SQL Warehouses they have access the following functions:
 
-* [Resample](code-reference/resample.md) 
+* [Resample](code-reference/query/resample.md) 
 
-* [Interpolate](code-reference/interpolate.md)
+* [Interpolate](code-reference/query/interpolate.md)
 
-* [Raw](code-reference/raw.md)
+* [Raw](code-reference/query/raw.md)
 
-* [Time Weighted Averages](code-reference/time-weighted-average.md)
+* [Time Weighted Averages](code-reference/query/time-weighted-average.md)
 
-* [Metadata](code-reference/metadata.md)
+* [Metadata](code-reference/query/metadata.md)
 
 !!! note "Note"
     </b>RTDIP are continuously adding more to this list so check back regularly!<br />