[Feature] ko/datagateway review

docs/Data-Gateway/Architecture/Infrastructure.md

@@ -1,39 +1,183 @@
-# Infrastructure Diagram
+# Infrastructure
 
-The below is the logical infrastructure interaction of the core Discover engine.
-Each plugin interacts with the specific service configuration that is being interpreted.
+This page describes how Data Gateway is composed and how the core services interact within SHI Cloud.
 
-## Diagram
+---
+
+## Platform topology
 
 ```mermaid
-flowchart BT
-subgraph shiLab["SHI Lab - Trust Boundary"]
-    AppService{{"Data Gateway"}}
-    SqlDb[("General Data<br>(SQL DB)")]
-    BulkData(["Bulk Data<br>(Storage Blob)"])
-    UpdateData(["Update Data<br>(Storage Blob + Table)"])
-end
-subgraph microsoftGlobal["Microsoft Global"]
-    entraId{"Entra ID"}
-    shiLab
-end
-subgraph clientEnvironment["Client Environment"]
-    client[/"REST API Client"\]
-end
-
-AppService --> entraId
-AppService -- CRUD Operation + Token --> SqlDb
-AppService -- CRUD Operation + Token --> BulkData & UpdateData
-BulkData <-- Validate Auth of CRUD Operation --> entraId
-SqlDb <-- Validate Auth of CRUD Operation --> entraId
-entraId -- Public Keys --> AppService
-entraId <-- Log into Entra ID --> client
-AppService -- "CRUD Operation + Token. Response included." --> client
-UpdateData <-- Validate Auth of CRUD Operation --> entraId
+flowchart TD
+
+  %% Entra ID Trust Boundary
+  subgraph CustomerTrustBoundary[Customer Environment]
+    Client(["API Client/SDK"])
+    Users([Users])
+  end
+
+  %% Entra ID Trust Boundary
+  subgraph EntraTrustBoundary[Microsoft Global Cloud]
+    Entra{"Entra ID (IDP)"}
+  end
+
+  %% GitHub Trust Boundary
+  subgraph GitHubTrustBoundary[GitHub Pages]
+    Dashboard((Data Gateway UI))
+  end
+
+  %% SHI Cloud trust boundary
+  subgraph ShiTrustBoundary[SHI Cloud]
+    direction LR
+
+    API{{Data Gateway API}}
+
+    %% Data services
+    subgraph DataStorage[Data Services]
+      direction TB
+      AzSql[("Relational Data (Azure SQL DB)")]
+      GeneralBlob[/"Bulk data (Azure Blob)"\]
+      UpdateBlob[/"Update packages (Azure Blob)"\]
+      AzTable[["Update config (Azure Table)"]]
+      OpenAi(["LLM Service (Azure OpenAI)"])
+    end
+  end
+
+  %% Relationships
+  Users --> | HTTPS | Dashboard
+  Client --> | Public/Secret Client Auth | Entra
+  Entra --> | Auth Code/Access Tokens | Client
+  Client --> API
+  Dashboard --> | Public Client Auth | Entra
+  Entra --> | Auth Code/Access Tokens | Dashboard
+  Dashboard --> | HTTPS + JWT | API
+  API --> | TDS + JWT | AzSql
+  API --> | HTTPS + JWT | GeneralBlob
+  API --> | HTTPS + JWT | UpdateBlob
+  API --> | HTTPS + JWT | AzTable
+  API --> | HTTPS + JWT | OpenAi
 ```
 
-## Threat Model
+!!! note "Key points"
+    - **Identity** - Entra ID is the only identity provider. The UI authenticates users and calls the API with bearer tokens.  
+    - **Data access** - All reads/writes are brokered by the API:  
+        - **Azure SQL Database** for processed relational data  
+        - **Azure Blob Storage** for bulk reports and update packages  
+        - **Azure Table Storage** for update service configuration  
+    - **LicenseGPT** - Chat interactions are transient. Prompts are sent to the API and responses are returned in-session, never stored.
+
+## SHIELD Update Service
+
+Data Gateway controls update delivery using **channels** (e.g., stable, beta, alpha) and **rings** (e.g., ring 0, ring 1). Configuration is stored in **Azure Table Storage**; package files are stored in **Azure Blob Storage**.
+
+### Example Channel Configuration
+
+The below channel configuration demonstrates how the update system can be configured where various tenants are assigned by default to a ring in a channel.
+The tenant's default channel can be overridden by API call. If this happens it defaults to ring 0 on the other channel.
+In the below diagram, the channel is not named, all channels follow the below architecture.
+There can be an unlimited number of rings in a channel. `N...` represents all numbers above 1.
+
+The Alpha channel is RBAC gated and is not available by default. SHI has to approve Alpha access per-tenant.
+
+```mermaid
+flowchart TD
+
+%% Tenant Configs
+CxTenant1[/Tenant 1\]
+CxTenant2[/Tenant 2\]
+CxTenant3[/Tenant 3\]
+DevTenant1[/Dev Tenant 1\]
+DevTenant2[/Dev Tenant 2\]
+
+%% Channel Example
+ subgraph Channel["Channel"]
+    Ring0(("Ring 0<br>Latest"))
+    Ring1(("Ring 1<br>Latest"))
+    RingN(("Ring N...<br>Previous"))
+  end
+
+%% Available Versions
+Versions["Latest: 3.0.0<br>Previous: 2.5.0"]
+
+%% Relationships
+Versions --> Channel
+Ring0 --> CxTenant1
+Ring1 --> CxTenant2
+RingN --> CxTenant3
+Ring0 --> DevTenant1
+Ring0 --> DevTenant2
+```
+
+### Data Flow
+
+The below chart demonstrates how the data storage systems relate to each other and how the configurations flow to Data Gateway.
+
+```mermaid
+flowchart LR
+
+  %% Tenant and API
+  Client([SHIELD])
+  API{{Data Gateway API}}
+
+  %% Configuration in Table Storage
+  subgraph AzureTableStorage[Azure Table Storage]
+    ChannelConfig[["Channels"]]
+    RingConfig[["Rings"]]
+    TenantConfig[["Tenant config"]]
+  end
+
+  %% Packages in Blob
+  PackageStorage[/"Update Packages (Azure Blob)"\]
+
+  %% Relationships
+  Client --> | Request Version | API
+  API --> | Request Config | TenantConfig
+  TenantConfig --> | Default channel, Ring, Alpha allowed | API
+  API --> | Request Config | ChannelConfig
+  ChannelConfig --> | Latest Version, Previous Version | API
+  API --> | Request Config | RingConfig
+  RingConfig --> | Is Latest Available | API
+  API --> | Request Package | PackageStorage
+  PackageStorage --> | Package Stream | API
+  API --> | Version or Package Stream | Client
+```
+
+### Update Selection Process  
+
+1. Tenant defaults define which channel and ring are used, and whether alpha builds are allowed.  
+2. If you explicitly request a channel and it's permitted, the API applies it; otherwise the tenant default applies.  
+3. The ring setting (latest or previous) determines the version within that channel.  
+4. The API returns the resolved version and streams the corresponding package from Blob Storage.
+
+### Data Flow Lifecycle
+
+1. **Authenticate** - The UI authenticates with Entra ID and receives a token.  
+2. **Authorize** - The UI calls the API over HTTPS with the JWT.  
+3. **Process** - The API validates the token and performs reads/writes against SQL, Blob, or Table Storage.  
+4. **Respond** - The API returns data or confirmation.  
+5. **Render** - The UI presents the result to the user. *(LicenseGPT chat is not stored.)*
+
+### Security and Reliability
+
+- **Identity** - Entra ID is the single identity provider.  
+- **Transport** - All calls use HTTPS.  
+- **Isolation** - Clients never access data stores directly; all access is via the API.  
+- **Controlled rollout** - Channels and rings enable predictable, staged updates.  
+- **Resilience** - Operations are designed to handle transient faults.
+
+### Component Map
+
+| Component           | Backing service(s)              | Role                                                                 |
+|---------------------|---------------------------------|----------------------------------------------------------------------|
+| Data Gateway UI     | Web Browser, [GitHub Pages](https://docs.github.com/en/pages) | Entry point for user interactions (Tenant Manager, LicenseGPT)       |
+| Data Gateway API    | HTTPS, [Entra ID auth](https://learn.microsoft.com/en-us/entra/identity-platform/application-model#multitenant-apps)            | Authenticates tokens and brokers access to data stores               |
+| Processed Data      | [Azure SQL Database](https://learn.microsoft.com/en-us/azure/azure-sql/database)              | Relational/reporting data; tenant metadata                           |
+| Bulk report Data    | [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/)              | JSON/report payloads                                                 |
+| Update Packages     | [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/)              | Versioned package archives                                           |
+| Update Config       | [Azure Table Storage](https://learn.microsoft.com/en-us/azure/storage/tables/)             | Channels, rings, and tenant update settings                          |
+| LicenseGPT          | [Azure OpenAI Services](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/overview)           | Provides LLM and Embeddings to the Data Gateway service              |
 
-For an annotated threat model of the application's infrastructure architecture, please see the attached Microsoft [Threat Model Tool](https://aka.ms/tmt) representation of the infrastructure architecture.
+## See also
 
-📄 [Data Gateway - Threat Model](../assets/threat-models/Data-Gateway.tm7)
+- [Usage Guide](../Usage-Guide/index.md)  
+- [Threat Model](../assets/threat-models/Data-Gateway.tm7)  
+- [API Reference](https://specs.shilab.com)

docs/Data-Gateway/Architecture/index.md

@@ -0,0 +1,5 @@
+# Architecture
+
+The Architecture section describes how Data Gateway is structured, including trust boundaries, components, and service interactions.
+
+- [Infrastructure](Infrastructure.md)

docs/Data-Gateway/Deployment/index.md

@@ -0,0 +1,24 @@
+# Deployment
+
+The Data Gateway service does not require installation in your tenant.  
+The service is hosted by SHI and is available directly via the UI and API.
+
+## Permission Grants
+
+The Data Gateway SaaS requires that permissions be granted before you can access the app.
+To grant permissions to a principal in your tenant, please register the `SHI - Data Gateway` app so that SSO is enabled by navigating to the following URL:
+
+- <https://login.microsoftonline.com/common/adminconsent?client_id=4c40281b-a305-4aaf-90a4-d5bbee6eb8ed&redirect_uri=https%3A%2F%2Fdashboard.shilab.com>
+
+!!! note
+    To register this app, you may require the [Cloud Application Administrator](https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/permissions-reference#cloud-application-administrator) role or higher in your tenant depending on the admin consent settings configured.
+
+When the app is registered, you will see an Enterprise App that has the name of `SHI - Data Gateway` with the Client ID of `4c40281b-a305-4aaf-90a4-d5bbee6eb8ed`.
+This enterprise app allows principals in your tenant to be able to generate Access Tokens (JWT) and make API calls to the Data Gateway service.
+This enterprise app does not allow the Data Gateway service to connect into your tenant.
+
+## Getting Started
+
+- Sign in with your Entra ID account.  
+- Navigate the [Usage Guide](../Usage-Guide/index.md) for common tasks.  
+- See [Reference](../Reference/index.md) for API integration details.

docs/Data-Gateway/Reference/index.md

@@ -0,0 +1,76 @@
+# Reference
+
+This page is a hub for the **Data Gateway** reference material: how to authenticate, where to find the live API documentation, and a quick tour of the most-used endpoint families.
+
+## Quick links
+
+<div class="grid cards" markdown>
+
+- :rocket: **Getting Started**  
+  Sign in, navigate the UI, and complete common tasks.  
+  [:octicons-arrow-right-24: Usage Guide](../Usage-Guide/index.md)
+
+- :triangular_ruler: **Architecture**  
+  Trust boundaries, components, and service interactions.  
+  [:octicons-arrow-right-24: Infrastructure](../Architecture/Infrastructure.md)
+
+- :gear: **API Reference (Swagger)**  
+  Browse the live OpenAPI reference and try requests in your browser.  
+  [:octicons-link-24: specs.shilab.com](https://specs.shilab.com)
+
+</div>
+
+## Authentication
+
+Data Gateway uses **Entra ID** (Microsoft identity platform) for authentication.  
+All requests must include a valid **JSON Web Token (Bearer/Access Token)** in the `Authorization` header.
+
+### Steps
+
+1. Sign in with your organization's Entra ID principal to obtain an access token for the Data Gateway application.  
+2. Include the token in each API request:
+
+```bash
+curl -sS https://api.shilab.com/datagateway/status \
+  -H "Authorization: Bearer <token>" \
+  -H "Accept: application/json"
+```
+
+### Notes
+
+- Tokens are validated by the API; users do **not** access SQL or Storage directly.
+- Tokens expire; refresh them using your chosen auth flow (authorization code, client credentials, etc.).  
+- LicenseGPT prompts and responses are **not persisted** - the API returns results to the UI for the current session.
+
+## Endpoint Families
+
+| Family | Purpose | Typical methods | Common paths* |
+|---|---|---|---|
+| **Health & metadata** | Service liveness and basic info | `GET` | `/Api/Core/Health` |
+| **Tenants** | Read and maintain tenant metadata (display name, parent association) | `GET`, `PATCH` | `/Api/Tenant`, `/Api/Tenant/{tenantId}` |
+| **LicenseGPT** | AI-assisted licensing & compliance analysis | `POST` | `/Api/Chat/LicenseGpt` |
+| **Updates (channels & rings)** | Resolve version and retrieve update package | `GET` (and streaming) | See Swagger (“Updates”) |
+
+\* For the complete, authoritative list (parameters, schemas, and responses), use the live reference at **[specs.shilab.com](https://specs.shilab.com)**.
+
+## Request & Response Basics
+
+- **Protocol:** HTTPS only  
+- **Content type:** `application/json; charset=utf-8` (unless explicitly streaming binaries)  
+- **Date/time:** ISO 8601 in UTC unless stated otherwise  
+- **Pagination/filters:** When applicable, filter and paging parameters are documented per endpoint in Swagger
+
+## Error handling
+
+The API uses standard HTTP status codes with JSON error payloads. Please see [MDN - Status Codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status) for more details on specific codes and how they should be interpreted.
+
+!!! note
+    The response body typically includes an explanatory message; consult Swagger for exact schemas.
+
+---
+
+## See also
+
+- [Usage Guide](../Usage-Guide/index.md)  
+- [Infrastructure](../Architecture/Infrastructure.md)  
+- [API Reference (Swagger)](https://specs.shilab.com)

docs/Data-Gateway/Troubleshooting.md

@@ -0,0 +1,5 @@
+# Troubleshooting
+
+🚧 **This section is coming soon.**
+
+Guidance for diagnosing common Data Gateway issues will be published here.