overture-stack · joneubank · Aug 30, 2024 · Aug 29, 2024 · Aug 29, 2024 · Aug 29, 2024
diff --git a/README.md b/README.md
@@ -1,8 +1,8 @@
 # Lectern - Data Dictionary Management and Validation
 
-[<img hspace="5" src="https://img.shields.io/badge/chat--with--developers-slack-blue?style=for-the-badge">](http://slack.overture.bio)
+[<img hspace="5" src="https://img.shields.io/badge/chat--with--developers-overture--slack-blue?style=for-the-badge">](http://slack.overture.bio)
 [<img hspace="5" src="https://img.shields.io/badge/License-AGPL--3.0-blue?style=for-the-badge">](https://github.com/overture-stack/lectern/blob/develop/LICENSE)
-[<img hspace="5" src="https://img.shields.io/badge/Code%20of%20Conduct-2.1-blue?style=for-the-badge">](CODE_OF_CONDUCT.md)
+[<img hspace="5" src="https://img.shields.io/badge/Code%20of%20Conduct-blue?style=for-the-badge">](CODE_OF_CONDUCT.md)
 
 
 Lectern is Overture's Data Dictionary Schema Manager, providing a system for defining Schemas that will validate the structured data collected by an application. The core of Lectern is a web-server application that handles storage and version management of data dictionaries. Lectern data dictionaries are collections of  schemas that define the structure of tabular data files (like TSV). This application provides functionality to validate the structure of data dictionaries, maintain a list of dictionary versions, and to compute the difference between dictionary versions.
@@ -36,22 +36,22 @@ The modules in the monorepo are organized into two categories:
    * __packages/__ - Reusable packages shared between applications and other packages. Packages are published to [NPM](https://npmjs.com).
    * __scripts__ - Utility scripts for use within this repo.
 
-| Component                                           | Package Name                       | Path                               | Published Location                                                                                                                                                                                          | Description                                                                                                                                                                                                                                                                                       |
-| --------------------------------------------------- | ---------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [Lectern Server](apps/server/README.md)             | @overture-stack/lectern-server     | apps/server/                       | [![Lectern GHCR Packages](https://img.shields.io/badge/GHCR-lectern-brightgreen?style=for-the-badge&logo=github)](https://github.com/overture-stack/lectern/pkgs/container/lectern)                         | Lectern Server web application.                                                                                                                                                                                                                                                                   |
-| [Lectern Client](packages/client/README.md)         | @overture-stack/lectern-client     | packages/client                    | [![Lectern Client NPM Package](https://img.shields.io/npm/v/@overture-stack/lectern-client?color=%23cb3837&style=for-the-badge&logo=npm)](https://www.npmjs.com/package/@overture-stack/lectern-client)     | TypeScript Client to interact with Lectern Server and Lectern data dictionaries. This library provides a REST client to assist in fetching data from the Lectern server. It also exposes the functionality from the Lectern Validation library to use a Lectern data dictionary to validate data. |
-| [Lectern Dictionary](packages/dictionary/README.md) |                                    | @overture-stack/lectern-dictionary | packages/dictionary/                                                                                                                                                                                        | [![Lectern Client NPM Package](https://img.shields.io/npm/v/@overture-stack/lectern-dictionary?color=%23cb3837&style=for-the-badge&logo=npm)](https://www.npmjs.com/package/@overture-stack/lectern-dictionary)                                                                                   | Dictionary meta-schema definition, includes TS types, and Zod schemas. This also exports all utilities for getting the diff of two dictionaries. |
-| [Lectern Validation](packages/validation/README.md) | @overture-stack/lectern-validation | packages/validation/               | [![Lectern Validation NPM Package](https://img.shields.io/npm/v/@overture-stack/lectern-client?color=%23cb3837&style=for-the-badge&logo=npm)](https://www.npmjs.com/package/@overture-stack/lectern-client) | Validate data using Lectern Dictionaries.                                                                                                                                                                                                                                                         |
+| Component                                           | Package Name                       | Path                               | Published Location                                                                                                                                                                                              | Description                                                                                                                                                                                                                                                                                       |
+| --------------------------------------------------- | ---------------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Lectern Server](apps/server/README.md)             | @overture-stack/lectern-server     | apps/server/                       | [![Lectern GHCR Packages](https://img.shields.io/badge/GHCR-lectern-brightgreen?style=for-the-badge&logo=github)](https://github.com/overture-stack/lectern/pkgs/container/lectern)                             | Lectern Server web application.                                                                                                                                                                                                                                                                   |
+| [Lectern Client](packages/client/README.md)         | @overture-stack/lectern-client     | packages/client                    | [![Lectern Client NPM Package](https://img.shields.io/npm/v/@overture-stack/lectern-client?color=%23cb3837&style=for-the-badge&logo=npm)](https://www.npmjs.com/package/@overture-stack/lectern-client)         | TypeScript Client to interact with Lectern Server and Lectern data dictionaries. This library provides a REST client to assist in fetching data from the Lectern server. It also exposes the functionality from the Lectern Validation library to use a Lectern data dictionary to validate data. |
+| [Lectern Dictionary](packages/dictionary/README.md) |                                    | @overture-stack/lectern-dictionary | [![Lectern Client NPM Package](https://img.shields.io/npm/v/@overture-stack/lectern-dictionary?color=%23cb3837&style=for-the-badge&logo=npm)](https://www.npmjs.com/package/@overture-stack/lectern-dictionary) | Dictionary meta-schema definition, includes TS types, and Zod schemas. This also exports all utilities for getting the diff of two dictionaries.                                                                                                                                                  |
+| [Lectern Validation](packages/validation/README.md) | @overture-stack/lectern-validation | packages/validation/               | [![Lectern Validation NPM Package](https://img.shields.io/npm/v/@overture-stack/lectern-validation?color=%23cb3837&style=for-the-badge&logo=npm)](https://www.npmjs.com/package/@overture-stack/lectern-client)     | Validate data using Lectern Dictionaries.                                                                                                                                                                                                                                                         |
 
 ## Developer Instructions
 
 You can install all dependencies for the entire repo from the root (as defined  the `pnpm-lock.yaml`) with the command:
 
 `pnpm install`
 
-Using `nx` will ensure all local dependencies are built, in the correct sequence, when building, running, or testing any of the applications and packages in the repo. To run a package.json script from any module - after installing dependencies - use a command of the form `pnpm nx <script> <package name>`. For example, to `build` the module `server` can be done with the command:
+Using `nx` will ensure all local dependencies are built, in the correct sequence, when building, running, or testing any of the applications and packages in the repo. To run a package.json script from any module - after installing dependencies - use a command of the form `pnpm nx <script> <package name>`. For example, to `build` the server module `@overture-stack/lectern-server` can be done with the command:
 
-`pnpm nx build server`
+`pnpm nx build @overture-stack/lectern-server`
 
 This will ensure that all dependencies of `server` are built in correct order before the `server` build is run.
 

diff --git a/apps/server/README.md b/apps/server/README.md
@@ -1,16 +1,19 @@
 # Lectern Server
 
-[<img alt="Container Registry Badge" src="https://img.shields.io/badge/GCHR.io-overture--stack%2Flectern-blue?style=for-the-badge&color=blue&cacheSeconds=0" />](https://github.com/overture-stack/lectern/pkgs/container/lectern)
+[<img alt="Container Registry Badge" src="https://img.shields.io/badge/Docker--Image-ghcr.io-blue?style=for-the-badge&color=blue&cacheSeconds=0" />](https://github.com/overture-stack/lectern/pkgs/container/lectern)
+[<img hspace="5" src="https://img.shields.io/badge/chat--with--developers-overture--slack-blue?style=for-the-badge">](http://slack.overture.bio)
+[<img hspace="5" src="https://img.shields.io/badge/License-AGPL--3.0-blue?style=for-the-badge">](https://github.com/overture-stack/lectern/blob/develop/LICENSE)
 
-Lectern Server is the standalone web service for Lectern. It provides an API to create, manage, and share Data Dictionary schemas.
+Lectern Server is a standalone web service that provides a REST API to manage and share Data Dictionary schemas.
 
 ## Technology
 
-Lectern is a NodeJS service written in TypeScript. It is published as a container on ghcr.io: [Lectern Container Registry](https://github.com/overture-stack/lectern/pkgs/container/lectern)
+Lectern Server is a NodeJS service written in TypeScript.
 
-To run the application from source, follow the [Development](#development) instructions below.
+It is published as a container on ghcr.io: [Lectern Container Registry](https://github.com/overture-stack/lectern/pkgs/container/lectern)
 
 ## Development
+
 ### PNPM Monorepo Package Manager
 This project uses `pnpm` instead of `npm` to facilitate a monorepo workspace.
 

diff --git a/apps/server/package.json b/apps/server/package.json
@@ -1,7 +1,7 @@
 {
 	"name": "@overture-stack/lectern-server",
 	"private": true,
-	"version": "2.0.0-next.0",
+	"version": "2.0.0-beta.1",
 	"description": "Overture Data Dictionary Management",
 	"scripts": {
 		"build": "tsc -p tsconfig.build.json",

diff --git a/docs/important-concepts.md b/docs/important-concepts.md
@@ -48,6 +48,11 @@ Placeholder
 
 ## Common Types
 
+### DataRecord and UnprocessedDataRecord
+The `DataRecord` type represents a single record from some Schema. They are objects with keys that match the [fields](#field) from a [schema](#schema) and a value that should be one of the valid Lectern data types. There is no guarantee that a `DataRecord` is "valid", it could have values that fail some restrictions from the schema.
+
+An `UnprocessedDataRecord` is very similar, but all values are raw string values. These represent a single record as it would be submitted in a test file, for example all the data from a single line in a TSV. These string values will need to be [parsed](#parsing) to be converted to their proper types as defined in a schema.
+
 ### TestResult
 
 Reference: [validation/src/types/testResult.ts](../packages/validation/src/types/testResult.ts)
@@ -80,4 +85,18 @@ Example Invalid `TestResult`, for a field validation that failed due to CodeList
 		]
 	}
 }
-```
+```
+
+## Data Processing Verbs
+
+### Parsing
+
+Placeholder
+
+### Validating
+
+Placeholder
+
+### Processing
+
+Placeholder
diff --git a/packages/client/README.md b/packages/client/README.md
@@ -1,21 +1,27 @@
 # Lectern Client
 
-[![NPM Version](https://img.shields.io/npm/v/@overture-stack/lectern-client?color=%23cb3837&style=for-the-badge&logo=npm)](https://www.npmjs.com/package/@overture-stack/lectern-client)
+![TypeScript](https://img.shields.io/badge/TypeScript-007ACC?style=for-the-badge&logo=typescript&logoColor=white)
+[<img hspace="5" src="https://img.shields.io/badge/chat--with--developers-overture--slack-blue?style=for-the-badge">](http://slack.overture.bio)
+[<img hspace="5" src="https://img.shields.io/badge/License-AGPL--3.0-blue?style=for-the-badge">](https://github.com/overture-stack/lectern/blob/develop/LICENSE)
 
-The Lectern client provides developers the mechanism to interact with Lectern servers and to use Lectern Dictionaries. The client provides all the validation logic to check that submitted data is valid based on the structure and restrictions of a Lectern dictionary. It also provides a REST client to fetch Lectern Dictionary data from a Lectern server.
+The Lectern Client provides developers TypeScript code tools to interact with Lectern servers and [Lectern Dictionaries](https://github.com/overture-stack/lectern). This package provides data processing functions that will parse and validate submitted data, ensuring that it adheres to the structure defined by the Dictionary. It also provides a REST client to fetch Lectern Dictionary data from a Lectern Server.
 
 ## Features
-- Interact with lectern servers:
+- REST client to interact with Lectern servers:
   - Fetch dictionary by name and version
   - Fetch difference summaries between dictionary versions
-- Process data using a Lectern Dictionary:
+- [Process data](#data-processing) using a Lectern Dictionary:
   - Convert raw string inputs into properly typed values.
   - Check the structure of input data is valid.
   - Apply all restrictions, both across schemas and on individual fields, to validate input data.
   - Report all validation errors found in the input data.
+- Expose [Lectern Validation](https://www.npmjs.com/package/@overture-stack/lectern-validation) library functionality:
+  - Parsing functions to check and convert data types from string values
+  - Validation functions to confirm the structure and content of records match Lectern schemas
+  - This functionality is combined in the Processing functions
 
-
-## Data Fetching Example
+## Developer Examples
+### Data Fetching
 
 ```ts
 import * as lectern from '@overture-stack/lectern-client';
@@ -25,13 +31,11 @@ const dictionaryName = 'dictionary-name';
 const currentVersion = "2.3";
 const previousVersion = "2.1";
 
-const dictionary = lectern.restClient.fetchSchema(lecternUrl, dictionaryName, currentVersion);
-const versionUpdates = lectern.restClient.fetchDiff(lecternUrl, dictionaryName, currentVersion, previousVersion);
+const dictionary = lectern.rest.fetchSchema(lecternUrl, dictionaryName, currentVersion);
+const versionUpdates = lectern.rest.fetchDiff(lecternUrl, dictionaryName, currentVersion, previousVersion);
 ```
 
-## Data Processing Usage
-
-### Process Data for a Single Schema
+### Data Processing
 
 The following example shows how to process data using the Lectern Client. The input `donorData` is presented as hardcoded, but in a typical scenario this would be submitted to the application through an uploaded TSV, form entry, or similar user submission system.
 
@@ -45,7 +49,7 @@ const dictionary = await getLecternDictionary();
 
 const donorData = [{submitter_donor_id: "abc123", gender: "Male", age: "28"}, {submitter_donor_id: "def456", gender: "Female", age: "37"}]
 
-const schemaProcessingResult = lectern.functions.processSchema(dictionary, "donors", donorData);
+const schemaProcessingResult = lectern.process.processSchema(dictionary, "donors", donorData);
 
 switch (schemaProcessingResult.status) {
 	case 'SUCCESS': {
@@ -63,3 +67,11 @@ switch (schemaProcessingResult.status) {
 		// records were parsed successfully, so this returns all parsed records
 	}
 ```
+
+## Lectern Dependencies
+The Lectern Client is a wrapper around Lectern submodules that allow functionality to be shared between client and server implementations. If you do not need the REST client, or the combined processing functions, you can consider including submodules directly to access the specific pieces of functionality you require.
+
+| Package            | Description                                                                                                                                                                                                                                               |
+| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [lectern-dictionary](https://www.npmjs.com/package/@overture-stack/lectern-dictionary) | Definition of the Lectern Dictionary structure. Includes TS types and schemas for validating the content of Lectern dictionary. Also includes functionality to comparing multiple Lectern Dictionary versions, and to analyze the differences between them. |
+| [lectern-validation](https://www.npmjs.com/package/@overture-stack/lectern-validation) | Provides functionality for validating data against a Lectern dictionary.                                                                                                                                                                                  |
diff --git a/packages/client/package.json b/packages/client/package.json
@@ -1,11 +1,11 @@
 {
 	"name": "@overture-stack/lectern-client",
-	"version": "1.5.0",
+	"version": "2.0.0-beta.1",
 	"files": [
 		"dist/"
 	],
 	"main": "./dist/index.js",
-	"typings": "./dist/index.d.ts",
+	"types": "./dist/index.d.ts",
 	"description": "TypeScript client to interact with Lectern servers and perform data validation versus Lectern dictionaries.",
 	"scripts": {
 		"build": "rimraf dist && tsc -p tsconfig.build.json",

diff --git a/packages/client/src/changeAnalysis/changeAnalyzer.ts b/packages/client/src/changeAnalysis/changeAnalyzer.ts
@@ -17,14 +17,8 @@
  * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  */
 
-import {
-	DictionaryDiff,
-	FieldChanges,
-	RestrictionRange,
-	SchemaField,
-	ValueChange,
-} from '@overture-stack/lectern-dictionary';
-import { restClient } from '../rest';
+import { DictionaryDiff, FieldChanges, SchemaField, ValueChange } from '@overture-stack/lectern-dictionary';
+import * as restClient from '../rest';
 import { ChangeAnalysis, RestrictionChanges } from './changeAnalysisTypes';
 
 const isValueChange = (input: FieldChanges): input is ValueChange => ValueChange.safeParse(input).success;