feat: Split library in to framework and primitives sub-packages (#42

)

* feat: update to monorepo + use turborepo

* feat: add changesets for versioning/publishing

* feat: add linting support

* style: fix styling with eslint

* change near-lake-framwork repo name to prevent it from getting published under a new name

* cache entire dist folder with its contents during build pipline

* remove eslint support + restore prettier config

* Update formatting in turbo.json

Co-authored-by: Morgan McCauley <morgan@mccauley.co.nz>

* add examples app for demo purposes

* add @near-lake/indexer-primitives

* types: convert some interfaces to types

* export only core types needed by near-indexer-framework

* only export types that are needed

* strict ts config to catch errors

* add helper for receipts conversion + clean up

* add helper to convert from state change view to state change type

* add implementation details for methods of the block class

* add indexer-primitives as a dependency to framework + change types

* remove version from indexer-examples repo

* style: change naming + change Array<> to []

* style: consistency with views

* remove accidetnally checked in code

* rename state_changes file + create LakeConfig + export types for near-lake-framework

* feat: make near-lake-framework work with new indexer primitives

* feat: run different indexer examples from root. restructing + simple indexer with new primitives

* fix: do not throw error for logToRawEvent, simply return

* switch to examples package

* formatting: code formatting

* convert block header to class with a static method

* convert Action to a class. Add Static methods for conversions

* Add Static methods for conversions for state changes

* format: make conversion for Block into a static method on Block

* refactor: remove apps/examples/*

* switch events to class type

* fix: fix demo script

* fix: update dependency path for examples

* refactor: use parameter properties approach for receipts, stateChanges, and transactions class

* refactor: use parameter properties approach for Block, Blockheader, and Events class

* refactor: morgan's suggestion on export

* refactor: change Action types into Class

* refactor: change stateChange types into clases

* refactor: events + block structure + morgan's suggestions

* style: update package name

* fix: change package names

* testing release with changeset, hey morgan

* chore: Update `package-lock.json`

* fix: change prepare instruction

* fix: Ensure `primitives` is installed when installing `framework`

* Add README to `/framework` (#29)

* add README

* add dependency explaination

* Update packages/near-lake-framework/README.md

Co-authored-by: Roshaan Siddiqui <siddiqui.roshaan@gmail.com>

* Update packages/near-lake-framework/README.md

Co-authored-by: Roshaan Siddiqui <siddiqui.roshaan@gmail.com>

* Update README.md

Remove Events interface spec

* Update packages/near-lake-framework/README.md

Co-authored-by: Bohdan Khorolets <bogdan@khorolets.com>

* Update packages/near-lake-framework/README.md

Co-authored-by: Bohdan Khorolets <bogdan@khorolets.com>

---------

Co-authored-by: Roshaan Siddiqui <siddiqui.roshaan@gmail.com>
Co-authored-by: Bohdan Khorolets <bogdan@khorolets.com>

* Add README for `@near-lake/primitives` (#30)

* add README

* add Receipt section

* Event section

* add StateChange section

* Update README.md

Remove executed receipts private field

* Update README.md

Remove private explanation from Block

* Apply suggestions from code review

Co-authored-by: Roshaan Siddiqui <siddiqui.roshaan@gmail.com>

* Update README.md

Add Read More on trx section note

* add event interface spec

* Update packages/near-lake-primitives/README.md

Co-authored-by: Bohdan Khorolets <bogdan@khorolets.com>

* Update README.md

* Update README.md

add missing return types

* Update README.md

add return type for from constructor

---------

Co-authored-by: Roshaan Siddiqui <siddiqui.roshaan@gmail.com>
Co-authored-by: Bohdan Khorolets <bogdan@khorolets.com>

* feat: can fetch logs from a block

* Add TypeDocs to lake-primitives (#35)

* typedocs

---------

Co-authored-by: Roshaan Siddiqui <siddiqui.roshaan@gmail.com>

* move docs

* fix: Current version of `turbo` breaks `npm i` in GitHub Actions (#44)

* fix: Current version of `turbo` breaks `npm i` in GitHub Actions

* chore: Remove `postinstall` hook to unblock GH action

* chore: Describe package split in `README.md` (#45)

* fix: Use correct NPM entrypoint for primitives

---------

Co-authored-by: Roshaan Siddiqui <siddiqui.roshaan@gmail.com>
Co-authored-by: Dennis <10233439+idea404@users.noreply.github.com>
Co-authored-by: Bohdan Khorolets <bogdan@khorolets.com>
Co-authored-by: Damián Parrino <bucanero@users.noreply.github.com>

.changeset/README.md

@@ -0,0 +1,8 @@
+# Changesets
+
+Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works
+with multi-package repos, or single-package repos to help you version and publish your code. You can
+find the full documentation for it [in our repository](https://github.com/changesets/changesets)
+
+We have a quick list of common questions to get you started engaging with this project in
+[our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md)

.changeset/brave-goats-talk.md

@@ -0,0 +1,5 @@
+---
+"@near-lake/framework": patch
+---
+
+Specify `@near-lake/primitives` via semver rather than local path to ensure that it is installed when pulling this package from the registry.

.changeset/config.json

@@ -0,0 +1,11 @@
+{
+  "$schema": "https://unpkg.com/@changesets/config@2.3.0/schema.json",
+  "changelog": ["@changesets/changelog-github", { "repo": "near/near-lake-framework-js" }],
+  "commit": false,
+  "fixed": [],
+  "linked": [],
+  "access": "public",
+  "baseBranch": "main",
+  "updateInternalDependencies": "patch",
+  "ignore": []
+}

.github/workflows/ci.yml

@@ -13,6 +13,6 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-node@v3
         with:
-          node-version: 16.x
+          node-version: 18.x
       - run: npm ci
       - run: npm test

.gitignore

@@ -1,7 +1,7 @@
 # See http://help.github.com/ignore-files/ for more about ignoring files.
 
 # compiled output
-/dist
+dist/
 /tmp
 /out-tsc
 
@@ -89,3 +89,5 @@ lerna-debug.log
 # System Files
 .DS_Store
 Thumbs.db
+
+.turbo

README.md

@@ -4,43 +4,44 @@
 
 Available in programming languages: [Rust](https://github.com/near/near-lake-framework) | **Javascript**
 
-**ATTENTION!** This library is in beta and hasn't been tested in production yet.
-
 NEAR Lake Framework is a small library companion to [NEAR Lake](https://github.com/near/near-lake). It allows you to build
 your own indexer that subscribes to the stream of blocks from the NEAR Lake data source and create your own logic to process
 the NEAR Protocol data.
 
----
-
 [Official NEAR Lake Framework launch announcement](https://gov.near.org/t/announcement-near-lake-framework-brand-new-word-in-indexer-building-approach/17668) has been published on the NEAR Gov Forum
-Greetings from the Data Platform Team! We are happy and proud to announce an MVP release of a brand new word in indexer building approach - NEAR Lake Framework.
-
----
 
 ## Example
 
-```typescript
-import { startStream, types } from "near-lake-framework";
+```ts
+import { startStream, types } from '@near-lake/framework';
 
 const lakeConfig: types.LakeConfig = {
-  s3BucketName: "near-lake-data-mainnet",
-  s3RegionName: "eu-central-1",
-  startBlockHeight: 63804051,
+    s3BucketName: 'near-lake-data-mainnet',
+    s3RegionName: 'eu-central-1',
+    startBlockHeight: 66264389,
 };
 
-async function handleStreamerMessage(
-  streamerMessage: types.StreamerMessage
+async function handleBlock(
+    block: types.Block,
 ): Promise<void> {
-  console.log(
-    `Block #${streamerMessage.block.header.height} Shards: ${streamerMessage.shards.length}`
-  );
+    // custom logic for handling the block
+    let events = block.eventsByAccountId("x.paras.near")
+    console.log(events)
 }
 
 (async () => {
-  await startStream(lakeConfig, handleStreamerMessage);
+    await startStream(lakeConfig, handleStreamerMessage);
 })();
 ```
 
+## Packages
+
+NEAR Lake Framework JS originally existed as a single library: [near-lake-framework](https://www.npmjs.com/package/near-lake-framework). This package is still avaiable on NPM and can be used as is. But to make package consumption easier, the library has since been split in to sub-packages:
+- [@near-lake/framework](./packages/near-lake-framework/README.md) - core library for streaming blocks from NEAR Lake
+- [@near-lake/primitives](./packages/near-lake-primitives/README.md) - companion library to help interaction with blockchain data
+
+In most cases you will only need `@near-lake/framework`. For more information on both, see their respective READMEs.
+
 ## Tutorial
 
 Please, read the tutorial [JavaScript NEAR Lake indexer basic tutorial](https://near-indexers.io/tutorials/lake/js-lake-indexer)
@@ -155,3 +156,13 @@ We use Milestones with clearly defined acceptance criteria:
 
 - [x] [MVP](https://github.com/near/near-lake-framework/milestone/1)
 - [ ] [1.0](https://github.com/near/near-lake-framework/milestone/2)
+
+## Running examples
+
+Inside `apps/` we have created a created a bunch of example implementations for using the indexer. 
+
+To test out any of the indexers, from the root, run:
+
+```bash
+$ npm run demo --name=<example_package_name>
+```