feat: v2 - add store, world and schema-type, cli table code generation #422

holic · 2023-02-20T14:21:17Z

This branch was used for fast iteration on v2 packages. However, maintaining two separate branches (main and v2) causes unnecessary overhead, and the v2 changes are mostly self-contained, so they can live side by side with the v1 packages on main.

* feat(solecs): wip prototype for new data model core library * feat(solecs): wip data model - add getData and split * feat(solecs): wip data model - add StoreSwitch lib * feat(solecs): wip data model - add Vec2Table and tests * feat(solecs): wip data model - improve gas for slice/get * feat(solecs): wip data model - add SystemTable and tests * feat(solecs): wip data model - add World and end-to-end tests * refactor(solecs): wip data model - rename schemas and tables * refactor(solecs): wip data model - convert bytes without copying * feat(solecs): wip data model - add more efficient packing methods * refactor(solecs): wip data model - change function names and schema representation * feat(solecs): wip data model - add logic to set partial word * feat(solecs): wip data model - add low level buffer library * feat(solecs): wip data model - add low level storage library * refactor(solecs): wip data model - use Storage lib in StoreCore * refactor(solecs): wip data model - change schema encoding and naming * feat(solecs): wip data model - add logic and tests to set and get dynamic data length * feat(solecs): wip data model - add logic and tests to read and write dynamic data * feat(solecs): wip data model - add logic and test to set and get individual fields * test(solecs): wip data model - add fuzzy test for Storage lib * test(solecs): wip data model - add fuzzy tests for Buffer and Bytes * feat(solecs): wip data model - add validity checks for registering schemas * feat(solecs): wip data model - move schema and packed counter to own library/types * refactor(solecs): wip data model - add different events for setting record, field and delete * refactor(solecs): wip data model - make encoded dynamic lengths part of the bytes blob * test(solecs): wip data model - add mixed table with dynamic data schema * feat(cli): add gas report command * test(solecs): wip data model - refactor buffer and bytes tests to use mud gas report * test(solecs): wip data model - add test for gas cost of custom encoding vs abi encode * feat(solecs): wip data model - add CallbackArray schema and tests, refactor typed lib definitions * feat(solecs): wip data model - add indexer functionality * feat(solecs): wip data model - add registerOnUpdateHook to interface * feat(store): move store core source into own package outside of solecs * build(store): add -rf to dist command to override existing files * test(store): generate gasreport * build(store): fix CI, add gas report action * build(store): update gas report action name * refactor(store): use setRecord instead of setStaticData * build(store): add mud cli as dev dependency * build(store): update forge-std and ds-test dependencies * chore(store): use forge-std Test, remove ds-test dependency, fix gas-report command * build(store): bring back ds-test bc forge-std actually needs it * build: use local mud cli in gasreport action * build: gas report action - install cli dependencies before building * build: gas report action - move cli linking action after general building * build: gas report action - force gh to use local cli * fix(store): remove sneaked in temp test file * build(store): fix push for gas report action * feat(store): add onDelete hook, rename IOnUpdateHooks to IStoreHooks, make naming consistent * build(store): add explicit commit author for gas report action * refactor(store): prefix store events with Mud * build: fix gasreport detached head * refactor(store): rename IStoreHooks to IStoreHook * refactor(store): rename Schema_ to SchemaLib, PackedCounter_ to PackedCounterLib * build(store): different way to solve gas report action detached head issue * refactor(store): more consistency for error naming * build: fix gas report action * refactor: rename gas report action * refactor: hopefully actually fix gas report action * refactor(store): move StoreCore internal functions to StoreCoreInternal * refactor(store): rename write to store and read to load in Storage and Memory * build: another attempt at fixing the gas report action * test: update gas report * refactor(store): move test out of src * test: update gas report * refactor(store): use named params for Storage and Memory lib calls, reorder params * test: update gas report * refactor(store): use abi.encodePacked for fixed length types instead of bytes.concat * test: update gas report * refactor(core): change argument name of registerHook * fix(cli): support gas-report with lines including quotes * chore: remove unnecessary conversion methods for primitive types * chore(store): remove unused code * refactor: use uint256 in PackedCounter accumulator * chore: change setting of schema length bytes * fix: don't reset dynamic data when deleting storage value but only length * chore: update comments * chore: move _getStaticData into the correct section * chore: remove misleading comment * chore: remove comment * chore: remove comment * test: update gas report --------- Co-authored-by: alvrs <alvrs@users.noreply.github.com>

* fix(store): emit store events before early return * test(store): add tests for emitting events

…a, register SchemaTable's schema (#383) * fix(store): emit MudStoreSetRecord event when registering a new schema * fix(store): register SchemaTable's schema * refactor(store): reuse StoreCore events/errors in StoreCoreInternal

…re flag (#394)

…387) * feat(store): replace Buffer with Slice and reorganize custom coders * chore(store): fix comment * test: update gas report * refactor(store): rename ABICoder to StorageCoder * test(store): add 1024x1024 memory copy gas test * refactor(store): rename tests for storagecoder * test: update gas report * refactor(store): rename Slice_ to SliceLib * test: update gas report * refactor(store): expand some short names * refactor(store): remove 2 unnecessary typecasts in StoreCore * refactor(store): clarify _encodeToLocation * refactor(store): fix typo Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com> * refactor(store): fix typo Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com> * test: update gas report * refactor(store): rename StorageCoder to TightCoder * refactor(store): change arguments order in Memory.copy --------- Co-authored-by: dk1a <dk1a@users.noreply.github.com> Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com>

Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com> Co-authored-by: dk1a <dk1a@users.noreply.github.com>

* feat(schema-type): create schema-type package, add solidity and typescript * refactor(schema-type): add more warnings * feat(schema-type): add DynamicSchemaType

* feat(cli): refactor to esm * feat(services): make services more compatible with esm * build(cli): add clarifications to tsup config * fix(cli): fix mud.ts shebang * refactor(cli): fix minor eslint problems * chore(cli): add lint to scripts * feat(cli): use moduleResolution nodenext when testing tsc * fix(create-mud): add missing peerDependency * feat(cli): remove mud create, deprecated in #424 * fix: resolve indirect conflicts * refactor(cli): add separate test:next to avoid test failures in CI * fix(store): use released cli for gas-report * test: update gas report --------- Co-authored-by: dk1a <dk1a@users.noreply.github.com>

--------- Co-authored-by: alvrs <alvrs@users.noreply.github.com>

* build(store): add store config to store for testing * feat(cli): add store config and basic schema generator * refactor(store): use some autogenerated schemas * refactor(cli): simplify shebang * build: update some typescript versions * refactor(store): use local mud.js for tablegen * build(cli): change dependencies * feat(cli): use fancy custom errors for config * feat(cli): use tagged template for conditional rendering * refactor(cli): rename config to storeConfig * chore(cli): add tablegen description Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com> * feat(cli): separate config logically, improve storeConfig validation * fix(cli): make shebang work on mac --------- Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com>

* refactor(cli): rename typeWithLocation * feat(cli): add routes, schemaMode (unimplemented) to config, improve validation * feat(cli): implement schemaMode flag * refactor(store): use new tablegen in store * refactor(cli): improve some rendering utils * feat(cli): add support for singleton tables * feat(cli): autoformat tablegen output * refactor(cli): settle on pure ESM config, add .ts support * refactor(cli): use zod for StoreUserConfig validation * chore(cli): remove outdated comment about hardhat * build(cli): clean up dependencies * refactor(cli): improve errors * refactor(cli): add validation error customization * feat(cli): add component mode * fix(cli): add NotESMConfigError and fix esnext resolution issues * build(cli): switch to nodenext resolution * build(cli): add dev script for tsup --watch * refactor(cli): extract common zod schemas * feat(cli): add component mode shorthand * fix(cli): report and continue for errors during formatting * fix(cli): fix commas for combined singleton and schema modes * refactor(cli): update some old conditionals with better syntax * feat(cli): improve struct and library naming * refactor(cli): improve config arguments * feat(cli): support no struct for multicolumn tables * refactor(cli): rename renderSchema to renderTable * feat(cli): improve code structure for renderTable * fix(cli): make dir if absent * feat(cli): generate push for dynamic fields * feat(store): make all tables autogenerated * feat(cli): generate deleteRecord * feat(cli): add primary key typing * refactor(store): use new tablegen * fix(cli): add missed bool cast * fix(cli): always render deleteRecord * feat(cli): add storeArgument option * refactor(store): update store tables * feat(world): use tablegen for RouteAccess * fix(cli): missing bracket for int typecast * chore(cli): change route doc comment Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com> * chore(cli): fix typo 1 Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com> * chore(cli): fix typo 2 Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com> * chore(cli): remove outdated comment --------- Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com>

alvrs · 2023-02-28T00:14:14Z

.github/workflows/gasreport.yml

+      - name: Install local CLI
+        run: cd packages/cli && yarn build && npm link && cd ../..
+
+      - name: Verify that gas report is up to date


TODO: fix gas report action (right now it just passes even if the gas report is out of date)

alvrs · 2023-02-28T00:39:38Z

packages/cli/src/render-table/field.ts

+    }
+    `;
+
+    // TODO: this is super inefficient right now, need to add support for pushing to arrays to the store core library to avoid reading/writing the entire array


link TODO with PR / issue

added to #438

alvrs · 2023-02-28T00:40:23Z

packages/cli/src/render-table/record.ts

+    ${renderEncodedLengths(dynamicFields)}
+    bytes memory _data = abi.encodePacked(${renderArguments([
+      renderArguments(staticFields.map(({ name }) => name)),
+      // TODO try gas optimization (preallocate for all, encodePacked statics, and direct encode dynamics)


link TODO with issue

added to #444

moved to #1132

alvrs · 2023-02-28T00:41:44Z

packages/cli/src/utils/errors.ts

+  if (error instanceof ValidationError) {
+    console.log(chalk.redBright(error.message));
+  } else if (error instanceof ZodError) {
+    // TODO currently this error shouldn't happen, use `fromZodErrorCustom`


link TODO with issue

added to #438

alvrs · 2023-02-28T00:41:54Z

packages/cli/src/utils/errors.ts

+  } else if (error instanceof NotInsideProjectError) {
+    console.log(chalk.red(error.message));
+    console.log("");
+    console.log(chalk.blue(`To learn more about MUD's configuration, please go to [TODO link to docs]`));


link TODO with issue

updated the link to https://mud.dev/packages/cli/ for now and added an issue to add docs for this here: #445

alvrs · 2023-02-28T00:56:36Z

packages/store/tsconfig.json

@@ -0,0 +1,102 @@
+{
+  "compilerOptions": {
+    /* Visit https://aka.ms/tsconfig.json to read more about this file */


remove comments for readability

alvrs · 2023-02-28T00:57:49Z

packages/world/gas-report.txt

@@ -0,0 +1,13 @@
+(test/World.t.sol) | validate route (empty string) [assertTrue(_isRoute(""), "empty string")]: 276


make sure this is up to date

alvrs · 2023-02-28T00:59:10Z

packages/world/src/World.sol

+   * Register a hook for a given system route
+   */
+  function registerSystemHook(string calldata systemRoute, ISystemHook hook) public {
+    // TODO


link TODO with issue

Added to #444

alvrs · 2023-02-28T01:00:50Z

packages/world/test/World.t.sol

+    world.call("/testRoute/testSystem", abi.encodeWithSelector(WorldTestSystem.emitCallType.selector));
+  }
+
+  // TODO: add a test for systems writing to tables via the World


link TODO with issue

Added to #444

alvrs · 2023-02-28T01:01:10Z

packages/world/tsconfig.json

@@ -0,0 +1,102 @@
+{
+  "compilerOptions": {
+    /* Visit https://aka.ms/tsconfig.json to read more about this file */


remove comments for readability

alvrs and others added 18 commits February 7, 2023 10:33

ci: add v2 to github test action

e5d1195

fix(store): emit store events before early return (#381)

3b58a1f

* fix(store): emit store events before early return * test(store): add tests for emitting events

refactor(store): optimize storage library (#386)

217e817

Merge remote-tracking branch 'origin/main' into v2

4e04c00

test(store): avoid fuzzy storage slot clash with test assertion failu…

3a3a18a

…re flag (#394)

refactor(store): rename events (#398)

3c7fb35

feat(store): add missing schema types (#395)

25fc99f

refactor(store): change tableId type from bytes32 to uint256 (#416)

a7adbbb

chore: merge main into v2 (#430)

7cce250

feat(store): add codegen for all types for tightcoder (#396)

b686e08

Co-authored-by: alvarius <89248902+alvrs@users.noreply.github.com> Co-authored-by: dk1a <dk1a@users.noreply.github.com>

feat(schema-type): create schema-type package (#429)

32a56f7

* feat(schema-type): create schema-type package, add solidity and typescript * refactor(schema-type): add more warnings * feat(schema-type): add DynamicSchemaType

feat(world): add World prototype (#405)

c4ea97b

--------- Co-authored-by: alvrs <alvrs@users.noreply.github.com>

alvrs reviewed Feb 28, 2023

View reviewed changes

This was referenced Feb 28, 2023

Fix gas-report action #442

Closed

Tracker: v2 tablegen todos #438

Open

Tracker: v2 gas optimization todos #443

Closed

Tracker: v2 todos #444

Open

Improve CLI readability by using execa instead of execLog #446

Open

alvrs and others added 2 commits February 27, 2023 22:48

chore: v2 polishes

7534096

Merge branch 'main' into v2

5aa602d

alvrs changed the title v2 feat: add store, world and schema-type packages, update cli to esm and add table generation Feb 28, 2023

alvrs marked this pull request as ready for review February 28, 2023 03:55

alvrs requested a review from a user February 28, 2023 03:55

alvrs self-assigned this Feb 28, 2023

alvrs approved these changes Feb 28, 2023

View reviewed changes

alvrs changed the title ~~feat: add store, world and schema-type packages, update cli to esm and add table generation~~ feat: v2 - add store, world and schema-type packages, update cli to esm and add table generation Feb 28, 2023

alvrs changed the title ~~feat: v2 - add store, world and schema-type packages, update cli to esm and add table generation~~ feat: v2 - add store, world and schema-type, cli table code generation Feb 28, 2023

alvrs merged commit cb731e0 into main Feb 28, 2023

alvrs deleted the v2 branch May 18, 2023 12:53

github-actions bot mentioned this pull request Jul 4, 2023

chore: release 2.0.0-next.0 #1103

Merged

dk1a mentioned this pull request Jul 10, 2023

Tracker: TightCoder #1132

Closed

7 tasks

This was referenced Jul 10, 2023

Turn SchemaTable into an codegen table instead of a special case #1134

Closed

Store gas metrics and optimizations #1133

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: v2 - add store, world and schema-type, cli table code generation #422

feat: v2 - add store, world and schema-type, cli table code generation #422

holic commented Feb 20, 2023 •

edited by alvrs

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

dk1a Jul 10, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

alvrs Feb 28, 2023

		@@ -0,0 +1,13 @@
		(test/World.t.sol) \| validate route (empty string) [assertTrue(_isRoute(""), "empty string")]: 276

feat: v2 - add store, world and schema-type, cli table code generation #422

feat: v2 - add store, world and schema-type, cli table code generation #422

Conversation

holic commented Feb 20, 2023 • edited by alvrs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holic commented Feb 20, 2023 •

edited by alvrs