fix!: increase encoding/decoding performance #58

achingbrain · 2022-08-05T09:47:46Z

Switches the internal serialization/deserialization code to use the Reader/Writer from protobufjs as it's very fast and also the least interesting thing about this module.

% node packages/protons-benchmark/dist/src/encode.js
pbjs x 1,067,457 ops/sec ±1.74% (83 runs sampled)
protons x 1,165,064 ops/sec ±2.78% (90 runs sampled)
protobufjs x 1,317,685 ops/sec ±2.73% (89 runs sampled)
Fastest is protobufjs

% node packages/protons-benchmark/dist/src/decode.js
pbjs x 1,997,186 ops/sec ±0.87% (86 runs sampled)
protons x 1,490,204 ops/sec ±2.40% (90 runs sampled)
protobufjs x 1,809,549 ops/sec ±0.58% (88 runs sampled)
Fastest is pbjs

 % node packages/protons-benchmark/dist/src/index.js 
pbjs x 11,307 ops/sec ±6.04% (86 runs sampled)
protons x 11,376 ops/sec ±2.33% (80 runs sampled)
protobufjs x 12,086 ops/sec ±1.84% (89 runs sampled)
Fastest is protobufjs,pbjs

% node packages/protons-benchmark/dist/src/rpc.js
protons x 3,308,190 ops/sec ±0.57% (92 runs sampled)
protobufjs x 3,135,946 ops/sec ±2.14% (91 runs sampled)
Fastest is protons

BREAKING CHANGE: the exported types of protons-runtime have changed and protobuf encoders/decoders will need to be regenerated

Attempts to optimise encoding behaviour

achingbrain · 2022-08-09T10:14:26Z

Switch tack slightly, incorporate the reader/writer from protobufjs as it's fast, but add native BigInt support from protobufjs/protobuf.js#1557 which means there's no need for the additional long dependency.

Speeds of all are comparable:

% node packages/protons-benchmark/dist/src/encode.js
pbjs x 1,067,457 ops/sec ±1.74% (83 runs sampled)
protons x 1,165,064 ops/sec ±2.78% (90 runs sampled)
protobufjs x 1,317,685 ops/sec ±2.73% (89 runs sampled)
Fastest is protobufjs

% node packages/protons-benchmark/dist/src/decode.js
pbjs x 1,997,186 ops/sec ±0.87% (86 runs sampled)
protons x 1,490,204 ops/sec ±2.40% (90 runs sampled)
protobufjs x 1,809,549 ops/sec ±0.58% (88 runs sampled)
Fastest is pbjs

 % node packages/protons-benchmark/dist/src/index.js 
pbjs x 11,307 ops/sec ±6.04% (86 runs sampled)
protons x 11,376 ops/sec ±2.33% (80 runs sampled)
protobufjs x 12,086 ops/sec ±1.84% (89 runs sampled)
Fastest is protobufjs,pbjs

Weirdly there seems to be a tax associated with instantiating classes defined in ESM vs CJS - the less work your protobuf codec does, the more this tax skews the benchmark. Essentially as the message size increases the performance of protons (ESM) vs protobufjs (CJS) converges, as the message size decreases it diverges. This appears to be a bug in node - see nodejs/node#44186 for more.

## [protons-runtime-v3.0.0](protons-runtime-v2.0.2...protons-runtime-v3.0.0) (2022-08-10) ### ⚠ BREAKING CHANGES * the exported types of `protons-runtime` have changed and protobuf encoders/decoders will need to be regenerated ### Bug Fixes * increase encoding/decoding performance ([#58](#58)) ([9987b97](9987b97))

## [protons-v5.0.0](protons-v4.0.3...protons-v5.0.0) (2022-08-10) ### ⚠ BREAKING CHANGES * the exported types of `protons-runtime` have changed and protobuf encoders/decoders will need to be regenerated ### Bug Fixes * increase encoding/decoding performance ([#58](#58)) ([9987b97](9987b97)) ### Trivial Changes * update sibling deps ([7568283](7568283))

## [3.0.0](v2.0.3...v3.0.0) (2024-01-31) ### ⚠ BREAKING CHANGES * singular fields should be optional to write (#83) * ts definitions will need to be generated from `.proto` files - singular message fields have become optional as message fields are always optional in proto3 * the exported types of `protons-runtime` have changed and protobuf encoders/decoders will need to be regenerated * Uses Uint8ArrayList v2 * This module is now ESM only ### Features * add custom protons options for limiting list/map sizes ([#120](#120)) ([a5ba36b](a5ba36b)), closes [#113](#113) * add strict option to CLI ([#119](#119)) ([8c039c5](8c039c5)) * add support for maps ([#75](#75)) ([e8dfc0a](e8dfc0a)) * define default types during decode ([#62](#62)) ([6453809](6453809)), closes [#43](#43) * runtime size limits for arrays and maps ([#128](#128)) ([a737d05](a737d05)) * support jstype custom options ([#117](#117)) ([ba35475](ba35475)), closes [#112](#112) * support no-copy serialization ([#54](#54)) ([caa0d71](caa0d71)) * transpile to ts ([#17](#17)) ([74d3b7a](74d3b7a)) ### Bug Fixes * add uint8arraylist peer dep ([#61](#61)) ([eb16e86](eb16e86)), closes [#59](#59) * adhere more closely to the language guide for proto3 default values ([#66](#66)) ([406d775](406d775)), closes [#43](#43) * encode enum values ([#30](#30)) ([676c01d](676c01d)) * improve uint64 perf ([#122](#122)) ([3234bb6](3234bb6)) * increase encoding/decoding performance ([#58](#58)) ([9987b97](9987b97)) * only import reader/writer to decrease bundle size ([#69](#69)) ([8eea129](8eea129)) * port protobuf reader/writer to ts ([#60](#60)) ([d101804](d101804)) * remove redundant defs and declare codec return type ([#28](#28)) ([c3ea5ec](c3ea5ec)) * remove writing default values ([#88](#88)) ([078c62f](078c62f)) * single instance codec ([#55](#55)) ([66d9387](66d9387)), closes [#51](#51) * singular fields should be optional to write ([#83](#83)) ([229afbc](229afbc)), closes [#42](#42) * sort imports ([#84](#84)) ([6f796f1](6f796f1)) * support empty messages ([#78](#78)) ([8a02910](8a02910)) * throw when .proto is empty ([#81](#81)) ([ed392cb](ed392cb)) * throw when unsupported fields are detected ([#80](#80)) ([8108875](8108875)), closes [#34](#34) * tidy up formatting of generated code ([#57](#57)) ([387c9e9](387c9e9)) * treat nested enums as enums and not messages ([#87](#87)) ([3af689b](3af689b)) * update aegir, make codec creation dynamic ([#26](#26)) ([ecc46cc](ecc46cc)) * update project config ([3199131](3199131)) * update sibling dep versions ([a77d027](a77d027)) * use CLI flag `--output` ([#46](#46)) ([58dc0ba](58dc0ba)) * use uint8-varint, byte-accesor and longbits modules ([#56](#56)) ([66d72f5](66d72f5)) * use uint8arrays alloc for new buffers ([#123](#123)) ([d1bfc94](d1bfc94)) * write string into output buffer as uint8array ([#118](#118)) ([03ab706](03ab706)) ### Trivial Changes * add or force update .github/workflows/js-test-and-release.yml ([#111](#111)) ([9898b47](9898b47)) * add protobuf-ts to benchmarks ([#68](#68)) ([559191d](559191d)) * benchmark Protobuf-ES ([#89](#89)) ([47a4dcb](47a4dcb)) * delete templates [skip ci] ([#110](#110)) ([196ca52](196ca52)) * fix generated code ([b6a9c18](b6a9c18)) * move bad fixtures to separate directory ([#82](#82)) ([76aa198](76aa198)) * **release:** 1.0.0 [skip ci] ([5aa82f7](5aa82f7)), closes [#17](#17) * **release:** 1.0.1 [skip ci] ([398f066](398f066)), closes [#26](#26) * **release:** 1.0.2 [skip ci] ([193dbfe](193dbfe)), closes [#27](#27) * **release:** 1.0.3 [skip ci] ([6a078c0](6a078c0)), closes [#28](#28) * **release:** 1.0.4 [skip ci] ([ab0d4ac](ab0d4ac)), closes [#30](#30) * **release:** 2.0.0 [skip ci] ([3fec56a](3fec56a)), closes [#54](#54) * **release:** 2.0.1 [skip ci] ([b2d78dd](b2d78dd)) * **release:** 2.0.2 [skip ci] ([8bf01d0](8bf01d0)), closes [#56](#56) * **release:** 3.0.0 [skip ci] ([fd1622d](fd1622d)), closes [#58](#58) * **release:** 3.0.0 [skip ci] ([6fd81c1](6fd81c1)), closes [#17](#17) [#19](#19) * **release:** 3.0.1 [skip ci] ([25e46dd](25e46dd)), closes [#59](#59) * **release:** 3.0.1 [skip ci] ([64fe094](64fe094)), closes [#22](#22) [#21](#21) * **release:** 3.0.2 [skip ci] ([f85d3fb](f85d3fb)), closes [#26](#26) [#27](#27) * **release:** 3.0.3 [skip ci] ([d85a9f4](d85a9f4)), closes [#28](#28) * **release:** 3.0.4 [skip ci] ([9915f7c](9915f7c)), closes [#30](#30) * **release:** 3.0.5 [skip ci] ([9b973ee](9b973ee)), closes [#46](#46) * **release:** 3.1.0 [skip ci] ([d1b93c8](d1b93c8)), closes [#43](#43) * **release:** 4.0.0 [skip ci] ([7a02ec4](7a02ec4)), closes [#43](#43) * **release:** 4.0.0 [skip ci] ([6ec6ef9](6ec6ef9)), closes [#54](#54) * **release:** 4.0.1 [skip ci] ([4f92c90](4f92c90)), closes [#69](#69) * **release:** 4.0.1 [skip ci] ([8991c51](8991c51)), closes [#56](#56) * **release:** 4.0.2 [skip ci] ([3317a6a](3317a6a)), closes [#76](#76) * **release:** 4.0.2 [skip ci] ([411c017](411c017)), closes [#51](#51) * **release:** 4.0.3 [skip ci] ([dab81db](dab81db)), closes [#57](#57) * **release:** 5.0.0 [skip ci] ([63827d9](63827d9)), closes [#83](#83) [#42](#42) [#84](#84) * **release:** 5.0.0 [skip ci] ([98a66fb](98a66fb)), closes [#58](#58) * **release:** 5.0.1 [skip ci] ([bd29083](bd29083)), closes [#104](#104) * **release:** 5.0.2 [skip ci] ([877d54e](877d54e)), closes [#108](#108) * **release:** 5.0.3 [skip ci] ([f29febd](f29febd)), closes [#116](#116) * **release:** 5.0.4 [skip ci] ([17b7a90](17b7a90)), closes [#60](#60) * **release:** 5.0.5 [skip ci] ([2f71170](2f71170)), closes [#118](#118) * **release:** 5.1.0 [skip ci] ([17aaad9](17aaad9)), closes [#112](#112) * **release:** 5.1.0 [skip ci] ([97dca54](97dca54)), closes [#43](#43) * **release:** 5.2.0 [skip ci] ([dfdee5a](dfdee5a)), closes [#113](#113) * **release:** 5.2.1 [skip ci] ([ddf1331](ddf1331)), closes [#126](#126) * **release:** 5.2.2 [skip ci] ([0c4f28e](0c4f28e)), closes [#127](#127) * **release:** 6.0.0 [skip ci] ([dc0de7d](dc0de7d)), closes [#43](#43) * **release:** 6.0.1 [skip ci] ([d5625d2](d5625d2)), closes [#67](#67) [#71](#71) * **release:** 6.0.2 [skip ci] ([26c569d](26c569d)), closes [#76](#76) * **release:** 6.1.0 [skip ci] ([51746ec](51746ec)), closes [#75](#75) * **release:** 6.1.1 [skip ci] ([3ac2c56](3ac2c56)), closes [#78](#78) * **release:** 6.1.2 [skip ci] ([a34a908](a34a908)), closes [#34](#34) * **release:** 6.1.3 [skip ci] ([311b622](311b622)), closes [#81](#81) * **release:** 7.0.0 [skip ci] ([62b2053](62b2053)), closes [#83](#83) [#42](#42) [#84](#84) [#82](#82) * **release:** 7.0.1 [skip ci] ([198e9a7](198e9a7)), closes [#87](#87) * **release:** 7.0.2 [skip ci] ([c7b136e](c7b136e)), closes [#88](#88) * **release:** 7.0.3 [skip ci] ([63eea21](63eea21)), closes [#104](#104) * **release:** 7.0.4 [skip ci] ([655a2f7](655a2f7)), closes [#97](#97) * **release:** 7.0.5 [skip ci] ([5f77393](5f77393)), closes [#108](#108) * **release:** 7.0.6 [skip ci] ([d8c4e6b](d8c4e6b)), closes [#116](#116) * **release:** 7.0.7 [skip ci] ([1d6e843](1d6e843)), closes [#60](#60) * **release:** 7.1.0 [skip ci] ([d5bf315](d5bf315)), closes [#119](#119) * **release:** 7.2.0 [skip ci] ([47359ee](47359ee)), closes [#112](#112) * **release:** 7.2.1 [skip ci] ([cbfe768](cbfe768)) * **release:** 7.3.0 [skip ci] ([23073eb](23073eb)), closes [#113](#113) * **release:** 7.3.1 [skip ci] ([d850acf](d850acf)), closes [#123](#123) * **release:** 7.3.2 [skip ci] ([a6014c6](a6014c6)), closes [#125](#125) * **release:** 7.3.3 [skip ci] ([aa3829c](aa3829c)), closes [#126](#126) * **release:** 7.3.4 [skip ci] ([9f03e47](9f03e47)), closes [#127](#127) * remove long dep ([#22](#22)) ([8795507](8795507)) * remove old example ([#20](#20)) ([00ccc6a](00ccc6a)) * remove redundant dep ([a53620a](a53620a)) * Update .github/workflows/stale.yml [skip ci] ([e86d817](e86d817)) * Update .github/workflows/stale.yml [skip ci] ([8102b91](8102b91)) * Update .github/workflows/stale.yml [skip ci] ([d4f2a98](d4f2a98)) * Update .github/workflows/stale.yml [skip ci] ([aa0601c](aa0601c)) * update project config ([c54b7ac](c54b7ac)) * update protons-runtime dep ([#19](#19)) ([e119076](e119076)) * update sibling dependencies [skip ci] ([a74ff6a](a74ff6a)) * update sibling dependencies [skip ci] ([c9291e0](c9291e0)) * update sibling dependencies [skip ci] ([c476d9d](c476d9d)) * update sibling deps ([7568283](7568283)) ### Documentation * update readme ([#27](#27)) ([0ccb1a3](0ccb1a3)) * update readme to remove breaking character ([#21](#21)) ([023c29b](023c29b)) ### Dependencies * bump aegir from 38.1.8 to 39.0.13 ([#104](#104)) ([912e0e6](912e0e6)) * bump aegir from 39.0.13 to 40.0.8 ([#108](#108)) ([8b54c80](8b54c80)) * bump aegir from 40.0.13 to 41.0.4 ([#116](#116)) ([b95e988](b95e988)) * bump aegir from 41.3.5 to 42.0.1 ([#127](#127)) ([02eafe9](02eafe9)) * bump lerna from 5.6.2 to 6.0.0 ([#70](#70)) ([17fc762](17fc762)) * bump meow from 10.1.5 to 11.0.0 ([#67](#67)) ([d489fd3](d489fd3)) * bump meow from 11.0.0 to 12.0.1 ([#97](#97)) ([33250df](33250df)) * bump meow from 12.1.1 to 13.0.0 ([#125](#125)) ([2b356b8](2b356b8)) * bump uint8arrays from 3.1.1 to 4.0.2 ([#71](#71)) ([b537e92](b537e92)) * bump uint8arrays from 4.0.10 to 5.0.1 ([#126](#126)) ([1e5a0b0](1e5a0b0)) * **dev:** bump lerna from 4.0.0 to 5.3.0 ([#53](#53)) ([eebccc3](eebccc3)) * remove lerna, update aegir ([#76](#76)) ([83a24f2](83a24f2)) * update sibling dependencies ([188704d](188704d)) * update sibling dependencies ([068e25a](068e25a)) * update sibling dependencies ([b1316fa](b1316fa)) * update sibling dependencies ([a7d567d](a7d567d)) ### Tests * regenerate custom options ([768573b](768573b))

achingbrain added 5 commits August 5, 2022 10:40

fix: increase encoding performance

1139377

Attempts to optimise encoding behaviour

chore: remove file

aabf1ed

fix: use list of properties and not map

79aec2e

fix: do not re-encode to get length during decode

75cde26

chore: update test

2f49ef4

fryorcraken mentioned this pull request Aug 8, 2022

Make it easier to swap nwaku for go-waku in tests waku-org/js-waku#763

Merged

achingbrain added 7 commits August 8, 2022 09:47

fix: do not lp encode if not necessary

6ac7d5f

chore: split return types

f6f1eab

fix: do not loop in message encoder

18ee2dd

feat: use protobufjs internals

4073400

chore: add missing dep

db78f65

chore: deps again

3b2451f

chore: range

cfadc93

chore: add benchmark

406c5a9

achingbrain changed the title ~~fix: increase encoding performance~~ fix!: increase encoding/decoding performance Aug 9, 2022

chore: add missing dep

d8f5c8d

achingbrain merged commit 9987b97 into master Aug 10, 2022

achingbrain deleted the fix/encode-performance branch August 10, 2022 06:25

achingbrain mentioned this pull request Aug 10, 2022

Slow compared to protobufjs #51

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix!: increase encoding/decoding performance #58

fix!: increase encoding/decoding performance #58

achingbrain commented Aug 5, 2022 •

edited

Loading

achingbrain commented Aug 9, 2022 •

edited

Loading

fix!: increase encoding/decoding performance #58

fix!: increase encoding/decoding performance #58

Conversation

achingbrain commented Aug 5, 2022 • edited Loading

achingbrain commented Aug 9, 2022 • edited Loading

achingbrain commented Aug 5, 2022 •

edited

Loading

achingbrain commented Aug 9, 2022 •

edited

Loading