-
-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unify Type System packages #808
Conversation
Possibly replaces #652 |
493b683
to
f56e541
Compare
Codecov Report
@@ Coverage Diff @@
## main #808 +/- ##
=======================================
Coverage 62.56% 62.56%
=======================================
Files 271 271
Lines 4266 4266
Branches 1004 1004
=======================================
Hits 2669 2669
Misses 1257 1257
Partials 340 340
Flags with carried forward coverage won't be shown. Click here to find out more. Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
69963d3
to
bcd6b5e
Compare
bcd6b5e
to
cd3df86
Compare
bb26c04
to
66163dc
Compare
6f0d4f6
to
8c231a2
Compare
2da12f4
to
8eb6518
Compare
This reverts commit 63f7a50
with: | ||
workspaces: crates/type-system | ||
|
||
- name: Install tools |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- name: Install tools | |
- name: Install Rust tools |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed it to be even more specific
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Alfie!
Itβs a bit of a shame that CI checks are taking 1.5-2 mins longer because of Rust, but we can see how to address this separately.
π What is the purpose of this PR?
Background
Due to quirks of WASM, the Type System package was split into 2 variations, one for use in node environments, and one in web. This was a temporary solution, as it was one we knew would cause problems in the long one. The main pain-point of this approach is when wanting to use the package in a multi-environment setup, such as Next.js which ideally wants a single package it can use in server-side and client-side code. We tried various attempts to unify the packages, defining multiple entrypoints, combining the various outputs of
wasm-pack
targets (web, ESM, etc.)We have settled on this solution for now, largely inspired by the excellent advice of Nick Babcock, and libraries written by him, or linked to by him:
In the future we may move to an ESM only approach, as this removes the need for an
initialization
step due to its integrated support for async imports, but to ensure the package is widely-usable at the moment, we have opted to produce various entrypoints that can be used depending on environment and needs.The unified package
The new package is generated by taking the
web
target ofwasm-pack
and performing another bundling step to produce a common package for (hopefully) all popular JS-based environments. The step is performed byrollup
for similar reasons to those outlined in Nick's post. The bundling process creates a number of targets, which are routed to through theexports
field of thepackage.json
(please consult the source-code for more context).This does mean that all targets now require an async initialization step, whereas previously only the web ones did. (Please see Known Issues below for more information).
As stated, the package has various entry points depending on the consuming environment. These have two different flavours, a fat, and a slim variant. Both variants allow the consumer to pass in a compiled WASM module should they wish (this is important for browser caching benefits and other things). The slim variants specifically do not include inlined versions of the WASM, and therefore are much smaller, but also require the user to acquire a copy of the WASM source from somewhere else. This should allow a single WASM source to be bundled in a complex application (such as an embedding application with many blocks), or for us to host the binary somewhere accessible.
There are a lot of subtleties and reasons for this, which again, are outlined in depth in Nick's post.
π Related links
π« Blocked by
dist
and uses the sameyarn
command layout as other packages, we expect distribution to just work once merged.π What does this change?
@blockprotocol/type-system
packagejest
tests over and modify them to appropriately initialize the WASMπ Does this require a change to the docs?
fetch
.init
function. Not just thebrowser
based ones.index
file which calls theinitialize
logic before exporting the rest of the code. During testing I found this produced issues when the target was imported into environments such as Next.js, which mangles the import name of the WASM asset and moves its location. This could perhaps be rectified by always having the WASM be inlined in this case, or dedicated resources to produce configs for bundler processes such as Next.js. Unfortunately producing these would be a lot of work, the previous solution we discovered did not work with the new bundle in my testing.initialize
, and having inconsistent API exports will be deeply annoying and/or make using the package broken in places like Next.js if the browser import is non-ESM.import wasm from "<module>.wasm"
to use in the slim variants causes problems when trying to compile it. Slim variants are still important and usable if you pass a URL into theinitialize
method, rather than trying to import a WASM object, this has been tested by putting the WASM asset into thepublic
folder of a Next.js app. The error messages arecan't resolve "wbg"
, which there doesn't seem to be a lot of content about online. We have a theory that this is to do with the wasm no-longer sitting alongside its respective JS, and perhaps needs to import from JS or something, but that will require deeper investigation. For now, using embedded strings is more convenient anyway, and as such this can be addressed in the future. Note, this issue only seems to occur when the functions are dealing withJsValue
inputs, (passing to and from JS) which we make heavy use of.πΎ Next steps
π‘ What tests cover this?
jest
tests included within the WASM package. We may want to consider introducing the following environments accompanied by something like Playwright to ensure it's adequately tested:ts-node
scriptβ How to test this?