feat(mdk,gate): hostcall

[Yohe-Am]

Introduces a mechanism for wasm materializers to access hostgate functions.

This implements a pretty basic JSON wire interface, a singular hostcall function that's exposed to materializers. The only implemented function on this interface are gql queries.

This is a stacked PR on top of #687.

MET-473.

• The change come with new or modified tests

Summary by CodeRabbit

• New Features

  • Added an import statement for std_url and a new task for installing WASI adapter related files.
  • Introduced new functionalities in the application's runtime to support additional parameters and error handling.

• Enhancements

  • Improved the application's handling of GraphQL queries with new error types and display methods.
  • Enhanced the WASM runtime build process to target a more appropriate architecture.

• Bug Fixes

  • Fixed issues in Python and WASM runtime tests to ensure reliability and performance.

• Documentation

  • Updated internal documentation to reflect new command interfaces and environmental interactions in the application's CLI tools.

• Refactor

  • Refactored various internal APIs to improve code maintainability and efficiency.

[linear]

MET-473 (gate) implement `hostcall`

[zifeo]

lgtm, the simple json protocol is more than enough. Shall we also test that in python and typescript?

[zifeo]

@CodiumAI-Agent /review

[CodiumAI-Agent]

PR Review 🔍

⏱️ Estimated effort to review [1-5]	4, because the PR introduces a complex host call mechanism with multiple layers of error handling and integration across different modules. The changes span multiple files with significant additions, including new functions, error handling, and integration with existing systems. The complexity of the changes, especially around asynchronous operations and error propagation, requires careful review to ensure correctness and maintainability.
🧪 Relevant tests	Yes
⚡ Possible issues	Possible Bug: The error handling in the hostcall function might not correctly propagate errors in all cases. The use of dbg for both logging and error handling could lead to missed error conditions if not carefully managed.
	Error Handling Concern: The hostcall function uses a generic error structure that might obscure the source of errors, making debugging more difficult. Specific error types or more detailed logging might be necessary.
🔒 Security concerns	No

Code feedback:

---

relevant file	typegate/src/runtimes/wit_wire/mod.ts
suggestion	Consider refining the error handling in the hostcall function to ensure that all potential error paths are correctly handled and logged. This could involve separating the concerns of debugging output and error management, possibly by using more specific error types or adding additional error logging at critical points. [important]
relevant line	return dbg(await hostcall(cx, op, json), "success");

---

relevant file	typegate/src/runtimes/wit_wire/mod.ts
suggestion	It's recommended to implement more specific error handling or categorization in the hostcall function to improve maintainability and debuggability. This could involve creating custom error classes that encapsulate different types of errors (e.g., network errors, validation errors) and using these to throw more descriptive exceptions. [important]
relevant line	throw err;

---

relevant file	typegate/src/runtimes/wit_wire/mod.ts
suggestion	To enhance the security and reliability of the hostcall function, consider implementing input validation for the op_name and json parameters. This could prevent potential injection attacks or errors due to malformed input. [important]
relevant line	async function hostcall(cx: HostCallCtx, op_name: string, json: string) {

---

relevant file	typegate/src/runtimes/wit_wire/mod.ts
suggestion	For better code organization and readability, consider refactoring the large hostcall function into smaller, more focused functions. This could involve separating the logic for handling different operation names into separate functions or modules. [medium]
relevant line	async function hostcall(cx: HostCallCtx, op_name: string, json: string) {

---

[coderabbitai]

Walkthrough

The updates encompass a broad range of changes across various files and modules, focusing on version upgrades, functional enhancements, and the introduction of new features. Key adjustments include the implementation of new interfaces and methods related to WASI standards, enhancements in error handling and GraphQL functionalities, and the integration of authentication mechanisms. These changes aim to improve system compatibility, extend functionality, and enhance user and developer interactions with the applications.

Changes

File Path	Change Summary
*.lock.json, wit/deps.toml, typegraph/node/pnpm-lock.yaml	Version updates and dependency management changes.
ghjk.ts	Added import statement for std_url and a new task for installing WASI adapter files.
.../src/runtimes/wit_wire.rs	Addition of new fields, methods, and parameter updates in TypegateHost and related functions.
.../src/runtimes/wit_wire/mod.ts	Added imports, parameter updates, and new functions related to HostCallCtx and gql.
.../tests/runtimes/wasm_wire/rust/build.sh	Updated build script to target wasm32-wasi with optimization changes.
.../tests/runtimes/wasm_wire/rust/lib.rs	Added JSON serialization, registered new handler, and implemented trait for entity generation.
.../tests/runtimes/wasm_wire/rust/mdk.rs	Interface changes, struct modifications, error handling enhancements, and new trait implementations.
.../wasm_wire_test.ts	Added imports for testing utilities and a new test step for GraphQL query response validation.

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 11

Out of diff range and nitpick comments (13)

typegate/engine/runtime.d.ts (1)

58-58: Ensure the callback function is properly documented.

Consider adding TypeScript documentation comments above the cb parameter to explain its purpose, expected behavior, and any side effects it might have. This will improve code readability and maintainability.

typegate/src/runtimes/wit_wire/mod.ts (1)

9-12: Consider removing the debug function in production code.

The dbg function is used for debugging purposes. If this code is meant for production, consider removing such functions or ensuring they are disabled or appropriately secured to avoid verbose logging in production environments.

wit/deps/io/streams.wit (1)

64-69: Ensure consistency in blocking behavior descriptions.

The description of blocking-read could be enhanced by explicitly stating how it differs from read in terms of blocking behavior, even though it mentions "Except for blocking, behavior is identical to read." This could improve clarity for users of the API.

wit/deps/sockets/udp.wit (1)

155-168: Check the behavior of buffer size settings.

The functions set-receive-buffer-size and set-send-buffer-size mention that values might be clamped or rounded, but do not specify how this behavior can be observed or verified by the caller. Adding a method to retrieve the actual set value could enhance usability and transparency.

typegate/engine/src/runtimes/wit_wire.rs (1)

Line range hint 58-64: Consider avoiding unsafe code unless absolutely necessary.

- unsafe {
-   let mut cwasm_bytes = vec![];
-   zstd::stream::copy_decode(&cwasm_zst_bytes[..], &mut cwasm_bytes)
-       .map_err(|err| format!("error decompressing serialized component: {err}"))?;
-   Component::deserialize(&engine, cwasm_bytes)
-       .map_err(|err| format!("error loading pyrt serialized component: {err}"))
- }
+ // Consider refactoring to avoid unsafe code or ensure that all safety invariants are met.

wit/deps/filesystem/types.wit (8)

35-35: Consider updating the documentation to clarify the renaming from filetype to descriptor-type.

---

58-58: Update the documentation to clarify the renaming from fdflags to descriptor-flags.

---

101-101: Update the documentation to clarify the renaming from filestat to descriptor-stat.

---

360-360: Clarify the renaming from fd_filestat_set_size to set-size in the documentation.

---

367-367: Clarify the renaming from fd_filestat_set_times to set-times in the documentation.

---

444-444: Clarify the renaming from fd_filestat_get to stat in the documentation.

---

453-453: Clarify the renaming from path_filestat_get to stat-at in the documentation.

---

466-466: Clarify the renaming from path_filestat_set_times to set-times-at in the documentation.

[coderabbitai]

Actionable comments posted: 1

Out of diff range and nitpick comments (1)

typegate/engine/src/runtimes/wit_wire.rs (1)

Line range hint 65-71: Consider replacing unsafe code with safer alternatives.

- unsafe {
-   let mut cwasm_bytes = vec![];
-   zstd::stream::copy_decode(&cwasm_zst_bytes[..], &mut cwasm_bytes)
-       .map_err(|err| format!("error decompressing serialized component: {err}"))?;
-   Component::deserialize(&engine, cwasm_bytes)
-       .map_err(|err| format!("error loading pyrt serialized component: {err}"))
- }
+ // Suggested to encapsulate unsafe blocks more securely or find a safe alternative.

[codecov]

Codecov Report

Attention: Patch coverage is 46.42857% with 60 lines in your changes are missing coverage. Please review.

Project coverage is 73.43%. Comparing base (1f2f7ce) to head (0478ea5).

Files	Patch %	Lines
typegate/src/runtimes/wit_wire/mod.ts	48.88%	46 Missing ⚠️
typegate/src/runtimes/python.ts	0.00%	7 Missing ⚠️
typegate/src/runtimes/wasm_wire.ts	0.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #706      +/-   ##
==========================================
- Coverage   73.60%   73.43%   -0.18%     
==========================================
  Files         121      121              
  Lines       14187    14295     +108     
  Branches     1416     1420       +4     
==========================================
+ Hits        10443    10497      +54     
- Misses       3717     3771      +54     
  Partials       27       27              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[michael-0acf4]

lgtm so far

[zifeo]

You just earned a new title: the boundary wizard, very nice! 🪄