Skip to content

mgrman/BlazorWith3d

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

278 Commits
.config
.config
 
 
.github/workflows
.github/workflows
 
 
samples
samples
 
 
src/BlazorWith3d
src/BlazorWith3d
 
 
.editorconfig
.editorconfig
 
 
.gitattributes
.gitattributes
 
 
.gitignore
.gitignore
 
 
BlazorWith3d.sln
BlazorWith3d.sln
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
global.json
global.json
 
 

Repository files navigation

BlazorWith3d

Demo

This repo showcases using source generators to improve JS and Unity interop with Blazor.

It is in alpha stage only (available only as repo, as manual adjustments for each project are expected), ie there is still cleanup, better separation and fixing bugs and edge cases needed!

With the intent of adding 3D rendering capability to Blazor, which can then be developed using native technology (Typescript or Unity) and source generator handles interop.

Mainly by generating TypeScript types (for ThreeJS and BabylonJS renderers), and setting up automatic implementation of interface methods to be routed through a binary interface (e.g. JS Interop to get data to Javascript world and then back to Unity).

So that developers do not have to add each method to the specific way interop is being handled, but only define interfaces and all boilerplate code is generated by the Source Generator.

Building upon similar ideas as my CodeFirstApi repo of using interfaces in shared library to generate the APIs.

Examples

Blazor to ThreeJS/BabylonJS

Instead of:

  • creating the TypeScript types manually that need to match the C# types
  • Marking each method in C# that should be callable from JS with attribute
  • Manually ensuring the TypeScript methods that the C# code expects are there and implemented
// Might be less initial boiler plate but then needs manual definition for each method and synchronizing the C# code and TS code

var module = await _jsRuntime.LoadModuleAsync("module.js");
_messageReceiverProxyReference = DotNetObjectReference.Create(new MessageProxy(OnRendererInitialized, ...));
_typescriptApp = await module.InvokeAsync<IJSObjectReference>(initMethod, _containerElementReference,_messageReceiverProxyReference,nameof(BinaryApiJsMessageReceiverProxy.OnRendererInitialized), ...);

public ValueTask InitializeRenderer(RendererInitializationInfo msg)
{
    return _typescriptApp.InvokeVoidAsync("InitializeRenderer", msg);
}
...

[JSInvokable]
public ValueTask OnRendererInitialized(RendererInitialized msg)
{
    return _eventHandler.OnRendererInitialized(msg, this);
}
...

You can:

  • define one or many interfaces in a shared library
  • Generated TS representation of the interface and all referenced types
  • Either (see BlocksOnGridThreeJSRenderer.cs):
    • the generator uses prepared methods for generic Binary API from TypeScript to Blazor (that can send binary array there and back)
      • the generator creates wrapper for each method of the interface using this Binary channel
      • and using MemoryPack as (for now hard coded) serialization method optimizes serialization of messages
    • and ensuring that any changes in the interfaces have to be taken into account in both Unity and Blazor codebases
  • Or (see BlocksOnGridBabylonDirectRenderer.cs):
    • Generating wrapper so that each method in C# that can be callable from JS using Blazor native JS interop
    • and that each interface method maps to a JS method in the generated interface
// needs to be initialized, but then interface implementations are directly callable, and TypeScript code is generated to match the C# code
IBlocksOnGrid3DController _eventHandler; // initialized to handler you want to handle any events

var binaryApi = new JsBinaryApiWithResponseRenderer(_jsRuntime, _logger); // prepare binaryAPI communication channel
await binaryApi.InitializeJsApp("module.js", _containerElementReference); // connect the binaryAPI to JS 
appApi = new BlocksOnGrid3DRendererOverBinaryApi(binaryApi, new MemoryPackBinaryApiSerializer(), new PoolingArrayBufferWriterFactory(), _eventHandler); // initialize app specific code
appApi.InitializeRenderer(new()); // invoke methods that are mapped to events on TypeScript side

Unity to Blazor interop

Instead of:

  • manually adding each method as Unity to JS method and JS to Blazor method
  • with added need for manually serializing the data as only primitive types are supported (due to WASM layer between Unity C# code and the JS world)
  • maintaining any change to these method signatures
  • Compiling Unity project into WebAssembly on each change of the Unity codebase
// Might be less initial boiler plate but then needs manual definition for each method and synchronizing the C# code and TS code

var module = await _jsRuntime.LoadModuleAsync("module.js");
_messageReceiverProxyReference = DotNetObjectReference.Create(new MessageProxy(OnRendererInitialized, ...));
_typescriptApp = await module.InvokeAsync<IJSObjectReference>(_initMethod, _containerElementReference,_messageReceiverProxyReference,nameof(BinaryApiJsMessageReceiverProxy.OnRendererInitialized), ...);

public ValueTask InitializeRenderer(RendererInitializationInfo msg)
{
    return _typescriptApp.InvokeVoidAsync("InitializeRenderer", msg);
}
...

[JSInvokable]
public ValueTask OnRendererInitialized(RendererInitialized msg)
{
    return _eventHandler.OnRendererInitialized(msg, this);
}

// JS code to connect Unity instance to Blazor and pass each message along to the right interop method 

// And on Unity side (for each method):
[DllImport("__Internal")]
private static extern string _InitializeEventHandlers(Action<int, int>? initializeRenderer, ...);
 

[MonoPInvokeCallback(typeof(Action<int, int>))]
private static void _InitializeRenderer(int size, int id)
{
}
...
    
[DllImport("__Internal")]
private static extern int _OnRendererInitialized();
...

You can:

  • define one or many interfaces in a shared library
  • the generator uses prepared methods for generic Binary API (that can send binary array there and back between Blazor and Unity)
    • the generator creates wrapper for each method of the interface using this Binary channel
    • and using defined serialization methods (e.g. MemoryPack being faster that JSON)
  • and ensuring that any changes in the interfaces have to be taken into account in both Unity and Blazor codebases
  • with added benefit of using Websockets as BinaryAPI and connecting to Unity Editor in dev builds (to have faster iteration time) 
IBlocksOnGrid3DController eventHandler;// initialized to handler you want to handle any events

binaryApi = new JsBinaryApiWithResponseRenderer(_jsRuntime, _logger); // prepare binaryAPI communication channel
await _binaryApi.InitializeJsApp("module.js", _containerElementReference); // connect the binaryAPI to JS 
appApi = new BlocksOnGrid3DRendererOverBinaryApi(binaryApi, new MemoryPackBinaryApiSerializer(), new PoolingArrayBufferWriterFactory(), eventHandler); // initialize app specific code
appApi.InitializeRenderer(new()); // invoke methods that are mapped to events on Unity side

// a generic binary two way channel is prepared in included JS files. 

// and Unity side (the methods and events are enforced by the interfaces and generators):

var eventHandler =...; // initialized to handler you want to handle any events

var binaryApi = UnityBlazorApi.Singleton; //  prepares binaryAPI communication channel and connects it to JS 
var controller = new BlocksOnGrid3DControllerOverBinaryApi(binaryApi, new MemoryPackBinaryApiSerializer(), new PoolingArrayBufferWriterFactory(), eventHandler); // initialize app specific code
await controller.OnRendererInitialized(new()); // invoke methods that are mapped to events on Blazor side

Blazor to Blazor

  • By creating abstraction via an Interface
  • And automatically handling all specifics
  • you can then easily create other implementaions of the interface using other approaches or Blazor itself
    • e.g. to create HTML only fallback mode in case GPU is not available on the client machine for performant 3D rendering
  • See BlocksOnGridHTMLComponent.razor

Projects

  • 4 Source Generators:

    • BlazorWith3d.CodeGenerator.BlazorBinding:
      • to generate Blazor JS interop classes to set up the Attributes needed to map JS methods to C# methods, and expose C# methods as JSInvokable methods.
    • BlazorWith3d.CodeGenerator.TypeScript:
      • to generate Typescript interfaces based on C# interfaces
      • to generate appropriate JS code to invoke the JSInvokable C# methods exposed in BlazorBinding generator
    • BlazorWith3d.CodeGenerator.DotnetBinaryApiBinding:
      • to generate implementations of marked interfaces that implement methods via messages over a two-way binary API in C# (for communication between Blazor and Unity, e.g. via Interop or Websocket connection to Unity Editor)
    • BlazorWith3d.CodeGenerator.MemoryPack:
      • to generate implementation of marked interfaces that implement methods via messages over a two-way binary API in TypeScript (for now tightly bound to using MemoryPack as serializer for binary messages)
      • to supplement MemoryPack's TypeScript generation capabilities with adding TypeScript class generation for Sequential Structs in C# (for use as serializer in code generated in DotnetBinaryApiBinding)

  • Blazor libraries

    • BlazorWith3d.Shared.Blazor
      • Shared library with Blazor specific types (items that are not specific to JS nor Unity renderers)
    • BlazorWith3d.Unity
      • Blazor specific code for creating components that use Unity Web and communicate over BinaryApi
    • BlazorWith3d.JsRenderer
      • Blazor specific code for creating components that use JS library and communicate over BinaryApi or native Blazor JS interop

  • .NET/Unity Shared libraries

    • BlazorWith3d.Shared
      • Shared library with types to define types and interfaces for creating BinaryApis
      • The same code is exposed as Unity package

  • Unity packages

    • BlazorWith3d.Unity.UnityPackage
      • Unity specific implementation of Binary API over JS interop
      • Expose connection WebSocket connection from Unity Editor for debugging/development purposes to be able to connect to Unity Editor from Blazor app to improve iteration speed
      • Contains binaries of the BlazorWith3d.CodeGenerator.DotnetBinaryApiBinding generator annotated in a way to be usable in Unity
        • Include Marker.BlazorWith3d.CodeGenerator.DotnetBinaryApiBinding.additionalfile file in an asmdef you want the generator to be used in Unity

Example app

Simple app for placing and moving blocks on grid. To test renderers and interop capabilities.

Demo

Supports using multiple renderers at the same time.

Main projects:

  • BlazorWith3d.ExampleApp.Client.Shared project contains the shared code definitions used by the generators.
  • BlazorWith3d.ExampleApp.Client project the main app itself.
  • BlazorWith3d.ExampleApp.AspNet project is the ASP.NET app that is used to run it.

Findings

Native Blazor interop is sufficient for most cases of JS libraries (but some binary approaches can still be faster than underling JSON)

Generators can be used to create the binding classes based on an interface. After initial set up, quick nice to work with and extend.

Simple method/event style interfaces can be implemented even with stream only. useful to get websocket connection to Unity Editr as it is slower than other approaches.

Coordinate systems between different rendering libraries can be a pain. Choose carefully, but conversions are always doable.

This repo shows numerous approaches, but in the end, one or two are probably enough for your app.

The level of abstraction how much to do in 3d specific library vs in main app should depend on use-case and the amount of code that should be shared. e.g. if only one renderer then easier to do more in the renderer. If more are expected than moving more to main Blazor app can be beneficial

This repo show generating a communication channel based on an interface. i.e. shows practical usage of source generators to help with boilerplate

But the approach (as in Unity Editor Websocket debugging) can be used for communication over any channel. The generator is not created to work over some kind of generic channel or Stream. Although it could, it is always intended that the final application might need to fork it and adjust it. As generic approaches loose performance (e.g. binary channels with direct response support being faster than binary channel without it, or having Blazor specific code)

This approach can be used for interprocess communication or any other kinds of native integrations. As source generators can create the binding code on both sides, or create apis on top of simple two-way stream (e.g. as in Unity WebGL interop, the interop is too messy to generate for each method signature, as it is more basic than Blazor/JS interop)

The serializers can be replaced, e.g. use built in JSON serializers. Although Unity one is very limited, e.g. no property support, no nullable struct support, root object must be class, .... Although MemoryPack has own limitations, e.g. no Typescript generation for structs (has to be done in fork)

Built in Unity serializer is not used as it breaks compatibility with native Blazor JS interop (which does not support serializing fields) But can be considered if the required renderers are not using native Blazor JS interop (as the field support can be added) Or a solution based on Newtonsoft JSON can be used, as that is more feature complete but is an extra dependency.

The TS generation is a bit intertwined with MemoryPack, since you could get rid of it, and in some cases it would work fine. But mixed modes need to sync TS code and C# code which is a bit messier. So it is kept connected for now to MemoryPack. But for DirectInterop any other or custom TS type generator from C# types could be used.

Renderers

  • coordinate systems
    • Blazor (adjustable as the app should choose this)
      • RightHanded
      • Screen (0,0) in top left
      • World X:right, Y: up, Z: toCamera
      • camera looks in NegativeZ
      • directionalLight shines in NegativeZ
      • Rotation is in Clockwise direction
    • Unity
      • LeftHanded
      • Screen (0,0) is Bottom Left
      • World X:right, Y: up, Z : fromCamera
      • camera looks in PositiveZ
      • Rotation order Z, X, Y (when going local to world)
    • Babylon
      • LeftHanded
      • Screen (0,0) in top left
      • World X:right, Y: up, Z : fromCamera
      • camera looks in PositiveZ
      • Rotation order Y, X, Z (when going local to world)
      • rotation is in Counterclockwise direction
    • ThreeJS
      • RightHanded
      • Screen (0,0) in center (positive in direction of top right) range: -1;1
      • World X:right, Y: up, Z : toCamera
      • camera looks in NegativeZ
      • Rotation order Z, X, Y (when going local to world) but can be chosen
      • rotation is in Counterclockwise direction

Unity WebGL (interop with Unity WASM via Binary Interop API)

Can use both MemoryPack serializer and Unity JsonUtility based serialization.

  • MemoryPack has better performance and more comprehensive support for language features (e.g. Properties) but needs extra libraries to be added (in case build size is a concern)
  • JsonUtility based approach has less dependencies, but performance is slower (messages take roughly 2x the time), and has very limited feature support (no properties and no nullable fields)

JS based libraries

There is npm run watch command to run in the assets subfolder of the project, e.g. \samples\BlazorWith3d.ExampleApp.Client.ThreeJS\assets folder, that does live recompilation of the TS codebase. Mainly to adjust the app.ts as needed.

The Typescript code is built when C# Project is. MemoryPack and own generator create .ts files as well and after build Webpack is triggered to create single JS file for the Blazor Component to load.

BabylonJS (interop with TypeScript via Binary Interop API)

Kept mainly for historical reasons, as ThreeJS seems to be more fitting for this usecase (as this is a game engine first, and threeJS is a renderer first).

ThreeJS (interop with TypeScript via Binary Interop API and Blazor interop)

Pure HTML (developed directly in Blazor)

Uses 2D screenshot for visuals, to look similar.

Blazor

Compilation flags

https://learn.microsoft.com/en-us/aspnet/core/blazor/performance?view=aspnetcore-9.0

https://learn.microsoft.com/en-us/aspnet/core/blazor/webassembly-build-tools-and-aot?view=aspnetcore-9.0

https://github.com/dotnet/runtime/blob/main/src/mono/wasm/features.md

Render modes

All render modes, including Auto mode are supported ( render mode can be chosen on the Home page)

Maui

Blazor Maui Hybrid is supported (only tested as Desktop app) https://learn.microsoft.com/en-us/aspnet/core/blazor/hybrid/tutorials/maui-blazor-web-app?view=aspnetcore-9.0

Benchmarks

Small msg is a message with a couple of numbers Large msg is a message with an almost 4k-character long string

Benchmarks are times for 10k messages.

Specific numbers should be only used as relative comparison, as this was times only on my machine.

InteractiveServer RenderMode

Timed on my machine with release build.

But in comparison to WebAssembly version, most time is lost on sending data back and forth from server to browser, then on interop specifics.

  • Unity with MemoryPack serializer
    • Small messages took 2596,00 ms (avg 0,26 ms)
    • Large messages took 3223,00 ms (avg 0,32 ms)
  • Unity with JsonUtility serializer
    • Small messages took 2791,00 ms (avg 0,28 ms)
    • Large messages took 4055,00 ms (avg 0,41 ms)
  • Blazor JS interop with MemoryPack
    • Small messages took 2069,00 ms (avg 0,21 ms)
    • Large messages took 2646,00 ms (avg 0,26 ms)
  • Blazor JS interop native
    • Small messages took 1909,00 ms (avg 0,19 ms)
    • Large messages took 2734,00 ms (avg 0,27 ms)

InteractiveWebAssembly RenderMode

Mainly for use as relative comparison, as they were timed on my machine with deployed demo app which is AOT compiled for webassembly.

For Javascript based render (e.g. Three.JS or Babylon.JS), Blazor JS native interop is quicker for smaller messages, but slower on big ones. But not significantly.

For Unity based renderer memory pack is faster for small messages and significantly faster for large ones, but it introduces extra dependencies making the build larger.

  • Unity with MemoryPack serializer
    • Small messages took 196.00 ms (avg 0.02 ms)
    • Large messages took 282.00 ms (avg 0.03 ms)
  • Unity with JsonUtility serializer
    • Small messages took 314.00 ms (avg 0.03 ms)
    • Large messages took 853.00 ms (avg 0.09 ms)
  • Blazor JS interop with MemoryPack
    • Small messages took 166.00 ms (avg 0.02 ms)
    • Large messages took 291.00 ms (avg 0.03 ms)
  • Blazor JS interop native
    • Small messages took 155.00 ms (avg 0.02 ms)
    • Large messages took 412.00 ms (avg 0.04 ms)

Initialization order

  • controller exists first (ie controller is singleton and gets a single renderer attached, the controller does not handle lifecycle of renderers, only should SetController to null when renderer is being replaced)
  • renderer is created and prepares to listen and sets the EventHandler to the controller
  • renderer calls SetRenderer on the Controller
  • Controller starts sending commands during SetRenderer execution
  • ie renderer can send messages to controller even before registering (as the controller has multiple renderers so it is considered more as API, than a tightly bound pair)
  • and renderer should expect messages to already arrive during SetRenderer execution

Deploy issues

CODE: 409 -> projectkudu/kudu#3042 (comment)

TODOs

Prio 0 (improve generic packages)

Add serializer concept to Typescript as MemoryPack generator needs to be split into BinaryApi and MemoryPack serializer

Handle warnings in Generator

extract the generators into own projects (as this repo is too 3D specific) and publish them and supporting libraries as nuget

Prio 1 (improve sample app)

  • showcase that there does not have to be direct mapping

    • generate wire connecting the instances
    • and generate cube under the gltf mesh

  • And drag and drop trigger to add blocks from HTML

  • Add context menu to delete block instance

  • check if GLTF instancing is working

Prio 2 (backlog)

Support of System.Numerics native types

Prio 3 (future ideas)

About

No description, website, or topics provided.

Resources

Readme

License

MIT license
Activity

Stars

3 stars

Watchers

1 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

No packages published

Languages