Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tracking] Publish .NET assemblies in Webcil files #80807

Closed
12 of 16 tasks
lambdageek opened this issue Jan 18, 2023 · 38 comments
Closed
12 of 16 tasks

[tracking] Publish .NET assemblies in Webcil files #80807

lambdageek opened this issue Jan 18, 2023 · 38 comments
Assignees
Labels
arch-wasm WebAssembly architecture area-VM-meta-mono tracking This issue is tracking the completion of other related issues.
Milestone

Comments

@lambdageek
Copy link
Member

lambdageek commented Jan 18, 2023

Summary

Webcil is a new container format for .NET assemblies that looks less like a normal Windows PE (Portable Executable) .dll file.
A .NET assembly in the Webcil format is a normal binary WebAssembly module with a particular set of exports that will copy a "Webcil payload" to WebAssembly linear memory.
It has a new .webcil extension a standard application/wasm MIME type and .wasm extension and the detailed format is specified in webcil.md.

There are two primary motivations for webcil:

  1. In some restricted environments, users may be prevented from downloading .dll or PE files, and some AV tools may quarantine .dll files found in browser caches.
  2. By using a new container, the runtime may be able to find additional flexibility in how we bundle and serve .NET assemblies. A goal is for the container to be invisible to higher layers of the runtime - unlike .netmodules in .NET Framework, as far as most of the runtime is concerned, we're still loading ordinary Ecma335 assemblies and any partioning is not visible at the metadata level. But at the loader level, the runtime may actually receive .webcil files that encapsulate one, part of one, multiple, or parts of multiple assemblies. (Version 0.0 of webcil only supports one assembly per .webcil container)

.NET 8 Preview 1

The goal is to make Webcil work in the wasm-experimental workload with the wasmbrowser template

The initial runtime work was done in #79416

Later in .NET8

Post .NET 8

  • Take advantage of webcil custom loader to optimize wasm bundles. (Define a multi-assembly .webcil container. Define a partial-assembly webcil container. Etc.)

Other

@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jan 18, 2023
@lambdageek lambdageek added arch-wasm WebAssembly architecture and removed untriaged New issue has not been triaged by the area owner labels Jan 18, 2023
@lambdageek lambdageek added this to the 8.0.0 milestone Jan 18, 2023
@ghost
Copy link

ghost commented Jan 18, 2023

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Follow-up to #79416

  • Enable debugger tests with Webcil
  • Implement a standalone WebcilGenerator MSBuild task for Blazor lazy loading scenarios
  • Gather feedback and identify further mitigations for restricted environments
  • Take advantage of webcil custom loader to optimize wasm bundles. (Define a multi-assembly .webcil container. Define a partial-assembly webcil container. Etc.)
Author: lambdageek
Assignees: -
Labels:

arch-wasm

Milestone: -

@lambdageek lambdageek added area-VM-meta-mono tracking This issue is tracking the completion of other related issues. labels Jan 18, 2023
@lambdageek lambdageek self-assigned this Jan 18, 2023
@lambdageek
Copy link
Member Author

lambdageek commented Jan 20, 2023

Code review nits:

@lambdageek
Copy link
Member Author

lambdageek commented Jan 23, 2023

lambdageek added a commit to lambdageek/xharness that referenced this issue Jan 23, 2023
WebCIL is a new container format for .NET assemblies used when
publishing WebAssembly apps (see dotnet/runtime#79416)

Related to dotnet/runtime#80807
lambdageek added a commit to lambdageek/dotnet-sdk that referenced this issue Jan 23, 2023
WebCIL is a new container format for .NET assemblies when publishing
to webassembly.  (See dotnet/runtime#79416)

Related to dotnet/runtime#80807
lambdageek added a commit to dotnet/xharness that referenced this issue Jan 23, 2023
WebCIL is a new container format for .NET assemblies used when
publishing WebAssembly apps (see dotnet/runtime#79416)

Related to dotnet/runtime#80807
lambdageek added a commit to lambdageek/aspnetcore that referenced this issue Jan 24, 2023
WebCIL is a new container format for .NET assemblies when publishing to webassembly.  (See dotnet/runtime#79416)

Related to dotnet/runtime#80807
lambdageek added a commit to dotnet/sdk that referenced this issue Jan 24, 2023
WebCIL is a new container format for .NET assemblies when publishing to webassembly.  (See dotnet/runtime#79416)

Related to dotnet/runtime#80807
ViktorHofer pushed a commit to dotnet/sdk that referenced this issue Jan 25, 2023
WebCIL is a new container format for .NET assemblies when publishing
to webassembly.  (See dotnet/runtime#79416)

Related to dotnet/runtime#80807
@lambdageek lambdageek changed the title [tracking] Webcil follow-up work [tracking] Publish .NET assemblies in Webcil files Jan 25, 2023
@charlesroddie
Copy link

As technically impressive as this work may be, it's an extensive workaround to the current blazor inadequacy of deploying dlls to user systems rather than compiling everything to WASM. Full compilation (via a fix to Mono WASM AOT or NativeAOT) will make this obsolete unless it finds some other application than browsers.

@ishepherd
Copy link

Re @charlesroddie's point. ☝️

Does the creation of .webcil - given the implied eternal support overhang - Is this an announcement that AOT won't and can't target WASM?

@jez9999
Copy link

jez9999 commented Feb 22, 2023

Maybe a bit of a dumb question, so sorry in advance, but if I'm using this new format, what will my dev tools Network tab look like if I do a hard reload? Will I see a ton of files being downloaded like I do now (the equivalent of the .dll's) or just one big webcil file? What will the MIME type be?

@Schaeri
Copy link

Schaeri commented Mar 12, 2023

Is it possible to try out the new WebCIL file format in Preview 1 of dotnet 8? We would like to get feedback from our customers as quickly as possible to determine whether this new format solves the firewall problems in our environment. Is there any documentation or how-to guide on how to create a Hello World Blazor WebAssembly application using the new WebCLI format?

@lambdageek
Copy link
Member Author

@Schaeri Webcil doesn't work with blazor yet - that's coming in a future .NET 8 Preview.

You can try it out by building a wasmbrowser app using the wasm-experimental workload (details in this blog post) and setting the WasmEnableWebcil property to true in the csproj:

  <PropertyGroup>
    <WasmEnableWebcil>true</WasmEnableWebcil>
  </PropertyGroup>

Something like this will work:

$ dotnet workload install wasm-tools
$ dotnet workload install wasm-experimental
$ dotnet new wasmbrowser
$ dotnet publish -c Release -p:WasmEnableWebcil=true

Now serve the bin/Release/net8.0/browser-wasm/AppBundle/ folder and have the affected customer load the page.
They should see something like "Hello, World! Greetings from [whatever URL you served]".

Please let us know whether it is working or not. If it doesn't work, it woudl be great to know what AV tool is being used and how it is configured (please ping me - we can discuss in private, if necessary)

@peterthorpe81
Copy link

That doesn't deal with the problem of tons of files being downloaded, then. Why can't they all be bundled into 1 file, at least optionally?

Yea, we have some flexibility here. Right now it's still 1 assembly -> 1 webcil file because I wanted to get feedback about whether this actually solves the AV & firewall issues for anyone. But with the infrastructure already in place we can be more flexible with how we bundle.

There could be issues with bundling everything into one file, however. for example if you deploy an update of an app, then it'll be a brand new bundle and users won't have it in their cache. the way things are today, if something (for example CoreLib, or some assemblies from some third party nuget) hasn't changed, that stuff will be in the user's cache already and only the updated assemblies would be downloaded.

I would think bundling would hit loading performance for Blazor? It couldn't begin loading until its got the full bundle rather than logical smaller files. In a https/2 or 3 world i'm not sure of the benefits of a bundle.

There could be some opportunities to reduce the perceived loading time of Blazor by partitioning files independently of dll structure. Blazor could download an initial webcil file containing the classes necessary to begin loading. Other webcil files download in the background and are hopefully obtained before they are needed as the site loads. Essentially a lazy loading process not bound by dll boundaries.

I think you will get more testers once its available for Blazor Wasm.

@rffsgate
Copy link

@Schaeri Webcil doesn't work with blazor yet - that's coming in a future .NET 8 Preview.

You can try it out by building a wasmbrowser app using the wasm-experimental workload (details in this blog post) and setting the WasmEnableWebcil property to true in the csproj:

  <PropertyGroup>
    <WasmEnableWebcil>true</WasmEnableWebcil>
  </PropertyGroup>

Something like this will work:

$ dotnet workload install wasm-tools
$ dotnet workload install wasm-experimental
$ dotnet new wasmbrowser
$ dotnet publish -c Release -p:WasmEnableWebcil=true

Now serve the bin/Release/net8.0/browser-wasm/AppBundle/ folder and have the affected customer load the page. They should see something like "Hello, World! Greetings from [whatever URL you served]".

Please let us know whether it is working or not. If it doesn't work, it woudl be great to know what AV tool is being used and how it is configured (please ping me - we can discuss in private, if necessary)

The small demo using webcil worked for us. Working with 2 companies that have ZScaler enabled, and both have an issue where blazor WASM projects don't load.

Need to have a solution for this before we can continue.

@paulguz-datapa
Copy link

Need to have a solution for this before we can continue.

BlazorWasmAntiVirus doesn't work for you?

@rffsgate
Copy link

rffsgate commented Apr 13, 2023

Need to have a solution for this before we can continue.

BlazorWasmAntiVirus doesn't work for you?

No, unfortunately it put my project in a state where it was just error after error. Spent way too many hours trying to recover afterwards.
Kept on coming up with errors that the file was locked, and couldn't be processed. Then it failed because it could not load the dll. Then it ended up that something was cached somewhere, because even after removing the nuget package, and deleting the compiled artifacts, it still didn't recover.
It put my installation in a state where I can't even create a new blazor web assembly project.
The dotnet 6 framework work, but the dotnet 7 one is broken properly.

@paulguz-datapa
Copy link

@rffsgate I have experienced the same on development machines. Sadly it is a known issue that the author is unable to fix. The workaround for us is to only use it in release builds, where it doesn't seem to be a problem.

@rffsgate
Copy link

Thanks @paulguz-datapa, at least I know I am not going nuts. Very frustrating.
I guess, that's why I am really looking forward to a solution that is built-in and not a work-around.

lambdageek added a commit to dotnet/aspnetcore that referenced this issue Apr 20, 2023
WebCIL is a new container format for .NET assemblies when publishing to webassembly.  (See dotnet/runtime#79416)

Related to dotnet/runtime#80807
@paulguz-datapa
Copy link

paulguz-datapa commented Apr 24, 2023

I've tried running this today using .Net 8 Preview 3.

The publish fails with the following, after a long pause:

C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.WebAssembly.Sdk\8.0.0-preview.3.23174.8\Sdk\WasmApp.Native.targets(
328,5): error MSB3073: The command "embuilder.bat build MINIMAL" exited with code 1. [C:\Users\xxx\wasmbrowser\wasmbro
wser.csproj]

There is no AppBundle folder. There is a 'publish' folder, but I get a JS error from dotnet.js if I run 'dotnet run' in that folder.

The embuilder error looks like a known issue that may well be fixed in Preview 4.

@Schaeri
Copy link

Schaeri commented Apr 28, 2023

@lambdageek: We received the first feedback from our three largest customers, who blocked our application even with the currently available workarounds.

The feedback was positive, for two out of three the first .Net 8.0 preview webcil sample worked. The blocking had nothing to do with the WebCil. The Azure URL for our test seems to be on a blacklist.

The Webcil solution seems to be promising.

We hope for more feedback, as soon as we hear more we will report it here again.

@lambdageek
Copy link
Member Author

lambdageek commented May 9, 2023

Hey folks,

Just want to highlight something that we're currently working on with webcil. Our updated plan is to wrap the webcil payload in a WebAssembly module. The upshot is that a browser-based .NET WebAssembly app, we will consist of:

  • JavaScript files (.js)
  • WebAssembly modules (.wasm)

In addition to the usual dotnet.wasm that contains the runtime, we will have additional .wasm files containing the .NET assemblies (encoded as Webcil).

The rationale is that we want to be in a situation where we only serve standard web app MIME types and file extensions: text/javascript and application/wasm

As an implementation note, the webcil-in-wasm modules will export a pair of functions getWebcilSize and getWebcilPayload that will copy the size (respectively data) of a webcil-encoded .NET assembly to linear memory.

The updated Webcil spec and implementation are in #85932 the goal is to land this in an upcoming .NET 8 Preview after Preview 4 is out. (That is, Preview 4 will not have webcil-in-wasm - it will have webcil as .webcil same as earlier previews)

@lambdageek
Copy link
Member Author

lambdageek commented May 10, 2023

@jeromelaban @MikeCodesDotNET @maxkatz6 FYI ^^^ #80807 (comment)

In .NET 8, we're going to change how we package .NET assemblies for the browser. For Blazor this affected some of their publishing support around lazy loading #85400, so you may also need to react in some way (or opt out)

  • .NET 8 Previews 1, 2, 3, 4: webcil is opt-in. If you opt in, assemblies are served as .webcil files
  • later .NET 8 Previews: webcil will be opt-out. If you opt in, assemblies are served as .wasm WebAssembly modules with webcil-in-wasm.

@MikeCodesDotNET
Copy link

@jeromelaban @MikeCodesDotNET @maxkatz6 FYI ^^^ #80807 (comment)

In .NET 8, we're going to change how we package .NET assemblies for the browser. For Blazor this affected some of their publishing support around lazy loading #85400, so you may also need to react in some way (or opt out)

  • .NET 8 Previews 1, 2, 3, 4: webcil is opt-in. If you opt in, assemblies are served as .webcil files
  • later .NET 8 Previews: webcil will be opt-out. If you opt in, assemblies are served as .wasm WebAssembly modules with webcil-in-wasm.

Thanks for the heads up! We really appreciate you letting us know of a potential issue for us. We'll investigate it this week and see what we need to do on our side.

lambdageek added a commit that referenced this issue May 16, 2023
Define a WebAssembly module wrapper for Webcil assemblies.
Contributes to #80807 

### Why

In some settings serving `application/octet-stream` data, or files with weird extensions will trigger firewalls or AV tools.  But let's assume that if you're interested in deploying a .NET WebAssembly app, you're in an environment that can at least serve WebAssembly modules.

### How

Essentially we serve this WebAssembly module:

```wat
(module
  (data "\0f\00\00\00") ;; data segment 0: payload size
  (data "webcil Payload\cc")  ;; data segment 1: webcil payload
  (memory (import "webcil" "memory") 1)
  (global (export "webcilVersion") i32 (i32.const 0))
  (func (export "getWebcilSize") (param $destPtr i32) (result)
    local.get $destPtr
    i32.const 0
    i32.const 4
    memory.init 0)
  (func (export "getWebcilPayload") (param $d i32) (param $n i32) (result)
    local.get $d
    i32.const 0
    local.get $n
    memory.init 1))
```

The module exports two WebAssembly functions `getWebcilSize` and `getWebcilPayload` that write some bytes (being the size or payload of the webcil assembly) to the linear memory at a given offset.  The module also exports the constant `webcilVersion` to version the wrapper format.

So a runtime or tool that wants to consume the webcil module can do something like:

```js
const wasmModule = new WebAssembly.Module (...);
const wasmMemory = new WebAssembly.Memory ({initial: 1});
const wasmInstance =
      new WebAssembly.Instance(wasmModule, {webcil: {memory: wasmMemory}});
const { getWebcilPayload, webcilVersion, getWebcilSize } = wasmInstance.exports;
console.log (`Version ${webcilVersion.value}`);
getWebcilSize(0);
const size = new Int32Array (wasmMemory.buffer)[0]
console.log (`Size ${size}`);
console.log (new Uint8Array(wasmMemory.buffer).subarray(0, 20));
getWebcilPayload(4, size);
console.log (new Uint8Array(wasmMemory.buffer).subarray(0, 20));
```

### How (Part 2)

But actually, we will define the wrapper to consist of exactly 2 data segments in the WebAssembly data section: segment 0 is 4 bytes and encodes the webcil payload size; and segment 1 is of variable size and contains the webcil payload.

So to load a webcil-in-wasm module, the runtime gets the _raw bytes_ of the WebAssembly module (ie: without instantiating it), and parses it to find the data section, assert that there are 2 segments, ensure they're both passive, and get the data directly from segment 1.

---

* Add option to emit webcil inside a wasm module wrapper

* [mono][loader] implement a webcil-in-wasm reader

* reword WebcilWasmWrapper summary comment

* update the Webcil spec to include the WebAssembly wrapper module

* Adjust RVA map offsets to account for wasm prefix

   MonoImage:raw_data is used as a base when applying the RVA map to map virtual addresses to physical offsets in the assembly.  With webcil-in-wasm there's an extra wasm prefix before the webcil payload starts, so we need to account for this extra data when creating the mapping.

   An alternative is to compute the correct offsets as part of generating the webcil, but that would entangle the wasm module and the webcil payload.  The current (somewhat hacky approach) keeps them logically separate.

* Add a note about the rva mapping to the spec

* Serve webcil-in-wasm as .wasm

* remove old .webcil support from Sdk Pack Tasks

* Implement support for webcil in wasm in the managed WebcilReader

* align webcil payload to a 4-byte boundary within the wasm module

   Add padding to data segment 0 to ensure that data segment 1's payload (ie the webcil content itself) is 4-byte aligned

* assert that webcil raw data is 4-byte aligned

* add 4-byte alignment requirement to the webcil spec

* Don't modify MonoImageStorage:raw_data

   instead just keep track of the webcil offset in the MonoImageStorage.

   This introduces a situation where MonoImage:raw_data is different from MonoImageStorage:raw_data.  The one to use for accessing IL and metadata is MonoImage:raw_data.

   The storage pointer is just used by the image loading machinery

---------

Co-authored-by: Larry Ewing <lewing@microsoft.com>
@lambdageek
Copy link
Member Author

lambdageek commented Jun 15, 2023

Hello everyone,

Webcil has now been turned on by default for Blazor WebAssembly projects in .NET 8 Preview 5. Blazor apps should now consist entirely of .js .wasm, .html and .css files. No more .dlls. If you previously had problems running Blazor WASM apps due to firewalls or AV software, please try building with P5 and let us know how it goes. I am interested both in successes and failures. If it's working, let me know. If it's not working, definitely let me know

@danroth27
Copy link
Member

@mihaimyh I believe you mentioned elsewhere that you're interested in a solution to Blazor WebAssembly apps getting blocked by proxy or antivirus software. Have you had a chance to give the new webcil packaging a try in .NET 8 Preview 5? Did it resolve any issues you may have been seeing?

@mihaimyh
Copy link

@mihaimyh I believe you mentioned elsewhere that you're interested in a solution to Blazor WebAssembly apps getting blocked by proxy or antivirus software. Have you had a chance to give the new webcil packaging a try in .NET 8 Preview 5? Did it resolve any issues you may have been seeing?

Hi @danroth27 I have tried webcil packaging with .NET 8 preview, and indeed, it solved antivirus blocking the download of the application binaries.

Looking forward for .NET 8 release date.

@rffsgate
Copy link

Thanks for the updates. I can also confirm that this worked for our organisation. Zscaler and AV didn't block it this time.

@paulguz-datapa
Copy link

I can confirm that this works with G DATA Internet Security. Thanks for your work on this.

@paulguz-datapa
Copy link

I can also confirm Sophos AV.

@lewing
Copy link
Member

lewing commented Jul 25, 2023

@lambdageek I think we can close this and let the transport feed issue stay open

@lewing lewing closed this as completed Jul 25, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Aug 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-wasm WebAssembly architecture area-VM-meta-mono tracking This issue is tracking the completion of other related issues.
Projects
None yet
Development

No branches or pull requests