Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WASM] Support for RenderTargetBitmap #10340

Open
jeromelaban opened this issue Nov 4, 2022 · 5 comments
Open

[WASM] Support for RenderTargetBitmap #10340

jeromelaban opened this issue Nov 4, 2022 · 5 comments
Labels
area/tests 🧪 Categorizes an issue or PR as relevant to tests difficulty/medium 🤔 Categorizes an issue for which the difficulty level is reachable with a good understanding of WinUI kind/enhancement New feature or request platform/wasm 🌐 Categorizes an issue or PR as relevant to the WebAssembly platform project/core-tools 🛠️ Categorizes an issue or PR as relevant to core and tools

Comments

@jeromelaban
Copy link
Member

jeromelaban commented Nov 4, 2022

We'll need two parts for this to work (Since HTML does not allow to take screenshots easily).

The first is give the ability to an app to capture its content (given the user's consent) using something like this:
Javascript provides this feature to capture the while screen:

const capture = async () => {
  const canvas = document.createElement("canvas");
  const context = canvas.getContext("2d");
  const video = document.createElement("video");

  try {
    const captureStream = await navigator.mediaDevices.getDisplayMedia();
    video.srcObject = captureStream;
    context.drawImage(video, 0, 0, window.width, window.height);
    const frame = canvas.toDataURL("image/png");
    captureStream.getTracks().forEach(track => track.stop());
    window.location.href = frame;
  } catch (err) {
    console.error("Error: " + err);
  }
};

capture();

What is not clear with this technique is that if we stop capturing, is there a second prompt showing up when capturing again. This would need some research. Also, given that the browser is asking for content, it may be best to place this feature behind a feature flag and explain what to do to enable it, and what the restrictions are.

The second part is more of an Uno.UITest change, which would require a new service that would take screenshots from an external process, during Runtime Tests on CI, but let's keep that one for later :)

@jeromelaban jeromelaban added kind/enhancement New feature or request platform/wasm 🌐 Categorizes an issue or PR as relevant to the WebAssembly platform area/tests 🧪 Categorizes an issue or PR as relevant to tests project/core-tools 🛠️ Categorizes an issue or PR as relevant to core and tools labels Nov 4, 2022
@amelield amelield self-assigned this Nov 4, 2022
@amelield
Copy link
Contributor

@jeromelaban There is a second prompt if we capture two images and I am not sure if we can, or should, change that. Do full page screenshot works or do we want only the section with our structure, or both? Were do we want the image to be saved (or downloaded) and do we have documentation on how to put a feature behind a feature flag? Thank you!

@jeromelaban
Copy link
Member Author

This would mean that we should be keeping the recording alive when it's started, so it does not prompt again. We can certainly keep a reference in our typescript layer.

@amelield
Copy link
Contributor

Do we want to start the recording the first time we take a screenshot or at init. I think it would be more effective especially for someone who does not need it a lot.
I am also working with the assumption that RenderTargetBitmap have to be accessible to all WASM and not only the Runtime test. Does this make sense?

@jeromelaban
Copy link
Member Author

@amelield it should be started only on demand (it's probably adding some CPU load). Yes, it makes sense to provide that feature to all users (still behind the flag that we'll set to true in the samples app).

@amelield
Copy link
Contributor

amelield commented Nov 21, 2022

@jeromelaban I am sorry for the amount of question. I may have thought too much this week-end or not enough at the end of the week.

First of all, I just saw the section about [JSImportAttribute] and I think my code should use that for dotnet7.
What is the test to make sure the version of dotnet used. Which file would be a good example of how to implement it differently for wasm dotnet6 and wasm dotnet 7? Where do public static partial class Imports for the [JSImportAttribute] would go in Uno?

Than my various questions:

Do the javascript script always have to be in a WasmScript section?

Should I touch Windows.Media.Capture partial class Screen Capture and the interface IMediaSource? I did not thought so since it seem disable for every platform. But I need to make sure.

Where do I put the activation (switch) in Sample and in which form?

Do I still need to be able to activate the code automatically in test or it is a problem for later?
Where do we save the RawBitmap. Do we want the option to download the file too?

Thank you for you time and for any answer you can provide.

@MartinZikmund MartinZikmund changed the title Support for RenderTargetBitmap on WebAssembly [WASM] Support for RenderTargetBitmap Aug 16, 2023
@MartinZikmund MartinZikmund added the difficulty/medium 🤔 Categorizes an issue for which the difficulty level is reachable with a good understanding of WinUI label Aug 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/tests 🧪 Categorizes an issue or PR as relevant to tests difficulty/medium 🤔 Categorizes an issue for which the difficulty level is reachable with a good understanding of WinUI kind/enhancement New feature or request platform/wasm 🌐 Categorizes an issue or PR as relevant to the WebAssembly platform project/core-tools 🛠️ Categorizes an issue or PR as relevant to core and tools
Projects
None yet
Development

No branches or pull requests

3 participants