[Docs-Sprite.FromFile] Is it true, Atlased sprites are not implemented?

[laultman]

Getting this when trying to use an Atlas file.
[SK diagnostic] sprite_create: Atlased sprites not implemented yet! Switching to single.
[SK warning] A GPU asset is blocking its thread until the main thread is available, has async code accidentally shifted execution to a different thread since SK.Initialize?

I've spent days building all the supporting structures for atlased handling and get down to this line straight out of the documentation,
Note: fi is a FileInfo object val.Key is a string with the value "00001" where this points to the atlas image 00001.png.

var sprite = Sprite.FromFile(fi.FullName, SpriteType.Atlased, val.Key);

It is in fact hanging the thread. The code never returns after the call to the method.

There are some other issues around this call. If the file name is invalid a hard exception is thrown somewhere down low in the C++ that is not trappable by the C#.

[maluoi]

Atlas support is currently in-progress in a feature branch! It'll get merged at some point, but this won't impact your code any, the fallback is sufficient. Atlasing will give a nice reduction in draw calls, but that's the only difference you'll see when that lights up. If you continue coding as though atlases do work, you'll get the performance boost when the feature is finished automatically!

The GPU asset is blocking warning may come up in some cases with async code near or between SK.Initialize and SK.Step/SK.Run. Async code can invisibly switch the current thread, which can be disruptive to StereoKit if it breaks initialization and stepping into separate threads. I'd love to know more about your code here! This warning is almost certainly the root cause of the hang.

I'll look into the crash on Sprite.FromFile as well, which version of StereoKit is this?

[maluoi]

I'm not able to reproduce a crash by passing an invalid path name to Sprite.FromFile, could you provide the callstack, or other additional details for that crash?

[laultman]

On the crash issue, I will roll back through my code and hopefully find the offending snippet. That was last week, which is a long time ago in the code :)
I am using 0.3.7 preview 5

The GPU asset is blocking warning may come up in some cases with async code near or between SK.Initialize and SK.Step/SK.Run. Async code can invisibly switch the current thread, which can be disruptive to StereoKit if it breaks initialization and stepping into separate threads. I'd love to know more about your code here! This warning is almost certainly the root cause of the hang.

You asked to know more about the code... so TL;DR
First, it is based on the holographic remoting using an edge compute service as the actual application execution. The edge computing service also does a lot of security, user asset/action control, and cloud data management. With all this, the application generates everything, so there is a no-code solution for HoloLens 2. It is successfully deployed now and there is no UWP involved. What I am adding is video/animation capability. I am also using voice commands and audio player now. SK is the heart of the delivery to the user. The edge application is also generating a ThreeJS for computer users and a WebGL rendering for Teams when using Microsoft Remote Assist. Latency is managed by a quasi-CDN because we know the users and the work schedules, minimizing the dynamic data loading requirements.

Code approach: I am trying to load assets in the background to local disk from Azure blobs. The user is experiencing a "bait & switch" screen with some user interactions to keep him occupied while I grab the application's current asset needs to my edge compute running the remote rendering package. The main asset load is initiated at the Program.cs level before any calls to SK. I am maintaining an app level static global "state-machine" (AppState) that is updated via api calls to the static service.
As soon as the load thread is kicked off it runs independently and sets a collection of "stateful" flags in AppState class that is visible to the entire app. As soon as the background operations are started, I call my class the represents SK.

#region Launches the 3D Remoting service
// AppRemote is effectively the "application". It has its own "Step" method independent of all other scenes.
// Avoid long running calls at this point because the user is not yet seeing anything from the remote app.
Console.WriteLine("Launching...");
app = new AppRemote(ipAddrs, appManager);				// Initialize the remote app services and passes the requesting HoloLens IP address
app.Initialize();										// Now call the Initialize method, before calling the SK.Run(app.Step()) method.
app.AddAppScenes(appManager.AppConfig.SplashSceneName);	// Makes the SceneSplash the currently "ActiveScene".

Console.WriteLine("Running...");
#endregion

The call to the "new AppRemote" method initiates the constructor on my SK framework with this simple code:

public class AppRemote : IStepper
...
public AppRemote(string ipRemoteDevice, AppManager appMgr)
{
    appManager = appMgr;	// receives an instance of the AppManager
    
    // Network IP address of HoloLens Device using this instance.
    StereoKit.SK.AddStepper(new HolographicRemoting(ipRemoteDevice));

    // Initialize StereoKit, must be done very early in the process
    IsSKready = StereoKit.SK.Initialize(startupSettings);

}

I have found that this method must return (seems to be synchronous) before doing anything else that involves async code. That said, my next call (coming from the Program) calls the SK initialize method.

public bool Initialize()
{
    // Awaits the VideoPlayer on its constructor. This allows the async Initialize method to load files
    // without blocking on the main thread.
    VideoRunner(new AppServices(), GetAssetPath(AppState.defaultTexturesFolderName), "lavengimg").GetAwaiter().GetResult();
    
    micSprite     = Sprite.FromFile("sk.png", SpriteType.Single);
    micMaterial   = Default.MaterialUnlit.Copy();
    micSpritePose = new Pose(new Vec3(-.5f, 0, -.1f), Quat.Identity);
    boundsMat     = Default.MaterialUIBox;  // now set the bounds material
    micBounds     = new Bounds(new Vec3(.1f, .1f, .1f));

    menuSelectPose.position    = new Vec3(0, -0.3f, -0.6f);
    menuSelectPose.orientation = Quat.LookDir(-Vec3.Forward);
    InitMenuAsync();


    sceneManager = new SceneManager(appManager);  // creates an instance of the SceneManager. See class docs for DI use.
    SceneManager.FindAllScenes(AppConstants.AssemblyNameForScenes); // the name of the assembly where the application scenes are located

    icon = Sprite.FromFile(Path.Combine(AssetsFolderPath, AppConstants.AssetsFolderName + "\\sk.png"), SpriteType.Single);
    HolographicTheme.Apply();

    // TODO: add any initialize activities here to cleanup before ending app.
    return true;
}

Since this method is not async, I am having to manage the threads carefully to ensure I don't go tripping into limbo land. Once the Initialize method is completed, an initial scene (splash) is set as the active scene, then the SK run is called.

#region Starts the render SK Engine
// Now pass execution over to StereoKit
SK.Run(app.Step, () => Log.Info("Bye!"));

Console.WriteLine("...App closed.");
#endregion

From here it is fairly, normal operation in SK.

My problem currently is the necessity for control over the video playback as a scene object, not as an independent video player. I need much more finesse over the video playback and its visual location. So, the seemingly simple approach was to paint a sprite animation on a 3D object (plane for example). My concept there is to have the "video player" created as an object derived from IStepper while my scenes all stem from my version of a scene manager that ultimately derives from IStepper. This separates the scene actions from the VideoPlayer, yet allows me to handle scene action affecting the video. (e.g. Gaze controlled auto-pause/play (gaze control -rewind/fast-forward, etc.)
My primary customer base is HoloLens industrial-base customers where users don't normally have a lot of free hand capability. I am not concerned in any way with cross platform capability.
In case you are not aware. Engineering information is presented to field technicians as "imaged" technical drawings. The most efficient means of delivery is a sight-controlled interface. Also, a must is the delivery of video of the actions to be performed (how-to examples) or for inspections, previous video or stills of the same area (rusting bridges for example).

[laultman]

Git did a crappy job of code display!

[maluoi]

Yeah, okay, the async calls are almost certainly still tripping things up here! The key thing to note is that async does not behave the way one intuitively expects it to. When you start an async method, the current thread is used for the async call, and the rest of the code continues on a new thread!

You can find a few extra details about this over here which is where I first learned about all this.

I think I'd recommend using Task.Run instead? Not 100% sure, but I believe this does manage to dodge the thread split issue. Or even better, start running/stepping SK immediately to make the app responsive, and then set up some initialization scene that'll also indicate progress as things load.

(I also edited your post a bit so the code shows up nicer :) Try triple backticks with the csharp tag!)

[laultman]

I'll take a look at the thread swap thing. I haven't really delved into it too much. That's a nice trick with the triple backtick, thanks for the tip!

[laultman]

Well after reading the" thread" over in #316 looks like some refactoring is in order.

[laultman]

I have it working now. I moved all the async code that involved loading Sprites from either the FromFile or FromTexture to a new class that is instantiated through its constructor as an async class. I read in the SK documentation that files and textures are handled async in the C++. Because this is on its own thread, my c# try/catch can't see any errors. It also means that if I try to use any of the async SK operations, I must ensure that I am returning to the thread where it was initially called.
By adding the GetAwaiter to the class and using a Task/Run in the GetAwaiter method I am able to use the built-in SK async operations without losing my thread. I also changed the signature to all async methods in the class to return a Task, no async void methods.

		// constructor
                public VideoPlayer(AppServices appServices, string textureSourcePath, string videoName, bool useAtlasSource = false)
		{
			_fso = appServices;
			_texPath = textureSourcePath;
			_srcVidName = videoName;
			_UseAtlas = useAtlasSource;
		}

		/// <summary>
		/// Gets the awaiter. Allows class to be "awaited" via its constructor.
		/// </summary>
		/// <returns>TaskAwaiter.</returns>
		public TaskAwaiter GetAwaiter()
		{
			return Task.Run(async () =>
			{
				await LoadInitialImageSequencer();
			}).GetAwaiter();
		}

In my app's code derived from IStepper and in its Initialize method I am calling a local private method.

		public bool Initialize()
		{
			// Awaits the VideoPlayer on its constructor. This allows the Initialize method to load files
			// without blocking on the main thread.
			VideoRunner(GetAssetPath(AppState.defaultTexturesFolderName), "lavengimg\\z0001").GetAwaiter().GetResult();

               ... rest of the initialize code follows

Then the private method:

		public async Task VideoRunner(string textureSourcePath, string videoName, bool useAtlas = false)
		{
			videoPlayer = new VideoPlayer(new AppServices(), textureSourcePath, videoName, useAtlas); // auto initialize to this video
			await videoPlayer; // causes the GetAwaiter method to be called in the class as an async

			if (useAtlas)
			{
				await videoPlayer.LoadVideoBufferFiles(0, 10);
			}
			else
			{
				// TODO: Handle the loading with parameters possibly
			}
		}

With this setup I can now use async in my class and it will ultimately return back awaited to its called point even with the SK using its own async services not exposed to the C#.

[laultman]

I now have a fully functional video player. It only has one external dependency on my framework. That is to load files from the device. I'm sure that could be implemented by anybody. My next improvement is to include the video file as a stream to frame-by-frame. Since I am using Holographic remoting, I am not dependent on the HoloLens 2 device to do the work.
I can share the code with you Nick - how should I do that? I only have one issue at the moment, I have not figured out how the Time.Elaspedf relates to actual time in milliseconds. Reading in the C++ it states clearly that there is no direct correlation to the system time (I assume for cross-platform), that it only allows for calculations of time passing. Video is rather sensitive to time intervals.
My code accumulates time slices used during the frames until I reach a threshold of time "delay" to trigger a new frame load.

			lastElasped += Time.ElapsedUnscaledf;
			fpsCalculator.CalculateSmoothedFrameRate();
			if (lastElasped * videoFPS >= videoFPS / fpsCalculator.SmoothedFrameRate || !firstFrameLoaded)
			{
				firstFrameLoaded = true;
				lastElasped = Time.ElapsedUnscaledf;

                       ... rest of the code to acquire the next frame

For some reason that I have not understood, the video is running faster that it should. FYI the videoFPS is set to 24.

[maluoi]

There's also a Time.Totalf you can use here to avoid accumulation errors! If you're doing a flipbook of images, I'd probably approach it something like this:

float startTime = Time.Totalf;
// ...
float currTime  = Time.Totalf-startTime;
float framerate = 1.0f / 24.0f;
int   frame     = (int)( currTime / framerate ) % maxFrames;

[maluoi]

Core topic for this issue is resolved, closing :)