Accessing global aggregate data in PostProcess phase

[devhawk]

My post metadata include categories and tags. I can aggregate the category and tag data from IExecutionContext easily enough, but I don't understand where to put that aggregate category and tag data so that it can be accessed while rendering razor templates during the PostProcess phase. Obviously, I could loop thru all the post IDocument instances and add the aggregated data as metadata, but that doesn't seem very elegant. I'm just learning Statiq, so I am wondering if there's a better way for handling such scenarios

[devhawk]

I just figured out that the archive example in the statiq.web repo shows how to do this.

[daveaglick]

Glad you got it worked out (especially since I was disconnected for the US Thanksgiving holiday 😄)! Let me know if anything jumped out as needing better documentation or more details in the docs.

[devhawk]

Actually, in the end the statiq.web stuff didn't work out the way I hoped, but I was able to do what I wanted to do via a BeforeEngineExecution event handler. The metadata for my posts is in sidecar JSON files, so I was able to build an event handler that looped thru the sidecar files, parsed them as json, extract the category and tag data, generate the HTML from the aggregated data and add that to engine settings for later retrieval during razor template rendering.

It's not the most performant solution - it requires parsing the sidecar files twice - but I like this approach better than treating the rendered, shared HTML as an IDocument. As far as I can tell, in Statiq the only places to put this kind of data are Settings (which need to be generated in full before execution starts) or Documents (which are obviously generated during execution). However, neither is a good fit in this specific case. I want to generate the shared HTML from a pipeline, but use it as a setting.

I'm new to statiq (but very impressed!) so maybe I'm missing something. I'm OK with the solution I ended up on, but being able to have pipeline specific settings come from previous pipeline executions - similar to how pipelines can pull documents from dependency pipelines - would be a good add to the library.

[daveaglick]

being able to have pipeline specific settings come from previous pipeline executions - similar to how pipelines can pull documents from dependency pipelines - would be a good add to the library

I think you're bumping up against the entire reason process and post-process were created as separate phases, and if I'm understanding correctly, it may already do what you're looking for. The idea between having those two phases, and why Razor is rendered during post-process, is because there's definitely a cart-before-the-horse element to this. Consider a simple case of blog posts and tags. Let's say you want to render a tag cloud on every blog post page. Or maybe you want to list other posts with the same tag as the current one on each post page. In both cases we need to have fully "processed" the posts before rendering the post page so that we know the full set of posts for each tag in order to do our rendering. If we did everything in one pass, I'd get to a post before visiting other posts and I wouldn't have the full picture.

So I think that's sort of related to what you're trying to do here. You have some data that comes from your files (in this case from sidecar files that are getting processed automatically and included as metadata in your documents) and you want to access that aggregate data from Razor templates (during the post-process phase). The trick to understanding all this is the way the phases allow access to other data - the post-processing phase can access all documents from any pipeline as output from those piplines process phases. And because metadata (including sidecar files) are parsed during the process phase, Razor should have access to all the aggregate data already.

Maybe what's missing is a good example of how to get at it. When inside a Razor page, you can call Outputs.FromPipeline("PipelineName") and that will return all the documents (including metadata from sidecar files) for the given pipeline - then you can filter, aggregate, run LINQ queries, etc. on them as needed. Alternatively, you can use the Outputs or OutputPages (same as Outputs but pre-filtered to "pages" like HTML, Markdown, etc.) properties and their indexers if you'd rather get process phase documents from all pipelines by specifying a globbing pattern - something like OutputPages["my/stuff/*.html"] (the indexer operates on destination path which is set during the process phase, so by the time you use it in Razor most pages will already have .html destination extensions).

Is this along the lines of what you were looking for? Or have I totally misunderstood 😆. In any case, happy you were able to figure something out anyway - I love it when folks find features like the event handlers and use them in creative ways!

[devhawk]

You've got the basic tag cloud scenario down, I guess it's not clear to me the best way to make the aggregate data available in a way the render razor module can get to it.

The way I've done it, the aggregate data is available via settings, so I can call Context.Settings.Get<IHtmlContent>(MyKey) in my razor template. The downside of this approach is processing the sidecar files twice (both in terms of touching the files twice as well as duplicate logic to access them).

I know how to produce the aggregate data in a pipeline, but I don't understand how to "publish" it so downstream pipelines and modules can access it. I have an 'Inputs'" pipeline that reads all the files + handles sidecar processing and I have a BlogPosts that takes the output from Inputs and filters down to just blog post files, sorts them, sets their destination, renders markdown, renders razor and the writes individual post files out. I also have a Tags pipeline that takes the output from BlogPosts and writes out summary pages (i.e. 5 posts per page) for each tag. All of these pages need the aggregate tag info. (example: http://devhawk.net/blog/tag/asp-net). I use a the same layout .cshtml file for both individual blog posts and multiple tag post pages, so whatever solution I end up on has to work for both pipelines.

So I guess I don't understand how to publish and then access the aggregate info during the BlogPosts processing phase so that it's accessible to both the BlogPosts post-processing phase as well as the dependend Tags pipeline post processing phase. Do I create a document for the aggregate info? What do I call it, given that I don't actually want to emit a file to the static web site for this information? How do I access the aggregate info in a downstream Razor template?

[devhawk]

I think I need to take a deeper look at https://github.com/dotnet-foundation/website. They seem to be doing what I need

[devhawk]

The dotnet foundation blog is an example of what I'm trying to do. The "archives" section on the right hand side of the page is implemented as an HTML partial. Does that partial get implemented on every page execution? Or is the value cached?

[daveaglick]

Does that partial get implemented on every page execution? Or is the value cached?

This is a great question! By default it's re-rendered (but not recompiled) on every page.

It's worth noting that the Razor engine has two main "phases": compiling and rendering. The compilation page creates an in-memory assembly (which Statiq actually saves to disk for caching from run-to-run) that includes a class for each page, layout, partial, etc. which literally consists of your Razor code (I.e. @{ ... }) and the equivalent of stream.WriteLine() statements. Then the rendering phase uses that class to call a render function (I think it's literally called Render() offhand but don't quote me on that).

In the case of partials, the compilation is only performed once and cached for each path on disk to a partial file. However, the rendering is performed each time the partial is used inside some other page. That can be really expensive if the partial does computationally heavy work. For that reason, I recently introduced CachedPartial() and CachedPartialAsync() HTML helpers for use in Razor pages (they're available automatically and can be used instead of @Html.Partial() and @Html.PartialAsync()). These will render the partial once and cache the result as well, returning the rendered HTML for subsequent usages. But here be dragons because things like the Statiq document, context, etc. will be from the first time the rendering was performed. So if the partial uses the current document, for example, you don't want to cache it.

[devhawk]

For that reason, I recently introduced CachedPartial() and CachedPartialAsync() HTML helpers for use in Razor pages (they're available automatically and can be used instead of @Html.Partial() and @Html.PartialAsync()).

That is exactly what I need. Thanks!

[daveaglick]

Cool, let me know if you run into any problems. It's a new feature and not documented yet, so if you need to refer to the code it's here:

Statiq.Framework/src/extensions/Statiq.Razor/IHtmlHelperExtensions.cs

Lines 293 to 432 in 0c192e4

	public static void RenderCachedPartial(
	this IHtmlHelper htmlHelper,
	string partialViewName) =>
	RenderCachedPartialAsync(htmlHelper, partialViewName).GetAwaiter().GetResult();
	public static void RenderCachedPartial(
	this IHtmlHelper htmlHelper,
	string partialViewName,
	object model) =>
	RenderCachedPartialAsync(htmlHelper, partialViewName, model).GetAwaiter().GetResult();
	public static void RenderCachedPartial(
	this IHtmlHelper htmlHelper,
	string partialViewName,
	object model,
	object cacheKey) =>
	RenderCachedPartialAsync(htmlHelper, partialViewName, model, cacheKey).GetAwaiter().GetResult();
	public static async Task RenderCachedPartialAsync(
	this IHtmlHelper htmlHelper,
	string partialViewName) =>
	await RenderCachedPartialAsync(htmlHelper, partialViewName, null, null);
	public static async Task RenderCachedPartialAsync(
	this IHtmlHelper htmlHelper,
	string partialViewName,
	object model) =>
	await RenderCachedPartialAsync(htmlHelper, partialViewName, model, model);
	public static async Task RenderCachedPartialAsync(
	this IHtmlHelper htmlHelper,
	string partialViewName,
	object model,
	object cacheKey)
	{
	CachedPartialContent content = await GetCachedPartialContentAsync(htmlHelper, partialViewName, model, cacheKey);
	content.WriteTo(htmlHelper.ViewContext.Writer, null); // We know this call doesn't use the HtmlEncoder
	}
	public static IHtmlContent CachedPartial(
	this IHtmlHelper htmlHelper,
	string partialViewName) =>
	CachedPartialAsync(htmlHelper, partialViewName).GetAwaiter().GetResult();
	public static IHtmlContent CachedPartial(
	this IHtmlHelper htmlHelper,
	string partialViewName,
	object model) =>
	CachedPartialAsync(htmlHelper, partialViewName, model).GetAwaiter().GetResult();
	public static IHtmlContent CachedPartial(
	this IHtmlHelper htmlHelper,
	string partialViewName,
	object model,
	object cacheKey) =>
	CachedPartialAsync(htmlHelper, partialViewName, model, cacheKey).GetAwaiter().GetResult();
	public static async Task<IHtmlContent> CachedPartialAsync(
	this IHtmlHelper htmlHelper,
	string partialViewName) =>
	await CachedPartialAsync(htmlHelper, partialViewName, null, null);
	public static async Task<IHtmlContent> CachedPartialAsync(
	this IHtmlHelper htmlHelper,
	string partialViewName,
	object model) =>
	await CachedPartialAsync(htmlHelper, partialViewName, model, model);
	public static async Task<IHtmlContent> CachedPartialAsync(
	this IHtmlHelper htmlHelper,
	string partialViewName,
	object model,
	object cacheKey) =>
	await GetCachedPartialContentAsync(htmlHelper, partialViewName, model, cacheKey);
	private static readonly ConcurrentCache<(string, object), Task<CachedPartialContent>> _cachedPartialContent
	= new ConcurrentCache<(string, object), Task<CachedPartialContent>>(true, true);
	private static async Task<CachedPartialContent> GetCachedPartialContentAsync(
	this IHtmlHelper htmlHelper,
	string partialViewName,
	object model,
	object cacheKey)
	{
	htmlHelper.ThrowIfNull(nameof(htmlHelper));
	IServiceProvider serviceProvider = (IServiceProvider)htmlHelper.ViewContext.ViewData[ViewDataKeys.StatiqServiceProvider];
	ICompositeViewEngine viewEngine = serviceProvider.GetRequiredService<ICompositeViewEngine>();
	// Get the normalized path so that we can match up the partial regardless of where it's called from or the name
	// Copied from HtmlHelper.RenderPartialCoreAsync()
	ViewEngineResult viewEngineResult = viewEngine.GetView(
	htmlHelper.ViewContext.ExecutingFilePath,
	partialViewName,
	isMainPage: false);
	if (!viewEngineResult.Success)
	{
	viewEngineResult = viewEngine.FindView(htmlHelper.ViewContext, partialViewName, isMainPage: false);
	}
	// If we can't find it this way, go ahead and try again normally and that'll throw the error
	if (!viewEngineResult.Success)
	{
	throw new Exception($"Could not find partial to cache with name {partialViewName}");
	}
	// Cache the partial results using the path name by writing to a memory stream
	return await _cachedPartialContent.GetOrAdd(
	(viewEngineResult.View.Path, cacheKey),
	async (key, args) =>
	{
	StringBuilder builder = new StringBuilder();
	using (StringWriter writer = new StringWriter(builder))
	{
	// Temporarily replace the writer in the view context for rendering the partial
	TextWriter originalWriter = args.htmlHelper.ViewContext.Writer;
	args.htmlHelper.ViewContext.Writer = writer;
	await (args.model is object
	? args.htmlHelper.RenderPartialAsync(args.partialViewName, args.model)
	: args.htmlHelper.RenderPartialAsync(args.partialViewName));
	args.htmlHelper.ViewContext.Writer = originalWriter;
	}
	return new CachedPartialContent(builder);
	},
	(partialViewName, htmlHelper, model));
	}
	private class CachedPartialContent : IHtmlContent
	{
	private readonly StringBuilder _builder;
	// StringBuilder is not thread-safe, but since we're only reading it's okay to use as the buffer
	public CachedPartialContent(StringBuilder builder)
	{
	_builder = builder;
	}
	public void WriteTo(TextWriter writer, HtmlEncoder encoder) => writer.Write(_builder);
	}
	}

[devhawk]

worked exactly as expected first time I tried it

[daveaglick]

Awesome!