Migrating from Wyam - Confused about Default Pipelines

[shortbloke]

Firstly, if this is the wrong place/method to ask questions, please point me in the right direction.

I'm currently using:

    <PackageReference Include="Statiq.Lunr" Version="1.0.0-beta.62" />
    <PackageReference Include="Statiq.Web" Version="1.0.0-beta.49" />

I feel I must be doing this wrong, as I've had to effectively disable all the default pipelines and create the same pipelines we had whilst using Wyam. I'm almost at the point of using Statiq.App instead of Static.Web, but for development purposes we want the hosted webserver, hence using Statiq.Web.

Here's some example code, which is working but I feel it must be the wrong approach: https://gist.github.com/shortbloke/de13dd33c0b12409353a3116dd8ce84c

Question 1: I'd have thought I'd be able to add the option to MinifyJs to the existing pipelines "Assets" though the use of:
.ModifyPipeline(nameof(Statiq.Web.Pipelines.Assets), p => p.PostProcessModules.Add(new MinifyJs()))
However this resulted in it attempting to minify .xltx, .png, .pdf, .config files. Is there a better/correct what of handling this?

Question 2: Does each pipeline re-read all the files? I'm wondering if this multiple pipeline approach for simple tasks is inefficient. Should I be creating a fewer pipelines that take action based on the file type? Using ExecuteIf?

Question 3: The default pipeline runs Analyzers. This triggers warnings for all the links in my files. I'm not sure why:

[WARN] [C:/src/Highlight/HelpPages/input/help/404.cshtml => help/404.html] Could not validate relative link: <a href="/help/">Help &amp;amp; Support</a> (ValidateRelativeLinks)

I thought I could disable the ValidateRelativeLinks but setting it to false in the settings.yml file doesn't work, and there doesn't appear to be settable via: .AddSetting(WebKeys.ValidateRelativeLinks, false) as neither Keys nor WebKeys contain a property that can be set. Maybe I'm looking in the wrong place.

Question 4: If using defined pipelines as I have is an acceptable way to go. Then is there a way to remove all the other default pipelines, rather than run them with an empty file list?

Number of output documents per pipeline and phase:

 | Pipeline          | Input          | Process       | PostProcess   | Output       | Total Time |
 |------------------------------------------------------------------------------------------------|
 | AnalyzeContent    | 0 (0 ms)       |               |               |              | 0 ms       |
 | Archives          |                | 0 (0 ms)      | 0 (10 ms)     | 0 (0 ms)     | 10 ms      |
 | Assets            |                | 0 (10 ms)     |               | 0 (3 ms)     | 13 ms      |
 | Content           |                | 0 (9 ms)      | 0 (10 ms)     | 0 (0 ms)     | 19 ms      |
 | Data              |                | 0 (14 ms)     |               | 0 (4 ms)     | 18 ms      |
 | DirectoryMetadata | 0 (2130 ms)    | 0 (1 ms)      |               |              | 2131 ms    |
 | Feeds             |                | 0 (0 ms)      |               | 0 (3 ms)     | 3 ms       |
 | Inputs            | 0 (2130 ms)    | 0 (31 ms)     |               |              | 2161 ms    |
 | JavaScript        | 5 (2130 ms)    |               | 5 (33 ms)     | 5 (6 ms)     | 2169 ms    |
 | RazorPages        | 255 (2130 ms)  | 255 (2293 ms) | 255 (3108 ms) | 255 (148 ms) | 7679 ms    |
 | Redirects         |                | 0 (3 ms)      |               | 0 (3 ms)     | 6 ms       |
 | Resources         | 1413 (2130 ms) | 1 (1244 ms)   |               |              | 3374 ms    |
 | Sass              | 1 (2130 ms)    | 2 (97 ms)     |               | 2 (11 ms)    | 2238 ms    |
 | SearchIndex       |                |               | 3 (1388 ms)   | 3 (1 ms)     | 1389 ms    |
 | Sitemap           |                |               | 1 (1 ms)      | 1 (7 ms)     | 8 ms       |

Pipeline phase timeline:

 | Pipeline          | Timeline (7685 total ms not including overhead)                                      |
 |----------------------------------------------------------------------------------------------------------|
 | AnalyzeContent    |                                                                                  I   |
 | Archives          |                        P                       TO                                    |
 | Assets            |                        P                       O                                     |
 | Content           |                        P                       TO                                    |
 | Data              |                        P                       O                                     |
 | DirectoryMetadata | I----------------------P                                                             |
 | Feeds             |                        P                       O                                     |
 | Inputs            | I----------------------P                                                             |
 | JavaScript        | I----------------------                        TO                                    |
 | RazorPages        | I----------------------P-----------------------T-------------------------------O-    |
 | Redirects         |                        P                       O                                     |
 | Resources         | I----------------------P------------                                                 |
 | Sass              | I----------------------P                       O                                     |
 | SearchIndex       |                                                T-------------O                       |
 | Sitemap           |                                                TO                                    |

[daveaglick]

Firstly, if this is the wrong place/method to ask questions, please point me in the right direction.

Nope, this is the right place! I'm trying to consolidate all the Q/A style issues here in org-level discussions.

I feel I must be doing this wrong, as I've had to effectively disable all the default pipelines and create the same pipelines we had whilst using Wyam.

Statiq Web is similar in concept (if not implementation) to the old Wyam Blog recipe. It creates a set of default pipelines that are designed, even more than Wyam's were, to be turnkey with customizations being entirely performed using features driven by input files like archives, feeds, and others.

For a site that used the in-box Wyam Blog recipe without much customization, the porting should be relatively straightforward. There's some guidance for porting from a simple Wyam blog to Statiq here: https://github.com/statiqdev/CleanBlog#porting-from-wyam (note that the instructions are related to a theme this time around - because Statiq Web customization is so heavily oriented around input files it means that the theme retains a lot of control over things like conventions).

That said, a Wyam site that had lots of custom pipelines is probably better off, at least initially, using Statiq Framework and the Statiq.App package with the Bootstrapper to get going and porting those custom pipelines wholesale instead of trying to fit them into the new Statiq Web functionality.

Does that make sense at a general level?

Question 1: I'd have thought I'd be able to add the option to MinifyJs to the existing pipelines "Assets"...However this resulted in it attempting to minify .xltx, .png, .pdf, .config files. Is there a better/correct what of handling this?

What's going on here is that the Assets pipeline is designed to handle all documents with a ContentType of Asset. Content and media types are a brand new Statiq Web concept that essentially allow separating extensions from how we treat files. The file extension usually serves as a hint or default way of determining content and media type, but then other things can change that like front matter, sidecar files, directory metadata, etc.

So in your case, by adding the MinifyJs module directly to the Assets pipeline the module is getting all documents identified with an asset content type as inputs. Because MinifyJs doesn't make any determination about the input files and assumes you're only inputting documents to it that it should process, it ends up processing all asset documents.

What you want to do in this case is use another new Statiq Web feature called templates. The idea is that if the content type of a document indicates what general kind of treatment it should get (and thus which pipelines to send it through), the media type of a document determines more specific file-type processing. This Program.cs file will apply minification to JavaScript files:

using Statiq.Minification;

await Bootstrapper.Factory
    .CreateWeb(args)
    .AddTemplate(MediaTypes.JavaScript, ContentType.Asset, Phase.PostProcess, new MinifyJs())
    .RunAsync();

It's definitely totally different than how Wyam did file-type-specific processing, but it's hopefully much more powerful and easier to customize.

Also note for what it's worth that in-box minification will eventually be added directly to Statiq Web and controlled by settings. I went searching for the corresponding issue to past here and was shocked I hadn't created one - it's just been in my head. So here's a new issue to track adding minification to Statiq Web by default, though I have no idea when I'll get to it: statiqdev/Statiq.Web#996

Does each pipeline re-read all the files?

Not in Statiq Web (and not really in Statiq Framework either when using the ReadFiles module). One of the key differences in Statiq and Wyam is that Wyam stored the content of documents internally in memory. Statiq uses a new content abstraction that lets it keep only a reference to the original file and stream it as necessary (until the content is mutated and new content must be stored somewhere, usually in memory at that point). The key here though is that once a file is read and a document is created for it by the ReadFiles module, the file itself isn't actually "read" until needed - and even then it's streamed asynchronously.

I'm wondering if this multiple pipeline approach for simple tasks is inefficient. Should I be creating a fewer pipelines that take action based on the file type?

And you've inadvertently arrived at exactly what Statiq Web does :) - in Statiq Web, all documents are "read" once in the Inputs pipeline and the other content-type specific pipelines (I.e. based on file type) all have a dependency on the Inputs pipeline and get their input documents from there, filtering by content type.

If not using Statiq Web, following a similar pattern is suggested as you pointed out. Set up one pipeline to gather all your input files and create documents, then use that pipeline as a dependency to feed as many other more purpose-specific pipelines as needed (ExecuteIf also works for this pattern as you noted, but I find using pipelines cleaner for these larger functionality boundaries).

The default pipeline runs Analyzers. This triggers warnings for all the links in my files. I'm not sure why

Analyzers are a new concept in Statiq Web. They've been through a few iterations and should be mostly stable and tested at this point, though it's always possible new patterns could confuse the ones in the box like ValidateRelativeLinks (which has proven to be tricky to get right).

That said, it looks like in this case the 404.cshtml file has a link to either /help.html or /help/index.html (depending on whether friendly URLs are used and index files are hidden by the web server - the analyzer checks for both cases). If neither one of those files is being output, that's why you're getting a warning. If they are, let me know and it looks like we've got a bug here.

I thought I could disable the ValidateRelativeLinks but setting it to false in the settings.yml file doesn't work

Analyzer usage can be set in settings, but the syntax is a little tricky. To disable this particular analyzer you'd write (see near the bottom of https://www.statiq.dev/guide/execution/analyzers):

Analyzers:
  - ValidateRelativeLinks=None

If using defined pipelines as I have is an acceptable way to go. Then is there a way to remove all the other default pipelines, rather than run them with an empty file list?

You could remove them all by calling .ConfigureEngine(engine => engine.Pipelines.Clear()) on the Bootstrapper. Just be careful to add your own pipelines to the engine.Pipelines collection inside the same ConfigureEngine() call after clearing the built in ones.

But if what you really want is Statiq Framework just with the preview server but without any other Statiq Web features, you can also reference Statiq.Web and then bootstrap it like this (notice the .CreateDefault(args) call instead of .CreateWeb(args)):

await Bootstrapper.Factory
    .CreateDefault(args)
    .AddHostingCommands()
    .RunAsync();

Note that this approach won't light up support for all the other Statiq Web features like templates, etc. though and you'll need to do all those manually (just like you would in Statiq Framework). That's because a lot of those features are directly tied to what the default pipelines do.

Hopefully that answers most of your questions! Always happy for a follow-up if anything is still unclear.

[shortbloke]

Thank you for your comprehensive response. I've quickly been able to use Statiq Framework, with .web for the preview server.
I'll take another look at how I might use Statiq.Web and modify the templates and pipelines to suit my needs.

[daveaglick]

Awesome! You know where to find me if you have any other questions.