How do I setup a Lunr search?

[MarkZither]

This follows on from #15, my wyam based blog has a search based on the SearchIndex module and these custom pipelines configured in the wyam.config

// https://randalvance.net/post/implementing-searching-wyam.html
// https://github.com/cvan/lunr-unicode-normalizer
Pipelines.Add("PostsSearch",
    ReadFiles("posts/*.md"),
    FrontMatter(Yaml()),
    Markdown(),
    Meta("SearchIndexItem", new SearchIndexItem("/" + @doc.String("RelativeFilePathBase"), @doc.String("Title"), @doc.String("Tags"))
            { Description = @doc.String("Lead"), Tags = @doc.String("Tags")  })
);

Pipelines.Add("SearchIndex",
	Documents("PostsSearch"),
	SearchIndex((FilePath)"stopwords.txt"),
	WriteFiles((doc,ctx) => string.IsNullOrEmpty(doc.Content) ? null : "assets/js/searchindex.js").UseWriteMetadata(false)
);

I found the [discussion on renaming modules which explains SearchIndex -> GenerateLunrIndex, so i have been trying to add a pipeline which calls GenerateLunrIndex.

Something like this gets me close (also tried BuildPipeline method and creating a pipeline but ended up stuck at a similar place)

await Bootstrapper
                .Factory
                .CreateWeb(args)
            .ConfigureEngine(x => x
                .Pipelines[nameof(Statiq.Web.Pipelines.Content)]
                .ProcessModules
                .Add(new GenerateLunrIndex(stopwordsPath: "/stopwords.txt").WithDestination("/assets/js/searchindex.js")))

In ExecuteContextAsync of GenerateLunrIndex i see the correct number of documents.

But after that I get an empty array for searchIndexItems. I think .SelectAwait(async x => await _searchIndexItem.GetValueAsync(x, context)) is returning null, being filtered by the .Where(x => !string.IsNullOrEmpty(x?.Title) && !string.IsNullOrEmpty(x.Content)) and therefore I hit this

if (searchIndexItems.Length == 0)
            {
                context.LogWarning("It's not possible to build the search index because no documents contain the necessary metadata.");
                return Array.Empty<IDocument>();
            }

What is the best way to hookup the GenerateLunrIndex module?

[MarkZither]

Merry Christmas, a good time for me to look at some issues, so i came up with a pipeline that is working for me to get the same output as my wyam blog, still some config issues to tidy up, but it works.

public class SearchIndex : Pipeline
    {
        public SearchIndex()
        {
            Dependencies.AddRange(
                nameof(Inputs),
                nameof(Content),
                nameof(Archives),
                nameof(Data));

            var docs = new GetPipelineDocuments(ContentType.Content);

            ProcessModules = new ModuleList
            {
                //new ExecuteIf(Config.FromSetting(CustomKeys.GenerateSearchIndex, false))
                new ExecuteIf(IExecutionContext.Current.GetBool("GenerateSearchIndex"))
                {
                    new GetPipelineDocuments(ContentType.Content),

                    // Filter to non-archive content
                    new FilterDocuments(Config.FromDocument(doc => !Archives.IsArchive(doc))),

                    //new FilterDocuments(Config.FromDocument(doc => doc.GetBool("IsPost"))),
                    
                    new ForEachDocument
                    {
                        new ExecuteConfig(Config.FromDocument((searchDoc, ctx) =>
                        {
                            ModuleList modules = new ModuleList();
                            // this isn't right, would like to use IsPost but it doesn't seem to be available
                            if(searchDoc.Destination.Parent.Name.Equals("posts", StringComparison.InvariantCultureIgnoreCase)){
                                var searchItem = new LunrIndexDocItem(
                                    searchDoc,
                                    searchDoc.GetTitle(),
                                    searchDoc.GetContentStringAsync().Result);
                                searchItem.Description = searchDoc.GetString("Lead");
                                searchItem.Tags = searchDoc.GetString("Tags");

                                modules.Add(new SetMetadata("LunrIndexItem",
                                    searchItem
                                ));
                            }
                            
                            return modules;
                        }))
                    },

                    new GenerateLunrIndex("LunrIndexItem", "stopwords.txt", true)
                        .WithDestination("assets/js/SearchIndex.js")
                }
            };

            OutputModules = new ModuleList
            {
                new WriteFiles()
            };
        }
    }

[daveaglick]

You're awesome for coming back and sharing your approach. I'm obviously having some trouble keeping up with discussions recently (work has been crazy, etc.) so I really appreciate the follow-up in case anyone else is having similar issues.