Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Wyam vNext strategy #668
After Wyam 2.x ships, work will commence on a large shift to the Wyam architecture and the way it works to better align with lessons-learned from the last several years. I'll continue to update this issue content as my thinking evolves.
Celilo (static site generator application)
Wyam has traditionally straddled the fence between a "static site generator" when used with one of the recipes or a "static generator toolkit" when used with custom configuration files. That distinction will be further emphasized by splitting Wyam into two projects: a static site generator application similar in nature to Hugo, Jekyll, etc. (still deciding on a name, possibly Celilo - will use that for now) and a framework for building your own custom static generators (called Wyam).
The application will ship in a variety of ways:
It will contain a single "recipe" that covers both of the existing blog and docs use cases. It will use Wyam libraries for most functionality (including the CLI commands) and mostly consists of the pipelines needed for the "recipe". Otherwise, the concept of distinct recipes for specific use cases will be totally removed (it was too confusing). The term "recipe" won't be used in either Celilo or Wyam going forward. Configuration will be provided via a JSON or YAML file (user's choice, maybe other formats too) which will allow for code-free configuration like specifying global settings.
Themes will be plain old files in the
A set of .NET Core templates for different kinds of sites, for example:
The core library used for Celilo that contains all the pipelines. This will be separated into a library and distributed on NuGet so that custom Wyam applications can use it as a starting point to extend the default behavior instead of starting totally from scratch if they want to.
Wyam (static generator toolkit)
Wyam will live on as the engine powering Celilo as well as whatever custom generators a user wants to build. It will use a builder model for configuring from code, similar to the one used by ASP.NET. CLI command handling will also be provided as a library so that the user's custom application can do things like serve the site on a preview host, manage themes, etc. the same way Celilo can. Other existing Wyam libraries will continue essentially the way they are (Wyam.Hosting, modules, etc.).
Will support both the new static site generator application as well as custom Wyam applications.
Shipped as a library for integration into Wyam applications (including Celilo).
A set of .NET Core application templates.
Other templates TBD.
It seems like it is a great idea for the future of Wyam, especially for the debugging/IntelliSense part. Having a compile time check for the configuration or the recipes would be a huge improvement. I like the idea of going on a ASP.NET Core like model.
However, as mainly a user of Wyam in a CI context, I have some difficulty to see how it would change the experience : at the present time, when I want to update my static site, I update the input files, and I use the executable (through Cake) to generate a new version of my site. How would this change with the new version ? If I understood the proposition correclty, i would have to build an executable each time I want to do some modifications ?
Aside from this, theses changes looks good.
I completely agree with you that we need to make use of preexisting and familiar tools. The more Wyam resembles a standard ASP.NET project and process, the easier and better it will be.
I would like to see a "normal" VS/dotnet build, output static files instead of DLLs. Any custom build components should be built into the project type/configuration. (i.e.
I'm not sure why we would need an exe for each project. Are you suggesting that the config helps generate a custom exe and then we use that exe to build the site (i.e.
I would love to see us use an ASP.NET project with
I think the setup should resemble an ASP.NET Razor Pages project as much as possible. This way, if someone wants to add dynamic functionality, they still can, while those who just want the static content can just grab the output directory.
So Wyam would be added using something like
app.UseWyam(options => options.WithRecipe(new BlogRecipe().WithXXXX(YYYY))
Also, instead of a pipeline, I'm thinking that we would just use Middleware, which already forms a pipeline.
I like the idea of better tooling. And this way nobody needs to create it from scratch.
Instead of build we now need to run our project. I think I can get used to it when I get debugging :)
You could still use some pre-generation stuff to instantiate modules using a code generator.
It seems this system will make Wyam less easy to use for non-technical users (or even technical users without a c# background).
As a .net developer, I can feel all the benefits of the classic csproj format, but I can't refrain myself from thinking that a lot of users will not be able to use the tool or fix any compilation issue they could face (not closing all parenthesis, difference between ',' and ';', ...).
How will you be able to document how to add a pipeline manually, customize a theme and so on ? I can easily think of a developer-oriented documentation, but something on the level of jekyll seems to be hard to achieve (or I can't perceive it).
(Also note I have just discovered wyam with the launch of discoverdot.net, I have already used jekyll and some other tools, and I am evaluating wyam to convert some of our websites, so I don't have an expert eye on the tool).
We would only need "compilation" for advanced scenarios, like Pipelines (which is taking the text and compiling it anyway, right?) And even that would only be necessary during initial development. Once all that was done, adding a new page would be as simple as creating a new page and running a single command from the command line. But your point is well received, I would love to see Wyam match Jekyll in its simplicity and power, but still making it even more awesome for those who want tooling, especially for initial development.
Thanks for the comments everyone - this is great because the dialog is helping clarify some things for me. Here's some initial thoughts...
I don't think it'll change much at all, especially if you're already using Cake. The Cake addin will change under the hood to call
Yes, except the config part. There won't be a
This probably won't happen directly - one of the guiding principles of Wyam is that it's an anything generator that most folks just happen to use for web sites. I don't want to get away from that and modeling the layout and structure (in addition to the tooling) after ASP.NET is probably too web-centric.
That said, I could see developing a package that contains Wyam middleware for ASP.NET (or maybe MSBuild tasks) that does exactly what you're talking about and lets you use Wyam to process static files in your dynamic ASP.NET application.
Yes, exactly. The app is essentially your custom, bespoke static site generator. As with other generators, you don't need to recompile the generator to add new content or pages, only to change the features of the generator itself.
As usual, you're already on my wavelength
You're absolutely correct. That's my biggest concern with this change. The whole scripting DSL was created specifically to make onboarding for non-.NET folks easier (which is why it does things like hide common lambda expressions behind the
That said, I suspect the idea that Wyam would ever catch on beyond the .NET community was naive. There are already great static site generators for the masses with huge contributor pools and giant usage numbers. I mean, Gatsby just scored over $3 million in funding last week - that boggles the mind. This change doubles-down on .NET developers while admitting that's who the project is mainly for. I think I'm okay with that.
I like the sound of the changes, my biggest concern is it is moving away from how most static blog generators already work. Most execute similarly to how Wyam currently works. This change does open up the ability to create a CLI on top that adds a little additional functionality that doesn't currently exist like generators for posts, drafts, pages, etc.
The change also seems to make it easily integrate with asp .net core sites which would be good for several workflows.
As a user, I feel like it opense up to evaluating if I want to stay on Wyam or explore other options that provide cross platform support, why I was looking forward to 2.0, and an additional want was not having to need the .net runtime installed.
I look forward to getting to try it as I think this is the best .NET generator out there.
@RLittlesII I originally thought we could do an ASP.NET Core project, but we need an app that runs, transforms and returns, so a Console project is more applicable. Whether we can use ASP.NET Core Middleware inside a console app, would be an interesting study. ASP.NET Core was made modular, so the possibility is there, but we'd have to see. Reusing pre-existing, familiar frameworks/modules is generally a win/win, if it makes sense.
It's definitely an interesting thought experiment. What I can say with pretty high confidence is that Wyam's pipelines (or probably modules is more accurate) won't change to ASP.NET middleware. IMO, ASP.NET middleware too tightly coupled to the idea of a web server serving a single request, and I'm not a huge fan of the amount of magic that is involved in passing data between them.
That said, I would be curious to see what it would look like to write a generic ASP.NET middleware wrapper around Wyam's
My thinking here is that tools like dotnet-script would fill this need nicely. You'd reference the new
To make sure I understand you, what do you mean by "maintain a script host"? Are you referring to
And then how does
Finally, I thought we wanted to create a custom generator, in the form of an exe, for each project. C# scripts will not create that for us.
NOTE: Azure Functions for VS was once built on C# scripting, but everyone kept asking for features that only existed in normal C#, so they changed their course and focused on C# "Classic".
Wyam uses a custom scripting host that compiles and evaluates your
The Wyam script host does a bunch of stuff:
That's...a lot. And it's all Wyam code. I'd be happy to make most of that someone else's problem.
dotnet-script already does all those things (except for the custom Wyam DSL bits, but that was probably a bad idea to begin with). In fact, the script host in that project and the one in Wyam are very similar feature-wise.
It's also worth point out that Wyam is already usable as a library without going through the Wyam script host today. I know for a fact that some folks even prefer to use it from LINQPad. The main difference with this proposal is that the scripting capability will go away and embedded use will become the preferred method along with some additional APIs to better support it.
That'll be the end result for the most common scenario once Wyam is only shipped as a library (as described above). However, the user just needs a way to write, compile, and run code to use that library. If they'd prefer do that via a scripting tool instead of MSBuild/project file, that's just fine. In that mode it won't look much different than how it does today.
I was thinking suggesting using scriptcs as a script host as a simpler alternative of csproj + cs. Initially, I supposed they were the same (using the same .csx extension) but it seems it is not the case (despite having some contributors in common) dotnet-script seems to be the new engine.
I can totally understand the need to remove all code which is better maintained by other projects to focus on core features. The hosting part is clearly a good candidate.
I would suggest to try using dotnet-script with a wyam.csx script file as a new config/engine handler, and write some test scripts to see how different scenarios could be written this way.
In the most simplest way, you will need to run commands such as
Scripts could looks like this:
#! "netcoreapp2.1" #r "nuget: Wyam.Scripting, 1.0.0" using Wyam.Scripting; // static "Settings" class with static indexer Settings["Host"] = "example.com"; Settings["Title"] = "Example Site"; // static "App" class which builds a single website // App.Use<BlogRecipe>(); App.UseBlogRecipe(); App.Use("Markdown", p => p .Read("*.md") .FrontMatter(Yaml()) .Markdown() .Write(".html") ); // mandatory to run the app App.Run(Args);
changed the title from
Feedback Wanted: Wyam vNext
Wyam vNext (client strategy for 2.x)
Jun 6, 2018
This was referenced
Jun 6, 2018
I like where this is going, but I'm still unsure about the
Note: I verified that Jekyll also does not make use of a special
referenced this issue
Jun 7, 2018
That'll still be a feature, but it'll be delivered via the local per-project CLI tool that gets pulled down via NuGet (using a
FYI to the watchers of this thread - I've edited the main issue a couple times and will probably do so a bit more. I've been going back and forth for months on a few ideas with regard to how the CLI is shipped and custom code is integrated:
You'll notice all these options make use of a custom user application - the big question is when is it required (just for extending vs. all the time) and what kicks off generation (a CLI app vs. the user app). Hopefully getting close to settling on an approach. Bear with me :)
@daveaglick I noticed earlier in the thread we made reference to json or yaml configuration. I was wondering, is your thought that Wyam configuration would go away?
I was also thinking that if it does not, having the ability to pull the wyam.config into visual studio and make it a first class citizen with intellisense might go a long way towards support. I know a lot of my question around Wyam come from what are the correct API's to use to customize my build. If I had intellisense, I could browse the public API like any .NET library and maybe find my way a bit better.
Yes. At least in it's current scripted form. The idea is that the new SDK-style
The alternative would be to build a language server or Visual Studio addin for Wyam config files - that's doable, but watching the Cake folks figure this out has given me an appreciation for how hard it is. I'd rather focus my effort elsewhere.
I've continued to refine my ideas about what comes after version 2.0 and have updated the initial issue to reflect my current thinking.
One thing I want to point out separately though (and get some feedback on) is the idea of splitting Wyam the static site generator from Wyam the static generator toolkit.
The former would become it's own application (possibly called Celilo) and would encompass what is now the blog and docs recipes in a single, pre-configured experience (think traditional site generators like Jekyll, Hugo, etc.). It will be configurable via JSON, YAML, etc. but will not support custom config files or C# code (outside of Razor files). Essentially, it would be similar to other static site generators.
Wyam the static generator toolkit would live on as a series of libraries with a builder pattern for creating your own application, similar to the way ASP.NET works. It would still include all the modules, pipeline support, etc. we have now. To create your own generator (which I used to call a custom configuration, though I think that name is confusing) you would write a .NET application that uses the Wyam libraries.
I think this split has a few advantages:
I agree that a solution for both paradigms (expert tooling & config vs ease-to-use) was difficult to find. I think your approach of splitting the projects in two is nice (even changing the name).
So you expect Celilo to be the "brand" (as Jekyll, hugo, etc.), and Wyam the name of the toolkit/technology behind ? Do you expect to have one single website for both ?
What will certainly be hard will be to find an easy path for users who started a website with Celilo then want to migrate to a full custom app. Celilo will both require to be easy to use (consumer) and easy to understand when browsing source code. Perhaps a "dotnet new celilo" project template could be a nice feature.
@tbolon Was updating the parent issue and you beat me to it :).
Yes, the intent is to totally separate the two into distinct projects (both under the Wyamio organization), each with their own website, brand, etc. Something like "Celilo, powered by Wyam".
We'll be shipping a
Totally agree - planning on two sets of project templates, one for Celilo sites and another for getting started with Wyam applications.