Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Feature Modification Suggestions #692

Closed
parkr opened this Issue · 37 comments
@parkr
Owner

From @imathis and myself:

  1. Easier config (accept multiple .yml config files)
  2. Offer something other than liquid? it's really limited and seems like abandonware
  3. Do not regenerate when, say, a sass file is changed, as Compass/Sass CLI would listen for that.
  4. jekyll --auto for specific things (posts/certain pages/plugins, even)

Aaaand GO.

@tombell

Thoughts

  1. The configuration file is mainly complicated because literally every part of Jekyll is some-what configurable. A million different markdown parsers, etc.
  2. When I met with @mojombo we talked about switching to mustache for templating
  3. Depends entirely on the plugin implementation, I would just regen anyway, the plugin is run as part of the build process so the plugins shouldn't be 'watching'
  4. The watching is actually an option of the build process, and you might as well just let it watch everything, adding specifics just complicates something which doesn't really need it

Misc

Class/Module Structure

A lot of classes which are subclasses of things like generators, converters etc are named like:

module Jekyll
  class GenericAGenerator < Generator
  end

  class SomethingBConvertor < Convertor
  end
end

And I feel it would be better as:

module Jekyll
  module Generators
    class GenericA < Generator
    end
  end

  module Converters
    class SomethingB < Converter
    end
  end
end

Converters

Jekyll currently supports 4 different markdown rendering gems. I would personally either drop them all and just use RedCarpet (as this will offer the markdown support most similar to markdown on GitHub itself), or maybe think about restructuring how they work.

module Jekyll
  class Converter
  end

  module Convertor
    class Markdown < Converter
      def render(...)
        case renderer
        when 'redcarpet'
          RedCarpetMarkdown.process(...)
        when 'kramdown'
          # ...
        when 'rdiscount'
          # ...
        when 'maruku'
          # ...
        else
          # not supported/invalid
        end
      end
    end

    class RedCarpetMarkdown
      def self.process(...); end
    end

    class KramdownMarkdown
      def self.process(...); end
    end

    class RDiscountMarkdown
      def self.process(...); end
    end

    class MarukuMarkdown
      def self.process(...); end
    end
  end
end

Only supporting 1 markdown (textile only supports one library at the minute) would reduce the number of engine specific configuration options.

Migrators

Migrators are a maintenance pain in the ass. The add a ton of random bin/jekyll arguments. None of the migrators actually rely on any Jekyll functionality. I would prefer these were ripped out into separate Ruby scripts which when APIs and stuff changed didn't actually need a new full Jekyll release to be shipped.

A lot of the stuff I would prefer to change relate to the long term maintenance of the project and making it easier for contributors to navigate and contribute to.

@imathis

@tombell I completely agree with removing support for anything Markdown other than RedCarpet. Other markdown processors have caused nothing but pain for the Octopress project.

On the point of watching, not watching, I'll expound. I'd like to make it easy for preprocessors to work with content. Whether that be Sass, Coffeescript, or something else. It would be nice to be able to tell Jekyll to flat out ignore a subdirectory. As it stands now, Jekyll can ignore, but modifications in an ignored directory still trigger the regeneration of the app. I tried using Jekyll's Convertors for this, but it was too restrictive and unnecessarily tied preprocessors into Jekyll.

@tombell

Preprocessors should really be implemented as Jekyll plugins, then they will all be run as part of the build process. I'd rather run 1 command, than run 2-3 to actually watch my files for changes while developing.

@imathis

Do you expect to use something like Guard Listen to manage the watching? That would make it quite easy to integrate preprocessors.

@tombell

I'm not sure what you mean, Jekyll uses directory_watcher to handle rebuilding the site on changes.

@kevinSuttle

Weighing in here.

I've had this wish list for a static-site generator sitting in my Evernote for a very long time now.
Here are my checkboxes:

✔ Markdown, HTML,
✔ ownership of static files
✔ CoffeeScript support
✔ Sensible file structure/URLs
✔ SASS/Compass support
   × SLIM support
   × Dropbox sync or SSH/rsync
   × Mustache or Hogan.js @tombell +1!
   × Stylus/Jade

Jekyll covers most of these but I think what some of our more left-brained ideas have overly complicated things.
For example, do we really think more config files is going to be helpful?

The ones that are left out are what right-brained folks want:

  • A default, simple file structure and theme (thinking @html5-boilerplate)
  • mobile-first and responsive default theme
  • To work out of the box, like Wordpress
  • Incremental compilation
@parkr
Owner

@tombell Guard Listen would be most appropriate, as it listens for changes to certain files (whether in a certain directory or have some file extension) and process those based on what or where they are. It's a more targeted approach than what Jekyll uses now. It would decrease build time once the site has been built once. I implemented CoffeeScript and Compass compilers in a Jekyll project (as plugins) that I worked on this summer and it was the absolute worst. I agree that they should be more integrated into Jekyll (as plugins), but only after we have a more targeted auto-regeneration technique.

@kevinSuttle In terms of config, we'd still only have one _config.yml for Jekyll configuration, but it would also accept other configs which would be available to the site variable in your liquid markup. I like that you're suggesting new features, but I don't think that's what @mojombo is after. When we talked earlier, he was very clear that he wants Jekyll to remain simple. That being said, he wanted to make it easily extendable, so that Sass and CoffeeScript (and whatever else) could be integrated via a plugin or something. But they would not be built into Jekyll by default.

@mattr-
Owner

let's also not forget that migrators are completely untested as well. They seem like good things to factor out into separate repos/gems/whatever.

:+1: to only supporting Redcarpet.

I honestly hadn't very thought ahead when it comes to a vision for Jekyll when I started working. I just wanted to close the open bugs and pull requests and for things that didn't seem like they belonged in Jekyll's core to push them out into plugins or separate entities. If I want coffeescript support, I'll pull in a plugin. If I want sass support, I'll pull in a plugin.

The one thing that I would like, and it's just because I'm lazy, is for Jekyll to have a set of sane defaults so that I don't have to put YAML front matter on every single file. But like I said, I'm lazy, and really don't expect that to happen. :smile:

@mattr-
Owner

Also, not necessarily a feature, but are we abandoning the mailing list? If so, it should probably be shut down.

@parkr
Owner

There's a mailing list?!

@mattr-
Owner

@parkr yup, on the home page of the wiki. granted, it's more or less inactive, but it is there nonetheless. i have no objections to nuking it.

@baldowl

What follows are my own personal opinions only.

@parkr & @imathis: no, liquid is definitely not abandonware.

Before dumping liquid for something else, have a look at the liquid project again and remember that GitHub Pages still uses a version of liquid released more than two years ago (see https://help.github.com/articles/using-jekyll-with-pages and https://rubygems.org/gems/liquid).

Then, if I may add a question for someone more knowledgeable than me: using, for example, mustache for the templates would require moving the logic we can express with liquid (https://github.com/Shopify/liquid/wiki/Liquid-for-Designers) from the templates to Ruby code, right? How could this work in a restricted environment like GitHub Pages?

Another question for the more knowledgeable ones: does RedCarpet support everything the other Markdown converters provide?

@davide-targa

What about erb for templating?

@tombell

ERB allows you to run arbitrary ruby code, not safe for GitHub pages.

@davide-targa

@tombell
Yes, you're absolutely right. But for other usages (I mean for personal web sites, not for GH pages) couldn't ERB be a good choice?

@parkr
Owner

@davide-targa Perhaps! It'd be better to offer consistency, though.

@tombell

I don't really like ERB, and if you make it available, people would want it as part of GitHub pages, which it can't be.

@tombell

Also, I don't get what you mean by 'targeted auto regeneration', if I change a single HTML or Markdown file, it would need to rebuild the entire site as that one file might affect the rest of the other static pages (categories, related posts, etc).

I think a lot of people are misunderstanding what a static site generator actually does, I may just end up maintaining my own fork.

Edit

Not a big fan of Cucumber either, feature files are overly verbose and don't really suit this type of project. It also takes ages for the Cucumber tests to run.

@jerodsanto

I like the idea of consolidating markdown processors.

One thing that RedCarpet doesn't support is MultiMarkdown, which has been a common feature request (#289, #379, #473) and something that I'd personally like to see as part of Jekyll (mostly I just want to use footnotes).

I've been using the code from #379 for awhile and it works as advertised. The only problem with it is when used with directory_watcher it crashes Jekyll's server, but that appears to be a bug in rpeg-multimarkdown

@kevinSuttle

Speaking of "Jekyll's server", has any thought been given to replacing Web Brick? Thin, Unicorn, something else?

@tombell

I don't see the point in replacing WEBrick with an external dependency, you don't need thin/unicorn, you're just serving static pages locally for development.

If you're actually serving your Jekyll site with jekyll --serve you're doing something wrong.

@parkr
Owner

@tombell I think we can find a solution for the targeted approach. I've been working on something with a cache -- sort of the way Sass handles things, but modified for this projects needs. Still in the works, but we need to focus on speed.

Agreed that an alternative to WEBrick is not necessary.

@StevenBlack

TL;DR: Chaining markdown converters works. It just needs framework-level support. Ultimately chaining potentially boosts the overall extensibility of that juncture.


It would be sweet (and probably wise) to allow markdown converters to be chained.

An unstated acid-test for any markdown converter is: play-nice with HTML markup.

So I see no reason this would not be workable. Jekyll could support any markdown syntax, potentially several simultaneous flavors of markdown, by allowing implementers to chain a number of converters at that juncture.

A config array is advisable because converter sequence probably matters in some instances.

THEREFORE Jekyll could recommend and support only RedCarpet, and also make this a hot-spot and allow converter chaining on an implementation-specific basis.

Unit-testing this is easy: you just invoke RedCarpet n-times.

Also, once chaining is exposed, it opens implementations to limitless other simple, single-task, atomic-feature converters that can be invoked at this same juncture, either before, after, or in-between markdown converters.

And Jekyll Core doesn't care what users do there. Just expose chaining that can be defined, with sequence, in config.yml.

@tombell

If I enable the related posts feature, and I change a post, the whole site needs to be rebuilt if I wish to display related posts in a layout etc. What about pagination? If I change my layout, all my posts need to be rebuilt again.

I cannot see a way out of it, you can't really cache any part of the site as you cannot know how a user is actually using Jekyll to build their sites, unless you're going to introduce restrictions, which is a bad idea.

EDIT

I can't see a need to chain markdown renderers either.

@parkr
Owner

@tombell But certainly you don't see any reasons to recompile static pages like .pdf files and other assets that don't depend upon any of those changes?

@tombell

Those don't go through any type of compilation though, they'll just get copied as is

@parkr
Owner

Right, but that takes time :)

@parkr
Owner

And if a static file is changed, we don't need to recompile the site at all.

@StevenBlack

@tombell the potential chaining of markdown converters is a concrete example of extensibility where currently not much exists. (I'm new to Jekyll so bear with me). As I understand it, the markdown converter is currently a unary option. Implementations can specify just one.

That's not much of a hook.

The point isn't whether you see a need to chain markdown converters. The point is, if this is to be a useful hook, let's make it good one. Because I can make an implementation case for that juncture, among other things I'd like to do exactly there.

See, it's more about architecting the juncture than it is about the sole Markdown converter currently being invoked there.

Currently Jekyll defines eight clear and distinct processing junctures. They are succinctly listed here:
https://github.com/mojombo/jekyll/wiki/How-Jekyll-works

If Jekyll extensibility is more than a buzz, then we need to talk about implementing extensibility in a more-or-less standard way at multiple junctures, and at others clear junctures that may emerge.

Becuause to be useful, hooks need to occur around clear junctures. And to be usable, hooks need to be implemented in a coherent and consistent way across all junctures. Without both, we end-up with extensibility that's both ad-hoc and difficult to document. If each juncture is the same, it's a breeze.

At minimum I hope to see pre- and post- hooks at junctures. And it would be sweet if juncture execution was more than an simple invocation. I'd like to see an ordered set of processes to sequentially call, all with the same input and output types.

Need a pre-hook? Place it before the Markdown call in the array. Need a post-hook? Place it after the Markdown call in the array. Want to call more than one Markdown parser? Knock yourself out. Want to auto convert this-to-that based on implementation needs? Add something to the execution queue at a sensible juncture.

Same story at all junctures. It's called extensibility. It needs a vision.

@tombell

@parkr

you can't tell what is static and what isn't though, easily. https://github.com/tombell/tombell.github.com/blob/master/index.html

@StevenBlack

Calling more than one Markdown parser is actually quite pointless. The first parser would convert the Markdown, leaving nothing for the next. I can't see a reason you would use one parser then a second for just one or two features, why not just use the second fully?


If these are ideas that the majority want to see in Jekyll, there is no point me trying to object at all. I see it as a bad idea in the long run. The only thing I can do is maintain a lightweight fork for my own needs. I would rather focus my time to implementing improvements that I feel are beneficial to my needs, rather than features that I feel are harmful in the long run.

@StevenBlack

@tombell I'm sorry if I offended you. My main concern in this discussion is extensibility, and to generate ideas on how extensibility is best approached and implemented.

Calling more than one Markdown parser is actually quite pointless

I can tell you that I've been writing and maintaining parsers of all kinds since 1995. It's what I do. I urge you to have an open mind about what extensibility means here.

To answer your question directly: it's a mistake to assume that one Markdown processor is a syntactical subset of the other, and therefore what I propose is a stupid idea. To say this is to not fully understand the problem.

Assume instead that by calling two parsers in series, both markdown syntaxes can be supported. Wikimedia Markdown links aren't like RedCarpet markdown links. What I'm telling you is, you can have both in the same package. This is especially desirable if that package is assembled from various sources at build-time.

I know this works because I've written wikis that simultaneously support multiple flavors of wiki syntax, including multiple flavors of Markdown. This is implemented by calling syntax parsers in series. By chaining them. edit: ...or iterating them.

If you have a clean clear juncture, and fire just one thing there, it better be generic, mutable, and not limited to one worldview. That's the antithesis of extensibility. I'm sure that's not what you are proposing.

@tombell

@StevenBlack

No offense was taken.

I think your example of multiple markdown parsers was a bad choice, if the markdown syntax isn't the same as the spec, it's not really markdown. But you can achieve this behaviour currently using the Plugin system.

The problem with making this type of extensibility first class is that it CANNOT be used on GitHub:Pages because it can be used to execute arbitrary ruby code.


I wish people would stop thinking about the idea of 'incremental' compilation and stuff. It's a static site, the whole site needs to be rebuilt, unless it's a static asset (images, stylesheets, javascripts), but if those change they would still need copying. The only logic you can have is:

  1. File changed, is it an asset or something else
  2. If it's an asset, just copy it over
  3. If it's something else rebuild the site

You cannot assume anything about how the user has built their site. All content is static so if you don't rebuilt page Y which depends on some information changed in Page X, but you don't rebuild Y because it never changed the user will assume Jekyll is broken.

@agarie

Well, you can enable the chaining of parsers as a configuration option (obviously) and always overwrite it when processing Github:Pages. So I can use as many processors as I want and it should be safe enough for Github.

But @StevenBlack, why can't this be implemented simply with some converters?

And @tombell, I think @parkr meant exactly what you said: if the file is an asset, just copy it and don't compile the site. Otherwise, compile everything. If that's wrong, I really didn't understand his rationale.

@tombell

@agarie yeah exactly, with non-asset files it's impossible not have to regenerate the whole site. I may open a pull request with the kind of logic I described implemented.

@StevenBlack

@argyle, two things,

First, I'd really like to focus on improving the extensibility model, as opposed to how one could or couldn't achieve some implementation detail under the current extensibility clusterfk.

Second, to see what I mean, check this out:

https://github.com/mojombo/jekyll/blob/master/lib/jekyll/converters/markdown.rb

I submit that this is a perfect example of something that needs re-thinking. Look at the setup and convert methods. They are both implementation-specific case structures.

One proposal is: let's support just RedCarpet which is fine, as far as it goes.

My proposal is: let's think about changing how we do things not just here, but at all junctures. The time to think about an extensibility model for Jekyll is right now, and I sense time is running out.

I think a good way forward is to plan on hooking all junctures in a standard way, and let third parties go nuts exploiting those hooks.

@parkr
Owner

New discussion should be started in one of the issues above. Moving to make discussions more specific – one issue per topic.

@parkr parkr closed this
@jsw0528

why not write a markdown processors by yourself?

markdown is too old, we all like MultiMarkdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.