Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP


Make `jekyll listen` (`--auto`) specific #706

parkr opened this Issue · 35 comments

10 participants

Parker Moore Tom Bell Alexey Zapparov Steven Black Brandon Mathis Carlos Agarie Nick Quaranto Anthony O. andrew morton Chris Hough
Parker Moore

If we could more specifically watch for auto-regeneration (or even regular regeneration), that'd be sweet.

It'd look like this:

jekyll --auto [folder] # regenerate like normal if [folder] is not specified,
                       # otherwise, regenerate only when something in 
                       # [folder] changes
jekyll --auto # auto-regenerate like normal
jekyll --auto ./_posts # auto-regenerate just on changes to files in `_posts` (and subdirs)

More discussion in #692 but post new comments here.

Tom Bell

I don't get the appeal of this, even if you're only changing files in _posts, it'll regen those, what's the point of making it specific?

Parker Moore

Doesn't look like it to me!

That code regenerates the entire site from scratch once it's called (site.process, to be exact). The idea would be to pass a filtered *args (filtered to be just those in the subfolder specified) to site.process and it only process those which were changed from that list.

Tom Bell

I've already explained, everything needs to be regenerated or the site will just break though.

Parker Moore

Not if you know your changes, and definitely not if you're just changing static files!

Alexey Zapparov

I agree with @tombell. Normally you don't "modify" static files. Most of modifications comes to layouts and posts. So we can say that most of the time you will need to regenerate whole site.

Steven Black

I agree with @parkr.

Let's not forget a typical workflow, which might mean lots of edits on a single post (where blast regeneration is OK) and, when done, a full-site regeneration can be triggered just before upload or git push into a live site.

Steven Black

How about this refinement: Leverage the post naming convention. Can specify a skeleton (which could be a folder, or a file selection wildcard).

jekyll --auto [skeleton] # regenerate like normal if [skeleton] is not specified,
# otherwise, regenerate only when something matching
# [skeleton] changes

Say we have lots of posts and we're in December 2012, so only regenerate this month's posts.

jekyll --auto # auto-regenerate like normal
jekyll --auto ./_posts/2012-12-* # auto-regenerate just on changes to December files in _posts

I'm guessing those who post frequently would appreciate this.

Tom Bell

And if you have your posts listed on the index page, or an archives page, or use a similar-posts plugin, your site will end up messed up.

Steven Black

@tombell ... which doesn't matter until you actually deploy.

Note that, under this proposal, the default remains to build all.

Tom Bell

I'm going to maintain my own fork at tombell/mrhyde, with the direction I believe I'd like to see Jekyll go, it will be possible to backport my changes that people wish to see in Jekyll.

Steven Black

Uh, @tombell, that's twice now you've said something like this in the very recent past..

Sorry, again, if I said something to offend you.

But at some level, we've got to be able to have a technical conversation on merits.

Without threats that valuable people are taking their hockey nets home.

It's almost like you're saying, you don't want to entertain any conversation about improved extensibility, upon explicit threats (twice now) of walking away.

If that's indeed the case, here's my proposal: you stay. I'll go.

Steven Black

I'd like to add one thing: when it comes to discussions about extensibility, replies along the lines of "I don't get the appeal of this" should be phrased as: "show me the use case".

Because if someone interjects into a discussion about extensibility with "I don't need it" then the conversation doesn't concern that person. It's for people with explicit needs that are different than yours.

"I don't need it" should not be impetus to add noise, nevermind shut down, a technical conversation.

Brandon Mathis

I realize that my issue #708 should probably be noted here. I think resolving that issue would probably solve many of my complaints about the way jekyll --auto works. I maintain that it would be better if Jekyll could understand the difference between dynamically compiled assets (like Coffeescript and Sass) and dynamically compiled templates like HTML and Markdown.

The problem as I see it is that Jekyll converters are designed to work with template files but not assets. Changes to CSS or Javascript should never require HTML to be regenerated and CSS and Javascript shouldn't be run through Liquid or required to have YAML headers. Basically the current converter concept is designed to work for compiling HTML templates, but not dynamic asset compilation and Jekyll doesn't offer a good solution for managing dynamically compiled assets.

It is possible that this issue could be resolved with by addressing #705 and #708.

Tom Bell

@StevenBlack I believe in majority rules, people are welcome to contribute what they like, it's open source and all that, not everyone is going to share a common goal. I'm happier to maintain a fork than discuss things which I don't agree with. If I don't like something, I'll build my own, that's the nature of open source to me. I've contributed a number of fixes/features to Jekyll, but if it's drifts in a direction that no longer suits my needs, I don't have to stick with using it.

I've explained in length the technical reasons why the issues discussed in this thread really aren't feasible as an end product to me. Honestly, there are plenty of applications to preview markdown and textile rendering, I don't see the need to focus on this in Jekyll. Jekyll is a static site generator. That's what it is to me. When something has too many concerns it becomes bloated in my eyes. If you want more specific options for watching the file system, use Guard with Jekyll. Don't start adding everything to Jekyll to cover every minor use case.

Steven Black

@tombell Ah. I see. Well, it's a shame you don't understand what I'm suggesting.

Tom Bell

I do understand, I just feel it will lead to a incomplete product if we follow through with the feature outlined in this issue.

Steven Black

It seems to have escaped you that somebody configuring his session's watcher for partial re-builds

  1. Probably has a damn good reason, and
  2. Would know to completely re-build before deployment and
  3. Even if not 2 above, it may not matter and, if it does, it'll be plainly evident, and
  4. The default Jekyll behavior doesn't change.

Anyway, I've said enough.

Alexey Zapparov

What I see, is that with such feature there will be a certain time, when site is completely broken. Yes, it might be OK when you are just editing one post, but, still this leads into situation when site will be only partially working and that is absolutely not what I would like to see in Jekyll. It increases risk of publishing broken website by accident (forgot to run full compile).

Also, after all, this feature might be useful for testing of edits of one particular post/page (I don't see any other usage). But in this case I don't see any reason to "optimize" compilation, as normally you either write a lot so full recompilation will not cause you too much pain, or you made a small change (fix typo) which is done rarely and does not requires immediate recompile to check.

Alexey Zapparov

@StevenBlack I understand that default behavior will not change, but it will be extra complexity in code, which we must avoid IMHO. Especially when profit is not so obvious. Just answer the questions:

  • How long compilation takes?
  • How often you need to compile?
Alexey Zapparov

After all I really believe that extra complexity is OK only if it's really needed. At the moment there were no real-world examples when it might be needed (only claims that "probably it would be damn good,", which is not enough to be a damn good reason IMHO :D)

Carlos Agarie

I think that @ixti has a very good point: is this change really necessary? I mean, isn't there more important things at the moment? For example, given the discussion so far, I think that the hooks suggested by @StevenBlack would be much more interesting for Jekyll than specific auto-regen.

How long compilation takes?

Not much for my super tiny personal website, but I don't think this time is really an issue even on bigger sites - the time spent on specific tasks would be more significant. Want an example? LSI classification of posts, which #664 addresses pretty nicely.

Brandon Mathis

Regarding compilation time, I've heard from several Octopress users with large blogs 900+ posts who say compilation time on the latest mac hardware takes over 2 minutes, and some on older hardware report 8-10 minute compile times. With related posts turned on, I've heard of people abandoning at half an hour.

I'd imagine at this scale, limiting or disabling pagination would speed things up a bit, but I think it's important to point out that heavy Jekyll users really feel the slowness.

Alexey Zapparov

@imathis That's the good example. I thought about something like that, but, I have a strong feeling that the bottle neck is LSI in that case, not rendering...

Tom Bell

To build a static site/blog with Jekyll, which is its function (static site generator) you need to build every, single, page, if you don't like how long it takes, maybe people are misinterpreting what a static site generator actually does. There is going to be a performance hit when generating a site of 900 blog posts, there is no way to make it faster.

At development time, Jekyll is used for previewing your SITE not your individual blog post you're currently working on.

Brandon Mathis

@tombell the problem I see isn't so much how long the generator takes, but how it needlessly regenerates. If a javascript or CSS file changes, the templates don't need to be regenerated. The fact that any file change (not just template files) triggers a full site regeneration is a problem. People who are iterating on javascript or CSS files for their site have to wait a long time to see those changes in the browser. If Jekyll could simply copy non template "static" assets to the site directory, the slow regeneration would be less painful, since it will no longer be happening needlessly.

Parker Moore

@tombell I think @imathis explains the struggle here expertly. The needless regeneration of files which have not changed (if I change a post, no other files need to be regenerated unless it's something that would exist on an index/archive page or in a related post link) is a problem and that's an area that we can and should improve. The fact of the matter is it takes a long time to regenerate a site, and that's partially because our processes for generation are not optimal. They're good, but they can be better. The argument for making things faster is based on the premise that our code is not optimal and we have the opportunity to improve upon that. The suggestion outlined in this issue would be an easy way to do this because jekyll would only "listen" to certain directories or files and only auto-regenerate those until deploy.

Tom Bell

The issue you're describing is irrelevant to the one described in this issue discussion...

Tom Bell

Everyone has talked about improving it, yet no one has proved they can improve it.

To quote Linus; talk is cheap, show me the code.

Rather than talk about doing it, just do it and open a pull request, time spent arguing over it is time wasted that could be spent implementing the feature. If you think it can happen, make it happen.

Nick Quaranto

I'd like to downvote this as well. I know I'm being a naysayer lately, but I reread this thread and I'm really not liking the direction this takes Jekyll into. I'd rather make Jekyll more simple instead of more complex. This feature obviously goes in the latter direction.

The --auto flag is something that my personal Jekyll workflow depends on, but I think leaving this concern to Guard (which didn't exist when Jekyll started) is fine now. Actually, I'd love to see the --auto flag go away (and the directory_watcher dependency along with it), and just promote usage of Guard instead. The new site can help with that too, and even provide an example Guardfile to use.

Tom Bell

What @qrush said probably makes the most sense, and then users are free to use whatever 'Guard'-like process they prefer.

Parker Moore parkr closed this
Anthony O.

Does someone have a real concrete example using "guard"-like process with jekyll in order to work around that closed issue ?

Parker Moore

@anthony-o Check out guard-jekyll-plus by @imathis. It's awesome.

andrew morton

+1 on guard-jekyll-plus it's awesome

Chris Hough

did guard-jekyll-plus fix everyone's slowness during rebuilding? I am using it, but I am not noticing that much of a speed boost :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.