New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto regeneration performance increase #3365

Closed
kaatt opened this Issue Jan 26, 2015 · 22 comments

Comments

Projects
None yet
5 participants
@kaatt
Contributor

kaatt commented Jan 26, 2015

When auto regeneration is enabled for a large project, changes to a single file take a long time (for mine, ~10s). Say I change something in style.css, jekyll auto regenerates BUT before the regeneration is finished, I change header.htm. Now jekyll will keep regenerating for style.css and after that's done, it will regenerate for header.htm.

For better performance, what it should do instead is stop the regeneration for style.css and start the regeneration immediately for header.htm.

@parkr

This comment has been minimized.

Show comment
Hide comment
@parkr

parkr Jan 27, 2015

Member

@alfredxing Would be awesome to get some insight into what it ends up determining to regenerate. Some console output with DEBUG turned on would be great 😄

Member

parkr commented Jan 27, 2015

@alfredxing Would be awesome to get some insight into what it ends up determining to regenerate. Some console output with DEBUG turned on would be great 😄

@parkr

This comment has been minimized.

Show comment
Hide comment
@parkr

parkr Jan 27, 2015

Member

@kaatt If you update an include or a layout, there's going to be a lot more regeneration taking place in general as all the docs which use them have to be regenerated as well.

Member

parkr commented Jan 27, 2015

@kaatt If you update an include or a layout, there's going to be a lot more regeneration taking place in general as all the docs which use them have to be regenerated as well.

@alfredxing

This comment has been minimized.

Show comment
Hide comment
@alfredxing

alfredxing Jan 27, 2015

Member

@kaatt Is this auto regen as in jekyll-watch or as in incremental rebuild?

Member

alfredxing commented Jan 27, 2015

@kaatt Is this auto regen as in jekyll-watch or as in incremental rebuild?

@kaatt

This comment has been minimized.

Show comment
Hide comment
@kaatt

kaatt Jan 27, 2015

Contributor

@alfredxing Auto regen. I've been annoyed by this since I've started using Jekyll.

Contributor

kaatt commented Jan 27, 2015

@alfredxing Auto regen. I've been annoyed by this since I've started using Jekyll.

@kaatt

This comment has been minimized.

Show comment
Hide comment
@kaatt

kaatt Jan 27, 2015

Contributor

@parkr In my project, even with incremental rebuild, the speed of regeneration doesn't change if I modify style.css (which other files are not dependent on) or a _includes/_layouts file (which many other files are dependent on). It feels like the entire site is being regenerated. That's why I want the older regeneration to be killed as soon as Jekyll knows it has to regenerate again.

Contributor

kaatt commented Jan 27, 2015

@parkr In my project, even with incremental rebuild, the speed of regeneration doesn't change if I modify style.css (which other files are not dependent on) or a _includes/_layouts file (which many other files are dependent on). It feels like the entire site is being regenerated. That's why I want the older regeneration to be killed as soon as Jekyll knows it has to regenerate again.

@alfredxing

This comment has been minimized.

Show comment
Hide comment
@alfredxing

alfredxing Jan 27, 2015

Member

@kaatt I don't think there's much we can do about the auto regen, since it just calls site.process every time a file changes...

As with the incremental, you can check the dependencies generated by looking in the .jekyll-metadata file in the source dir.

Member

alfredxing commented Jan 27, 2015

@kaatt I don't think there's much we can do about the auto regen, since it just calls site.process every time a file changes...

As with the incremental, you can check the dependencies generated by looking in the .jekyll-metadata file in the source dir.

@kaatt

This comment has been minimized.

Show comment
Hide comment
@kaatt

kaatt Jan 28, 2015

Contributor

I know nothing about jekyll internals, this was just something I thought would improve the speed.

I already know about .jekyll-metadata. Although it lists deps: [] for binary files, just renaming them in my project takes about ~8s to incremental regen. Also, the speed decreases as the number of regens increase.

  Regenerating: 1 file(s) changed at 2015-01-28 06:14:28 ...done in 3.004067 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:15:00 ...done in 3.468349 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:16:36 ...done in 3.588527 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:28:29 ...done in 8.836812 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:29:02 ...done in 4.534146 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:30:50 ...done in 4.273488 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:31:58 ...done in 4.326855 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:32:27 ...done in 4.559026 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:32:44 ...done in 6.180277 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:35:26 ...done in 4.875153 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:37:04 ...done in 4.943501 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:38:09 ...done in 5.157025 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:39:43 ...done in 5.610551 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:42:31 ...done in 6.032605 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:50:28 ...done in 8.788579 seconds.
  Regenerating: 2 file(s) changed at 2015-01-28 06:50:37 ...done in 8.189694 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:51:01 ...done in 8.821763 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:51:10 ...done in 9.474697 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:51:19 ...done in 8.980164 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:51:28 ...done in 8.328257 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:51:37 ...done in 9.13793 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:58:06 ...done in 8.207349 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:58:24 ...done in 7.777429 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 07:02:30 ...done in 13.169898 seconds.

The same task took 3s in the beginning but in the end it doubles. If I restart jekyll, it becomes fast again.

Contributor

kaatt commented Jan 28, 2015

I know nothing about jekyll internals, this was just something I thought would improve the speed.

I already know about .jekyll-metadata. Although it lists deps: [] for binary files, just renaming them in my project takes about ~8s to incremental regen. Also, the speed decreases as the number of regens increase.

  Regenerating: 1 file(s) changed at 2015-01-28 06:14:28 ...done in 3.004067 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:15:00 ...done in 3.468349 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:16:36 ...done in 3.588527 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:28:29 ...done in 8.836812 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:29:02 ...done in 4.534146 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:30:50 ...done in 4.273488 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:31:58 ...done in 4.326855 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:32:27 ...done in 4.559026 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:32:44 ...done in 6.180277 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:35:26 ...done in 4.875153 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:37:04 ...done in 4.943501 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:38:09 ...done in 5.157025 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:39:43 ...done in 5.610551 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:42:31 ...done in 6.032605 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:50:28 ...done in 8.788579 seconds.
  Regenerating: 2 file(s) changed at 2015-01-28 06:50:37 ...done in 8.189694 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:51:01 ...done in 8.821763 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:51:10 ...done in 9.474697 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:51:19 ...done in 8.980164 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:51:28 ...done in 8.328257 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:51:37 ...done in 9.13793 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:58:06 ...done in 8.207349 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 06:58:24 ...done in 7.777429 seconds.
  Regenerating: 1 file(s) changed at 2015-01-28 07:02:30 ...done in 13.169898 seconds.

The same task took 3s in the beginning but in the end it doubles. If I restart jekyll, it becomes fast again.

@alfredxing

This comment has been minimized.

Show comment
Hide comment
@alfredxing

alfredxing Jan 28, 2015

Member

@kaatt That's weird. Is it like this in the last release (which didn't include incremental regen)?

Member

alfredxing commented Jan 28, 2015

@kaatt That's weird. Is it like this in the last release (which didn't include incremental regen)?

@kaatt

This comment has been minimized.

Show comment
Hide comment
@kaatt

kaatt Jan 28, 2015

Contributor

I don't know about the last release because it didn't include the times but most likely it didn't have any problem.

Contributor

kaatt commented Jan 28, 2015

I don't know about the last release because it didn't include the times but most likely it didn't have any problem.

@kaatt

This comment has been minimized.

Show comment
Hide comment
@kaatt

kaatt Jan 28, 2015

Contributor

There's a serious bug in incremental regen. I just spent a lot of time trying to figure out why the markdown filter wasn't working in a specific page. The full generation time also changed from 4s to 1s. Later when I manually deleted .jekyll-metadata, everything became normal.

Contributor

kaatt commented Jan 28, 2015

There's a serious bug in incremental regen. I just spent a lot of time trying to figure out why the markdown filter wasn't working in a specific page. The full generation time also changed from 4s to 1s. Later when I manually deleted .jekyll-metadata, everything became normal.

@kaatt

This comment has been minimized.

Show comment
Hide comment
@kaatt

kaatt Jan 28, 2015

Contributor

Okay, another bug (or maybe the same one in my last comment). I forced quit jekyll using Ctrl+Z because Ctrl+C was taking too long. Then rm -rf _site and jekyll s. The images folder in my project which just contained image binaries wasn't copied by jekyll. Deleted .jekyll-metadata and jekyll started behaving again.

EDIT: Was able to reproduce with jekyll new, force quit isn't responsible.

Contributor

kaatt commented Jan 28, 2015

Okay, another bug (or maybe the same one in my last comment). I forced quit jekyll using Ctrl+Z because Ctrl+C was taking too long. Then rm -rf _site and jekyll s. The images folder in my project which just contained image binaries wasn't copied by jekyll. Deleted .jekyll-metadata and jekyll started behaving again.

EDIT: Was able to reproduce with jekyll new, force quit isn't responsible.

@kaatt

This comment has been minimized.

Show comment
Hide comment
@kaatt

kaatt Jan 28, 2015

Contributor

Got another bug. Noticed this today and a day ago. Until I delete .jekyll-metadata, jekyll build says:

Liquid Exception: undefined method `[]' for false:FalseClass

Contributor

kaatt commented Jan 28, 2015

Got another bug. Noticed this today and a day ago. Until I delete .jekyll-metadata, jekyll build says:

Liquid Exception: undefined method `[]' for false:FalseClass

@alfredxing

This comment has been minimized.

Show comment
Hide comment
@alfredxing

alfredxing Jan 28, 2015

Member

That second-last one was pretty much expected since Jekyll doesn't check for the existence of the output dir or any of its contents (so incremental won't copy the files over because it doesn't know they were deleted). I'll think about a solution to this, but in the meantime you can use the new jekyll clean command.

The last one is a bug; can you reproduce it while running jekyll build --trace and see what that gives you?

Member

alfredxing commented Jan 28, 2015

That second-last one was pretty much expected since Jekyll doesn't check for the existence of the output dir or any of its contents (so incremental won't copy the files over because it doesn't know they were deleted). I'll think about a solution to this, but in the meantime you can use the new jekyll clean command.

The last one is a bug; can you reproduce it while running jekyll build --trace and see what that gives you?

@kaatt

This comment has been minimized.

Show comment
Hide comment
@kaatt

kaatt Jan 28, 2015

Contributor

The last one is rare, I think it happens sometimes when I force kill jekyll. It happened two times, and both times it said this:

Liquid Exception: undefined method `[]' for false:FalseClass in '_layouts/categories.htm'

The categories.htm layout is for your jekyll-archives gem.

Contributor

kaatt commented Jan 28, 2015

The last one is rare, I think it happens sometimes when I force kill jekyll. It happened two times, and both times it said this:

Liquid Exception: undefined method `[]' for false:FalseClass in '_layouts/categories.htm'

The categories.htm layout is for your jekyll-archives gem.

@alfredxing

This comment has been minimized.

Show comment
Hide comment
@alfredxing

alfredxing Jan 29, 2015

Member

If you can reproduce that last one without force killing, I'll look into it, but generally force killing a process brings about unexpected behaviour.

Member

alfredxing commented Jan 29, 2015

If you can reproduce that last one without force killing, I'll look into it, but generally force killing a process brings about unexpected behaviour.

@parkr

This comment has been minimized.

Show comment
Hide comment
@parkr

parkr Jan 29, 2015

Member

The last one would happen if .jekyll-metadata is emptyZ

Member

parkr commented Jan 29, 2015

The last one would happen if .jekyll-metadata is emptyZ

@alfredxing

This comment has been minimized.

Show comment
Hide comment
@alfredxing

alfredxing Jan 29, 2015

Member

@parkr Thanks for the heads-up. Probably because it didn't write when force killed. I'll see what I can do in initialize.

Member

alfredxing commented Jan 29, 2015

@parkr Thanks for the heads-up. Probably because it didn't write when force killed. I'll see what I can do in initialize.

@eksperimental

This comment has been minimized.

Show comment
Hide comment
@eksperimental

eksperimental Feb 8, 2015

Contributor

@parkr how do you debug jekyll?
i see --trace, but that's for when an error occurs

Contributor

eksperimental commented Feb 8, 2015

@parkr how do you debug jekyll?
i see --trace, but that's for when an error occurs

@eksperimental

This comment has been minimized.

Show comment
Hide comment
@eksperimental

eksperimental Feb 8, 2015

Contributor

Instead of copying one file, jekyll is regenerating everyhing.
I'm updating a css file it's taking 5 second to update that, same time it takes to update any html page

Contributor

eksperimental commented Feb 8, 2015

Instead of copying one file, jekyll is regenerating everyhing.
I'm updating a css file it's taking 5 second to update that, same time it takes to update any html page

@parkr

This comment has been minimized.

Show comment
Hide comment
@parkr

parkr Feb 8, 2015

Member

@parkr how do you debug jekyll?

We should have a DEBUG mode (if you set the env var), but I'm not sure how well it works right now. We should beef it up!

Member

parkr commented Feb 8, 2015

@parkr how do you debug jekyll?

We should have a DEBUG mode (if you set the env var), but I'm not sure how well it works right now. We should beef it up!

@eksperimental

This comment has been minimized.

Show comment
Hide comment
@eksperimental

eksperimental Feb 9, 2015

Contributor

@parkr from what I can see, setting ENV[DEBUG] is the same as calling serve with the --verbose argument

Contributor

eksperimental commented Feb 9, 2015

@parkr from what I can see, setting ENV[DEBUG] is the same as calling serve with the --verbose argument

@parkr

This comment has been minimized.

Show comment
Hide comment
@parkr

parkr Dec 13, 2015

Member

This is a constant issue and is being worked on actively. The root cause is usually garbage collection or memory overhead, where each site rebuild generates millions of objects that require much time to be done away with. This is likely an artifact of this broader issue.

Member

parkr commented Dec 13, 2015

This is a constant issue and is being worked on actively. The root cause is usually garbage collection or memory overhead, where each site rebuild generates millions of objects that require much time to be done away with. This is likely an artifact of this broader issue.

@parkr parkr closed this Dec 13, 2015

@jekyll jekyll locked and limited conversation to collaborators Feb 27, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.