DocPad got slower around v6.30, 6.31, 6.32

[neilbaylorrulez]

Hi there.

I am running a highly-customized docpad site with about 150 pages.

I've been running version 6.30.3 for quite a while now, and on average, on my dev machine, file generation takes about 2.5 seconds.

Today, I upgraded to the latest version of docpad, and file generation jumped to nearly 15 seconds!

As such, I did some research as to which versions have "slowed" down generation for me.

Here are my results:

6.30 - ~2.5s
6.31- ~5s
6.32 - ~14s
6.33 - ~15s
6.34 - ~15s

I noticed that if I run version 6.34 with "docpad run -d", my console will hang on "Adding document: DOCUMENT NAME" for a few seconds before carrying on

However, the document that it's "hanging" on is well-formed and deleting that document does not significantly speed up generation (it just hangs on a different document).

Any ideas?
Fast generation is imperative to the success of our project.

Here are the detailed Results:

version: 6.30.3
info: Watching setup
info: The action completed successfully
info: Regenerating at 15:46:40
info: Generating...
info: Generated all 143 files in 2.679 seconds
info: Regenerated at 15:46:43
info: Regenerating at 15:46:50
info: Generating...
info: Generated all 143 files in 2.511 seconds
info: Regenerated at 15:46:53

version: 6.31.0
upgrade instructions: http://docpad.org/upgrade
info: Watching setup
info: The action completed successfully
info: Regenerating at 15:53:34
info: Generating...
info: Generated all 143 files in 5.33 seconds
info: Regenerated at 15:53:39
info: Regenerating at 15:53:44
info: Generating...
info: Generated all 143 files in 5.159 seconds
info: Regenerated at 15:53:50

version: 6.31.6
upgrade instructions: http://docpad.org/upgrade
info: Watching setup
info: The action completed successfully
info: Regenerating at 15:51:16
info: Generating...
info: Generated all 143 files in 5.594 seconds
info: Regenerated at 15:51:21
info: Regenerating at 15:51:26
info: Generating...
info: Generated all 143 files in 5.543 seconds
info: Regenerated at 15:51:32

version: 6.32.0
upgrade instructions: http://docpad.org/upgrade
info: Watching setup
info: The action completed successfully
info: Regenerating at 15:55:52
info: Generating...
info: Generated all 143 files in 13.412 seconds
info: Regenerated at 15:56:05
info: Regenerating at 15:56:10
info: Generating...
info: Generated all 143 files in 14.133 seconds
info: Regenerated at 15:56:24

version: 6.33.0
upgrade instructions: http://docpad.org/upgrade
info: Watching setup
info: The action completed successfully
info: Regenerating at 15:59:33
info: Generating...
info: Generated all 143 files in 15.034 seconds
info: Regenerated at 15:59:48
info: Regenerating at 16:00:22
info: Generating...
info: Generated all 143 files in 16.182 seconds
info: Regenerated at 16:00:39

version: 6.34.2
info: Generating...
info: Generated all 143 files in 15.274 seconds
info: Watching setup starting...
info: Watching setup
info: The action completed successfully
info: Regenerating at 16:04:43
info: Generating...
info: Generated all 143 files in 15.01 seconds
info: Regenerated at 16:04:58

[balupton]

That's really strange, as for the DocPad website, generation has gotten faster for me. Can you give me access to your repo and I'll debug when I get a chance.

Looking at the version differences, the change from 6.30 to v6.31 (3 second increase) was the removal of using relativePath as document IDs, and by v6.31.6 (.5 second increase) we also added outPath change listener to detect collisions.

The change from v6.31 to v6.32 (10 second increase) was a move to Caterpillar v2 (now streams based), very strange it would cause such a performance hit.

What version of Node.js are you on, as well as what platform/os and architecture?

[dimitarkolev]

I can confirm the speed problems i believe that i saw major decrease in
speed when progress bars were introduced (not sure that it has anything to
do with them but it is what i have noticed)

On Fri, May 17, 2013 at 8:00 AM, Benjamin Arthur Lupton <
notifications@github.com> wrote:

That's really strange, as for the DocPad website, generation has gotten
faster for me. Can you give me access to your repo and I'll debug when I
get a chance.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/508#issuecomment-18043855
.

[pismute]

I have about 400 files including images. My site takes 20 seconds over. I found out a trick for writing and debugging.

I added w environment in docpad.coffee:

  # Environments
  # Allows us to set custom configuration for specific environments
  environments:
    w:  # for writing and debugging
      ignoreCustomPatterns: /2008|2009|2010|2011|2012/ # ignore old articles.

In writing and debugging some bugs, I do 'docpad -e w run'. Docpad ignores old documents.

ignoreCustomPatterns is regex pattern. I wonder Docpad takes a regex pattern instead of a testing function. If It is a function, I will make me more easy building complex ignore pattern.

Ref #451

[balupton]

Can someone provide a repo which incurs the steep performance hit?

I've run the different versions on the docpad website skeleton, and the performance hit is bad, but not as steep as the other reports.

^C05:42:50:balupton@balair.local:/Users/balupton/Projects/bevry/docpad-sites/docpad-website:master
$ docpad generate
info: Fetching Contributors...
info: Fetched Contributors
info: Welcome to DocPad v6.30.5
info: Plugins: coffeekup, coffeescript, eco, feedr, highlightjs, livereload, partials, repocloner, robotskirt, services, sitemap, stylus, text
info: Environment: development
info: Generating...
info: Updating DocPad Documentation...
notice: There's a new version of DocPad available, you should probably upgrade...
current version:          6.30.5
new version:              6.42.2
upgrade instructions:     http://docpad.org/upgrade
From https://github.com/bevry/docpad-documentation
 * branch            master     -> FETCH_HEAD
Already up-to-date.
info: Updated DocPad Documentation
info: Skipped ignored file: docs/README.md
info: Currently on writeFiles at 18%
info: Generated all 60 files in 14.377 seconds
info: The action completed successfully
^C05:43:36:balupton@balair.local:/Users/balupton/Projects/bevry/docpad-sites/docpad-website:master
$ ./node_modules/.bin/docpad run
info: Fetching Contributors...
^C05:43:43:balupton@balair.local:/Users/balupton/Projects/bevry/docpad-sites/docpad-website:master
$ ./node_modules/.bin/docpad generate
info: Fetching Contributors...
info: Fetched Contributors
info: Welcome to DocPad v6.42.2
info: Plugins: coffeekup, coffeescript, eco, feedr, highlightjs, livereload, partials, repocloner, robotskirt, services, sitemap, stylus, text
info: Environment: development
From https://github.com/bevry/docpad-documentation
 * branch            master     -> FETCH_HEAD
Already up-to-date.
info: Generating...
info: Updating DocPad Documentation...
info: Updated DocPad Documentation
info: Skipped ignored file: docs/README.md
info: Generated all 60 files in 19.496 seconds
info: The action completed successfully

[neilbaylorrulez]

Hey Benjamin

I can provide you with this, would you like to set up a time or do you want to just clone my repo?

Thanks!
Neil

[balupton]

For now I'll just clone it out and try to reproduce :) 👍

[dospuntocero]

do you have a bitbucket account benjamin?

francisco arenas
dospuntocero.cl
fb.me/dospuntocero.cl

2013/6/25 Benjamin Arthur Lupton notifications@github.com

For now I'll just clone it out and try to reproduce :) [image: 👍]

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/508#issuecomment-20002224
.

[balupton]

yeah, balupton and b@lupton.cc are my ids for everything

[dospuntocero]

nice. just invited to my repo. mine turns crazy after load some pages then
you cant do anything.. that happens on my macosx machine at least lots of
weird errors on watch, etc.

francisco arenas
dospuntocero.cl
fb.me/dospuntocero.cl

2013/6/25 Benjamin Arthur Lupton notifications@github.com

yeah, balupton and b@lupton.cc are my ids for everything

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/508#issuecomment-20007419
.

[neilbaylorrulez]

@balupton My companies' repo is private and I am not the administrator, I will get you access today and shoot you a url to clone

Thanks!

[balupton]

I was able to cut off 5 seconds of the DocPad website regeneration with the v6.46 branch, bringing my time to 20 seconds, instead of the 25 seconds of v6.42 and the 15 seconds of v6.30 - which is a good 50% difference.

[quirkyjack]

I can second that, i'm using docpad for a project and saw a multiple fold increase in build time in a recent version. Haven't had the chance to debug further but would love to help with any issues.

[balupton]

@quirkyjack @jlukic considering the change I just pushed up to you, do you think it is related to user error or docpad error?

[simonh1000]

I have the same issue I think. regeneration begins and at a certain moment (e.g. at 2/9 docs) it pauses for several seconds 'fetching' something. https://github.com/hotbelgo/losvast.git

[dospuntocero]

i left using docpad because of the same issue. unusable because of speed. even with a bare repo or one of the skeletons from scratch (jade skeleton for example) it took like 4-5 seconds with the bare one. if you add any css, less or sass you just cant work like that. i moved to gulp (gulpjs.com) all compiles to the µs

im using a 2.7 core i7 with 16gb ram laptop

[dimitarkolev]

Unfortunately it seems that docpad development efforts are going into something different than performance. I noticed the performance degradation happing release after release from much longer than v6. In the release cycle you are referring it just got awful. I love docpad but changing a stylesheet and having the result in your browser should happen in less than 1s not 20s and more even for the most complicated site.

[balupton]

@MehdiK has reported that generations are taking more than a minute, this needs to be looked into: MehdiK/www.mehdi-khalili.com#1

[balupton]

To be honest, considering all the complaining that some people are doing about speed on various issues here, I'd love it if all that energy spent complaining went to something more productive like implementing #336 or the other suggestions at #529 to actually solve this issue.

Alternatively, Bevry does accept sponsorship to focus on particular issues. Generally at $500USD/day or $2000USD/week. Alternatively, bountysource.com lets you place bounties on issues that are important to you.

[simonh1000]

Have to say that speed is much better now - thanks!

[balupton]

Going to close this issue as there is no action task assigned to it. See #529 for relevant action tasks.