Sort static files just once. #3204

parkr · 2014-12-11T20:49:12Z

Ref: #2075.

parkr · 2014-12-11T20:49:21Z

@gjtorikian can you try this?

gjtorikian · 2014-12-11T21:46:44Z

Shaved about forty seconds! I'll take it!

parkr · 2014-12-11T21:51:03Z

😮 Wooooow. Can you run stackprof again with this code and see what else is causing the bottleneck? 40 seconds of a 5-min build is a big win, but it shouldn't take more than 1 min to compile the entire site.

The sorting should probably also take place after generate... hm.

gjtorikian · 2014-12-11T21:52:54Z

lib/jekyll/static_file.rb

@@ -15,6 +17,7 @@ def initialize(site, base, dir, name, collection = nil)
      @dir  = dir
      @name = name
      @collection = collection
+      @relative_path = File.join(*[@dir, @name].compact)


Would this ever have a nil value? Just curious about the compact.

if both @dir and @name are nil, then it'd be File.join() with nothing. It should be theoretically impossible for @name to be empty, but @dir can be.

gjtorikian · 2014-12-11T22:11:05Z

To be clear, here's what's going on in my build:

It's five minutes total, but we have several collections and several static files, not to mention minifying JS
One collection is a clear bottleneck, taking two and a half minutes to complete
Zoning in on just this one collection, it takes a minute-and-a-half to build (the other minute is coming from somewhere else, not dealing with it now)

That's where I discovered this sort issue.

Here are the top three rows from stackprof:

      3761  (16.0%)        3761  (16.0%)     Jekyll::StaticFile#relative_path
      2461  (10.5%)        2453  (10.4%)     Set#initialize_copy
      2306   (9.8%)        1075   (4.6%)     block in Jekyll::URL#generate_url

Clear drop-off after Set#initialize_copy.

Interestingly, that relative_path now is coming from the sitemap plugin:

def html_files
  @site.static_files.select { |file| File.extname(file.relative_path) == ".html" }
end

If I remove this plugin, I gain just a few more seconds.

The other one seems to come from site.rb:

def documents
  collections.reduce(Set.new) do |docs, (_, collection)|
    docs + collection.docs + collection.files
end.to_a

parkr · 2014-12-11T22:28:04Z

We could cache the extname call for sure... As for the #documents call, how often is it being called itself?

gjtorikian · 2014-12-11T22:40:25Z

2501    98.85%  block in Jekyll::Site#documents
16  0.63%   block in Sass::Selector::SimpleSequence#do_extend
13  0.51%   Sass::Selector::SimpleSequence#with_more_sources

I think that is what you wanted? 😟 I'm not sure! We can also put this PR on pause and talk IRL.

parkr · 2014-12-11T23:14:16Z

2501 98.85% block in Jekyll::Site#documents

WHOA. Site#documents is called 2500 times?! geepers creepers. Let's talk IRL when you have time.

gjtorikian · 2014-12-12T00:17:14Z

If you saw the site you'd probably weep. That's not such a crazy figure: https://help.github.com/sitemap.xml

parkr · 2014-12-18T08:12:29Z

@gjtorikian Thanks for taking the time to check these. Can you try out d10a941? :)

gjtorikian · 2014-12-18T16:19:51Z

Wow:

$ script/benchmark 
Building took 81.17 seconds (1.35 minutes)

For reference, here's the exact same site--all of it!--using the copy of Jekyll provided by github-pages (2.4.0):

$ script/benchmark 
Building took 232.16 seconds (3.87 minutes)

This is better than fantastic. I can run another stackprof if you like, if you think that'll suss out more perf gains. 😄

parkr · 2014-12-18T17:03:35Z

HOLY MOLY! Ok cool. Yeah another stackprof would be splendid.

gjtorikian · 2014-12-18T18:23:44Z

For full disclosure, the benchmark script is simply:

time = Benchmark.realtime do
  `jekyll build --config=_config.yml,_config.dev.yml`
end

Here's the stackprof:

     TOTAL    (pct)     SAMPLES    (pct)     FRAME
     21072  (39.1%)       21072  (39.1%)     Jekyll::StaticFile#relative_path
     27623  (51.3%)        6551  (12.2%)     block in Jekyll::Site#site_payload
      4594   (8.5%)        4580   (8.5%)     Set#initialize_copy
      2331   (4.3%)        1080   (2.0%)     block in Jekyll::URL#generate_url
      1124   (2.1%)         813   (1.5%)     URI::Escape#escape
      5159   (9.6%)         750   (1.4%)     Liquid::Context#variable
       680   (1.3%)         680   (1.3%)     block (2 levels) in Gem::BasicSpecification#contains_requirable_file?
      5783  (10.7%)         624   (1.2%)     Liquid::Context#resolve
       613   (1.1%)         613   (1.1%)     Jekyll::Utils#slugify
       533   (1.0%)         533   (1.0%)     Liquid::Context#has_interrupt?
       509   (0.9%)         508   (0.9%)     Jekyll::Convertible#converters
       507   (0.9%)         507   (0.9%)     Jekyll::EntryFilter#symlink?
       498   (0.9%)         482   (0.9%)     Set#add
       442   (0.8%)         442   (0.8%)     Jekyll::Tags::IncludeTag#resolved_includes_dir
       550   (1.0%)         346   (0.6%)     block in FileUtils::Entry_#copy_file
      7301  (13.6%)         331   (0.6%)     block in Jekyll::Site#read_directories
       311   (0.6%)         311   (0.6%)     URI::Parser#escape
       297   (0.6%)         297   (0.6%)     Nokogiri::XML::Document#decorate
      1131   (2.1%)         283   (0.5%)     FileUtils#fu_each_src_dest0
       267   (0.5%)         267   (0.5%)     Nokogiri::XML::XPathContext#register_namespaces
       253   (0.5%)         253   (0.5%)     Liquid::Context#[]=
       251   (0.5%)         251   (0.5%)     FileUtils#fu_mkdir
       794   (1.5%)         244   (0.5%)     FileUtils::Entry_#copy_file
     27392  (50.8%)         238   (0.4%)     block in Liquid::Block#render_all
       232   (0.4%)         232   (0.4%)     block in Jekyll::Convertible#output_ext
       220   (0.4%)         220   (0.4%)     block in Jekyll::URL#sanitize_url
       204   (0.4%)         204   (0.4%)     block (2 levels) in FileUtils::Entry_#copy_file
       178   (0.3%)         178   (0.3%)     Jekyll::StaticFile#path
      1201   (2.2%)         177   (0.3%)     block in Jekyll::Site#site_payload
       323   (0.6%)         176   (0.3%)     Jekyll::Document#read

I can pass the file along to you too if it helps.

parkr · 2014-12-18T23:54:48Z

That stackprof looks old – can you re-run with the latest code here please? :)

Also your benchmark could be full-Ruby:

require 'jekyll'
time = Benchmark.realtime do
  Jekyll::Commands::Build.process({ "config" => %w[_config.yml _config.dev.yml] })
end

parkr · 2014-12-19T01:45:57Z

I LOVE THE PARALLEL GEM IT IS AMAZING

~/jekyll/jekyll#sort-static-files-once$ time be jekyll build -s site
Configuration file: site/_config.yml
            Source: site
       Destination: /Users/parker/jekyll/jekyll/_site
      Generating...
                    done.
 Auto-regeneration: disabled. Use --watch to enable.

real    0m1.414s
user    0m1.986s
sys 0m1.220s
~/jekyll/jekyll#sort-static-files-once$ git co master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
~/jekyll/jekyll#master$ time be jekyll build -s site
Configuration file: site/_config.yml
            Source: site
       Destination: /Users/parker/jekyll/jekyll/_site
      Generating...
                    done.
 Auto-regeneration: disabled. Use --watch to enable.

real    0m6.303s
user    0m6.014s
sys 0m0.285s
~/jekyll/jekyll#master$

gjtorikian · 2014-12-19T02:00:49Z

Hah. Let me know when you need a rerun. I'll hold off for now.

alfredxing · 2014-12-19T02:39:25Z

✨😮✨

parkr · 2014-12-19T07:52:30Z

Turns out we have to restructure some stuff in order to use parallel.

Ref: #2075.

gjtorikian · 2014-12-28T22:10:32Z

Definitely a gigantic improvement here. Thanks a ton! Running off of master:

The build took 72.82 seconds (1.21 minutes)

Unfortunately it looks like Liquid-3 is handling tags a bit differently. I know this isn't the right issue for that, but I'm not sure where to post this:

It looks like YAML keys with digits in them are no longer allowed. For example, in a data file, we have a key called whatever.2fa.whatever. Liquid throws a syntax error at the unexpected number. This was never a problem in 2.6.1.
Liquid seems to be parsing vendor/JS files, too. It threw an exception at this line in our Bower'ed mustache.js file: https://github.com/janl/mustache.js/blob/2ba7481d41e356869dc2db695ccc4a0cf3ce38cd/mustache.js#L97

parkr · 2015-01-02T20:42:22Z

It looks like YAML keys with digits in them are no longer allowed.

This could have to do with the strict parser. If you search for strict in lib/jekyll.rb and remove that line, does it work?

Liquid seems to be parsing vendor/JS files, too

If these files are include_relative'd anywhere, then this'll happen as a result of the rendering process. :/

mattr- · 2015-01-02T22:18:50Z

Liquid seems to be parsing vendor/JS files, too

If these files are include_relative'd anywhere, then this'll happen as a result of the rendering process. :/

That would mean that we'd need to work around this change in behavior before releasing Liquid 3 in a Jekyll 2.x version, correct?

parkr · 2015-01-02T22:33:39Z

That would mean that we'd need to work around this change in behavior before releasing Liquid 3 in a Jekyll 2.x version, correct?

We'd have to have like {% include_relative blah.js norender %} – I think we manually render in the IncludeTag but not sure off the top of my head. I don't know if it affects Jekyll 2 – this is a Jekyll 3 PR.

gjtorikian · 2015-01-08T17:35:00Z

This could have to do with the strict parser. If you search for strict in lib/jekyll.rb and remove that line, does it work?

Yeeeeeep, that was it; turning strict off made it happy. Whatever, though, I'm fine with it not accepting numbers.

We'd have to have like {% include_relative blah.js norender %}

shrug janl/mustache.js#418

In the meantime, it's just one character.

parkr added the fix label Dec 11, 2014

gjtorikian reviewed Dec 11, 2014
View reviewed changes

gjtorikian mentioned this pull request Dec 11, 2014

Option to not build locally? jekyll/jekyll-sitemap#60

Closed

parkr force-pushed the sort-static-files-once branch from 460e377 to c10c642 Compare December 18, 2014 03:04

parkr added 2 commits December 26, 2014 22:23

Sort static files just once.

0eb2796

Ref: #2075.

Only call #site_payload once for all collection work.

2f8248a

parkr force-pushed the sort-static-files-once branch from 3beeb56 to db3eebb Compare December 27, 2014 03:30

Call it site_payload instead of payload.

b6641be

parkr force-pushed the sort-static-files-once branch from db3eebb to b6641be Compare December 27, 2014 03:40

parkr added a commit that referenced this pull request Dec 27, 2014

Merge pull request #3204 from jekyll/sort-static-files-once

f81601e

parkr merged commit f81601e into master Dec 27, 2014

parkr deleted the sort-static-files-once branch December 27, 2014 04:05

parkr added a commit that referenced this pull request Dec 27, 2014

Update history to reflect merge of #3204 [ci skip]

657b165

ghost referenced this pull request Feb 5, 2015

Update history to reflect merge of #3360 [ci skip]

128427c

wjdp mentioned this pull request Oct 27, 2015

Upgrade to Jekyll3 newtheatre/history-project#118

Merged

jekyll locked and limited conversation to collaborators Feb 27, 2017

jekyllbot added the frozen-due-to-age label Feb 27, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sort static files just once. #3204

Sort static files just once. #3204

parkr commented Dec 11, 2014

parkr commented Dec 11, 2014

gjtorikian commented Dec 11, 2014

parkr commented Dec 11, 2014

gjtorikian Dec 11, 2014

parkr Dec 11, 2014

gjtorikian commented Dec 11, 2014

parkr commented Dec 11, 2014

gjtorikian commented Dec 11, 2014

parkr commented Dec 11, 2014

gjtorikian commented Dec 12, 2014

parkr commented Dec 18, 2014

gjtorikian commented Dec 18, 2014

parkr commented Dec 18, 2014

gjtorikian commented Dec 18, 2014

parkr commented Dec 18, 2014

parkr commented Dec 19, 2014

gjtorikian commented Dec 19, 2014

alfredxing commented Dec 19, 2014

parkr commented Dec 19, 2014

gjtorikian commented Dec 28, 2014

parkr commented Jan 2, 2015

mattr- commented Jan 2, 2015

parkr commented Jan 2, 2015

gjtorikian commented Jan 8, 2015

Sort static files just once. #3204

Sort static files just once. #3204

Conversation

parkr commented Dec 11, 2014

parkr commented Dec 11, 2014

gjtorikian commented Dec 11, 2014

parkr commented Dec 11, 2014

gjtorikian Dec 11, 2014

Choose a reason for hiding this comment

parkr Dec 11, 2014

Choose a reason for hiding this comment

gjtorikian commented Dec 11, 2014

parkr commented Dec 11, 2014

gjtorikian commented Dec 11, 2014

parkr commented Dec 11, 2014

gjtorikian commented Dec 12, 2014

parkr commented Dec 18, 2014

gjtorikian commented Dec 18, 2014

parkr commented Dec 18, 2014

gjtorikian commented Dec 18, 2014

parkr commented Dec 18, 2014

parkr commented Dec 19, 2014

gjtorikian commented Dec 19, 2014

alfredxing commented Dec 19, 2014

parkr commented Dec 19, 2014

gjtorikian commented Dec 28, 2014

parkr commented Jan 2, 2015

mattr- commented Jan 2, 2015

parkr commented Jan 2, 2015

gjtorikian commented Jan 8, 2015