Permalink
Browse files

optimization: write static file to the destination only if source fil…

…e timestamp differs

Also make sure static files get regenerated when they are missing in destination.

This is useful in --server --auto mode when it reduces disk/cpu load and also plays nice with xrefresh-server (which was my main motivation) -> soft CSS refresh works again!
  • Loading branch information...
1 parent ace9911 commit f91954be7635fe2354c11d45981d5c0045f22010 @darwin darwin committed Mar 18, 2010
Showing with 110 additions and 4 deletions.
  1. +52 −4 lib/jekyll/static_file.rb
  2. +58 −0 test/test_site.rb
View
@@ -1,6 +1,8 @@
module Jekyll
class StaticFile
+ @@mtimes = Hash.new # the cache of last modification times [path] -> mtime
+
# Initialize a new StaticFile.
# +site+ is the Site
# +base+ is the String path to the <source>
@@ -15,13 +17,59 @@ def initialize(site, base, dir, name)
@name = name
end
- # Write the static file to the destination directory.
+ # Obtains source file path.
+ #
+ # Returns source file path.
+ def path
+ File.join(@base, @dir, @name)
+ end
+
+ # Obtain destination path.
# +dest+ is the String path to the destination dir
#
- # Returns nothing
+ # Returns destination file path.
+ def destination(dest)
+ File.join(dest, @dir, @name)
+ end
+
+ # Obtain mtime of the source path.
+ #
+ # Returns last modifiaction time for this file.
+ def mtime
+ File.stat(path).mtime.to_i
+ end
+
+ # Is source path modified?
+ #
+ # Returns true if modified since last write.
+ def modified?
+ @@mtimes[path] != mtime
@qiqi789

qiqi789 Jul 1, 2010

@@mtimes only exists when --server --auto is on. Could you make this optimization patch work for just simple case in which only running jekyll. I usually upload the resulted _site on to my web server via rsync. But everytime all the posts even those without modification will always be transferred, which is inefficient.

One alternative modification at this point is that how about we compare the modified time of a post file between its version in _posts and that in _site left by last time. If the post wasn't changed, its modified time in _posts will be earlier than that in _site, otherwise it must be changed. If it's changed, then we transformed it into _site again.

Hows this sound to you?
Thanks.

+ end
+
+ # Write the static file to the destination directory (if modified).
+ # +dest+ is the String path to the destination dir
+ #
+ # Returns false if the file was not modified since last time (no-op).
def write(dest)
- FileUtils.mkdir_p(File.join(dest, @dir))
- FileUtils.cp(File.join(@base, @dir, @name), File.join(dest, @dir, @name))
+ dest_path = destination(dest)
+ dest_dir = File.join(dest, @dir)
+
+ return false if File.exist? dest_path and !modified?
+ @@mtimes[path] = mtime
+
+ FileUtils.mkdir_p(dest_dir)
+ FileUtils.cp(path, dest_path)
+
+ true
+ end
+
+ # Reset the mtimes cache (for testing purposes).
+ #
+ # Returns nothing.
+ def self.reset_cache
+ @@mtimes = Hash.new
+
+ nil
end
end
View
@@ -34,6 +34,64 @@ class TestSite < Test::Unit::TestCase
assert before_time <= @site.time
end
+ should "write only modified static files" do
+ clear_dest
+ StaticFile.reset_cache
+
+ @site.process
+ some_static_file = @site.static_files[0].path
+ dest = File.expand_path(@site.static_files[0].destination(@site.dest))
+ mtime1 = File.stat(dest).mtime.to_i # first run must generate dest file
+
+ # need to sleep because filesystem timestamps have best resolution in seconds
+ sleep 1
+ @site.process
+ mtime2 = File.stat(dest).mtime.to_i
+ assert_equal mtime1, mtime2
+
+ # simulate file modification by user
+ FileUtils.touch some_static_file
+
+ sleep 1
+ @site.process
+ mtime3 = File.stat(dest).mtime.to_i
+ assert_not_equal mtime2, mtime3 # must be regenerated!
+
+ sleep 1
+ @site.process
+ mtime4 = File.stat(dest).mtime.to_i
+ assert_equal mtime3, mtime4 # no modifications, so must be the same
+ end
+
+ should "write static files if not modified but missing in destination" do
+ clear_dest
+ StaticFile.reset_cache
+
+ @site.process
+ some_static_file = @site.static_files[0].path
+ dest = File.expand_path(@site.static_files[0].destination(@site.dest))
+ mtime1 = File.stat(dest).mtime.to_i # first run must generate dest file
+
+ # need to sleep because filesystem timestamps have best resolution in seconds
+ sleep 1
+ @site.process
+ mtime2 = File.stat(dest).mtime.to_i
+ assert_equal mtime1, mtime2
+
+ # simulate destination file deletion
+ File.unlink dest
+
+ sleep 1
+ @site.process
+ mtime3 = File.stat(dest).mtime.to_i
+ assert_not_equal mtime2, mtime3 # must be regenerated and differ!
+
+ sleep 1
+ @site.process
+ mtime4 = File.stat(dest).mtime.to_i
+ assert_equal mtime3, mtime4 # no modifications, so must be the same
+ end
+
should "read layouts" do
@site.read_layouts
assert_equal ["default", "simple"].sort, @site.layouts.keys.sort

3 comments on commit f91954b

@@mtimes only exists when --server --auto is on. Could you make this optimization patch work for just simple case in which only running jekyll. I usually upload the resulted _site on to my web server via rsync. But everytime all the posts even those without modification will always be transferred, which is inefficient.

One alternative modification at this point is that how about we compare the modified time of a post file between its version in _posts and that in _site left by last time. If the post wasn't changed, its modified time in _posts will be earlier than that in _site, otherwise it must be changed. If it's changed, then we transformed it into _site again.

Hows this sound to you?
Thanks.

Contributor

darwin replied Jul 1, 2010

You should use rsync switch to compare checksums instead of modification time. Look for --checksum option.

Problem solved.

Thanks for the quick answer. I will check it out.

Please sign in to comment.