Merge branch 'master' of github.com:pjotrp/once-only

README.md

@@ -2,14 +2,46 @@
 
 [![Build Status](https://secure.travis-ci.org/pjotrp/once-only.png)](http://travis-ci.org/pjotrp/once-only)
 
-Relax with PBS!
+Relax with PBS! 
+
+No worries about running jobs concurrently from the command line (also
+on multi-core).  Once-only is inspired by the Lisp once-only function,
+which wraps another function and calculates a result only once, based
+on the same inputs. Simply prepend your command with once-only:
+
+When running
+
+```bash
+  once-only -d cluster00073 --pbs --in output.best.dnd ~/opt/paml/bin/codeml ~/paml7-8.ctl
+```
+
+This is what you want to see when same the job was executed before
+
+```bash
+  **STATUS** Job 00073codemla4817 already completed!
+```
+
+This is what you see when a job is running
+
+```bash
+  **STATUS** Job 00073codemla4817 is locked!
+```
+
+With PBS, this is what you want to see when a job is already in the queue
+
+```bash
+  **STATUS** Job 00073codemla4817 already in queue!
+```
+
+Features
 
 * Computations only happen once
-* A completed job does not get submitted again to PBS
+* A completed job does not get submitted again (to PBS)
 * A job already in the queue does not get submitted again to PBS
 * A completed job in the PBS queue does not run again
+* A running job is locked
 * Guarantee independently executed jobs
-* Do not worry about submitting serial jobs
+* Do not worry about submitting serial jobs multiple times
 
 and coming
 
@@ -19,12 +51,18 @@ and coming
 Once-only makes a program or script run only *once*, provided the inputs don't
 change (in a functional style!). This is very useful when running a range of
 jobs on a compute cluster or GRID. It may even be useful in the context of
-webservices.  Once-only makes it relaxed to run many jobs on compute clusters!
-A mistake, interruption, or even a parameter tweak, does not mean everything
-has to be run again. When running jobs serially you can just batch
-submit them after getting the first results. Any missed jobs can be
-run later again. This way you can get better utilisation of the
-cluster.
+webservices.  
+
+Once-only makes it relaxed to run many jobs on compute clusters!  A
+mistake, interruption, or even a parameter tweak, does not mean
+everything has to be run again. When running jobs serially you can
+just batch submit them after getting the first results. Any missed
+jobs can be run later again. This way you can get better utilisation
+of your cores or a cluster. You can even use it as a poor-mans PBS on
+your multi-core machine, or over NFS by firing up scripts
+concurrently.
+
+Examples:
 
 Instead of running a tool or script directly, such as
 
@@ -89,12 +127,6 @@ md5sum on the one-only has file, for example
 grep MD5 bio-table-ce4ceee0d2ee08ef235662c35b8238ad47fed030.txt |awk 'BEGIN { FS = "[ \t\n]+" }{ print $2,"",$3 }'|md5sum -c
 ```
 
-Once-only is inspired by the Lisp once-only function, which wraps another
-function and calculates a result only once, based on the same inputs. It is
-also inspired by the NixOS software deployment system, which guarantees
-packages are uniquely deployed, based on the source code inputs and the
-configuration at compile time.
-
 ## Installation
 
 Note: once-only is written in Ruby, but you don't need to understand
@@ -113,8 +145,8 @@ library dependencies.
 
 'md5sum' is used for calculating MD5 hash values.
 
-'pfff' is optional and used for calculating pfff hash values on very
-large files (nyi).
+'pfff' probablistic finger printing is optional and used for
+calculating pfff hash values on large files.
 
 When you are using the --pbs option, once-only will use the 'qsub' and
 'qstat' commands.
@@ -234,6 +266,18 @@ Note that files that come with a path will be stripped of their path
 before execution. When files are very large you may want to consider
 the --scratch option.
 
+### Precalculated hashes
+
+The --precalc option allows for using precalculated hash values. The
+extension says what hash to use. Example:
+
+```sh
+once-only --precalc hash.md5 /bin/cat ~/.bashrc
+```
+
+Once-only will pick up the values from 'hash.md5' and use those after
+making sure the time stamp of the hash file is most recent.
+
 ### Use the scratch disk with --scratch (nyi)
 
 watch this page
@@ -249,6 +293,10 @@ how to contribute, see
 
   http://github.com/pjotrp/once-only
 
+See also the design document in 
+
+  http://github.com/pjotrp/once-only/doc/design.md
+
 ## Cite
 
 If you use this software, please cite 

VERSION

@@ -1 +1 @@
-0.2.1
+0.2.3-pre1

bin/once-only

@@ -19,10 +19,14 @@ Usage:
    --skip-regex regex  skip making checksumes of filenames that match the regex (multiple allowed)
    --skip-glob regex   skip making checksumes of filenames that match the glob (multiple allowed)
    --include|--in file include input filename for making the checksums (file should exist)
+   --precalc file      use precalculated Hash values (extension .md5)
+   --pfff              use a Pfff checksum for files larger than 20MB
    -v                  increase verbosity
    -q                  run quietly
    --debug             give debug information
    --dry-run           do not execute command
+   --ignore-lock       ignore locked files (they expire normally after 5 hours)
+   --ignore-queue      do not check the queue
    --force             force execute command
 
 Examples:
@@ -67,7 +71,7 @@ def exit_error errval = 1, msg = nil
 end
 
 def parse_args(args)
-  options = { :skip => [], :skip_regex => [], :skip_glob => [], :include => [] }
+  options = { :precalc => [], :skip => [], :skip_regex => [], :skip_glob => [], :include => [] }
 
   consume = lambda { |args|
     if not args[0]
@@ -112,6 +116,13 @@ def parse_args(args)
       when '--copy' 
         options[:copy] = true
         consume.call(args[1..-1])
+      when '--precalc'
+        p args
+        options[:precalc] << args[1]
+        consume.call(args[2..-1])
+      when '--pfff' 
+        options[:pfff] = true
+        consume.call(args[1..-1])
       when '-h', '--help'
         print USAGE
         exit 1
@@ -127,6 +138,12 @@ def parse_args(args)
       when '--dry-run' 
         options[:dry_run] = true
         consume.call(args[1..-1])
+      when '--ignore-lock' 
+        options[:ignore_lock] = true
+        consume.call(args[1..-1])
+      when '--ignore-queue' 
+        options[:ignore_queue] = true
+        consume.call(args[1..-1])
       when '--force' 
         options[:force] = true
         consume.call(args[1..-1])
@@ -158,6 +175,10 @@ once_only_args = OnceOnly::Check.drop_pbs_option(once_only_args)
 once_only_args = OnceOnly::Check.drop_dir_option(once_only_args)
 once_only_command = once_only_args.join(' ')
 
+# --- Fetch the pre-calculated checksums
+precalc = OnceOnly::Check.precalculated_checksums(options[:precalc])
+
+# --- Calculate the checksums for the items in the list
 command = args.join(' ')
 command_sorted = args.sort.join(' ')
 command_sha1 = OnceOnly::Check::calc_checksum(command_sorted)
@@ -173,7 +194,10 @@ base_dir = Dir.pwd
 executable = args[0]
 args = args[1..-1] if options[:skip_exe]
 
+# filter all arguments that reflect existing files
 file_list = OnceOnly::Check::get_file_list(args)
+
+# remove filenames that ought to be skipped
 options[:skip_regex].each { |regex|
   file_list = OnceOnly::Check::filter_file_list(file_list,regex)
 }
@@ -186,16 +210,41 @@ OnceOnly::Check::check_files_exist(options[:include])
 file_list += options[:include]
 file_list = file_list.uniq
 
-checksums = OnceOnly::Check::calc_file_checksums(file_list)
+pfff = if options[:pfff]
+         bin = OnceOnly::Check::which('pfff') 
+         if not bin
+           raise "Pfff binary not found. Please install in the PATH before using the --pfff switch."
+         end
+         bin+'/pfff -k 1'
+       end
+
+checksums = OnceOnly::Check::calc_file_checksums(file_list,precalc,pfff)
 checksums.push ['SHA1',command_sha1,command_sorted] if not options[:skip_cli]
 
 # ---- Create filenames
 once_only_filename = OnceOnly::Check::make_once_filename(checksums,File.basename(executable))
 $stderr.print "Check file name ",once_only_filename,"\n" if options[:verbose]
 error_filename = once_only_filename + '.err'
-tag_filename = once_only_filename + '.run'
 $stderr.print "**STATUS** Job file exists ",once_only_filename,"!\n" if options[:debug] and File.exist?(once_only_filename)
 
+# ---- The 'run' file is used to prepare for a job
+tag_filename = once_only_filename + '.run'
+
+# ---- The 'lock' file is used when the job is running
+lock_filename = once_only_filename + '.lock'
+if File.exist?(lock_filename) and not options[:force] and not options[:ignore_lock]
+  $stderr.print "**STATUS** Job is locked with #{lock_filename} '#{original_commands}'!\n" if not options[:quiet]
+  # ----- Sleep for 30 seconds and try again
+  sleep(30)
+  if File.exist?(lock_filename)
+    if File.mtime(lock_filename) < Time.now - 18000
+      $stderr.print "**STATUS ** Lock is stale, retrying now\n"
+    else
+      exit 0
+    end
+  end
+end
+
 # ---- Create job name
 dirname = File.basename(Dir.pwd).rjust(8,"-") # make sure it is long enough
 
@@ -207,7 +256,7 @@ if options[:copy]
   copy_dir = base_dir + '/' + File.basename(once_only_filename,".txt") 
 end
 
-if options[:force] or not File.exist?(once_only_filename)
+if options[:force] or not File.exist?(once_only_filename) 
   $stderr.print "Running #{command}\n" if not options[:quiet]
   OnceOnly::Check::write_file(tag_filename,checksums)
   if options[:pbs]
@@ -225,10 +274,12 @@ if options[:force] or not File.exist?(once_only_filename)
     $stderr.print("PBS command: ",pbs_command,"\n") if options[:verbose]
 
     # --- Check if job is already queued in PBS
-    qstat = `/usr/bin/qstat`
-    if qstat =~ /#{job_name}/
-      $stderr.print "**STATUS** Job #{job_name} already in queue!\n"
-      exit 0
+    if not options[:ignore_queue]
+      qstat = `/usr/bin/qstat`
+      if qstat =~ /#{job_name}/ 
+        $stderr.print "**STATUS** Job #{job_name} already in queue!\n"
+        exit 0
+      end
     end
 
     if !options[:dry_run]
@@ -240,6 +291,7 @@ if options[:force] or not File.exist?(once_only_filename)
   else
     # --- Run on command line
     if !options[:dry_run]
+      File.open(lock_filename, "w") {}
       success = 
         if options[:copy]
           exit_error(1,"Directory #{copy_dir} already exists!") if File.directory?(copy_dir)
@@ -274,6 +326,7 @@ if options[:force] or not File.exist?(once_only_filename)
           system(command)
         end
       Dir.chdir(base_dir) if options[:copy]
+      File.unlink(lock_filename)
       if not success
         OnceOnly::Check::write_file(error_filename,checksums)
         File.unlink(tag_filename) if File.exist?(tag_filename)
@@ -283,7 +336,7 @@ if options[:force] or not File.exist?(once_only_filename)
         File.unlink(error_filename) if File.exist?(error_filename)
         OnceOnly::Check::write_file(once_only_filename,checksums)
         File.unlink(tag_filename) if File.exist?(tag_filename)
-     end
+      end
     end
   end
 else

lib/once-only.rb

@@ -1,4 +1,3 @@
-require 'once-only/once-only.rb'
 require 'once-only/sha1.rb'
 require 'once-only/check.rb'
 

lib/once-only/check.rb

@@ -9,7 +9,11 @@
 module OnceOnly
 
   module Check
-    # filter out all arguments that reflect existing files
+
+    def Check::which(binary)
+      ENV["PATH"].split(File::PATH_SEPARATOR).find {|p| File.exists?( File.join( p, binary ) ) }
+    end
+    # filter all arguments that reflect existing files
     def Check::get_file_list list
       list.map { |arg| get_existing_filename(arg) }.compact
     end
@@ -31,10 +35,40 @@ def Check::filter_file_list_glob list, glob
       list.map { |name| ( Dir.glob(glob).index(name) ? nil : name ) }.compact
     end
 
-    # Calculate the checksums for each file in the list
-    def Check::calc_file_checksums list
+    # Return a hash of files with their hash type, hash value and check time
+    def Check::precalculated_checksums(files)
+      precalc = {}
+      files.each do | fn |
+        dir = File.dirname(fn)
+        raise "Precalculated hash file should have .md5 extension!" if fn !~ /\.md5$/
+        t = File.mtime(fn)
+        File.open(fn).each { |s|
+          a = s.split
+          checkfn = File.expand_path(a[1],dir)
+          precalc[checkfn] = { type: 'MD5', hash: a[0], time: t }
+        }
+      end
+      precalc
+    end
+
+    # Calculate the checksums for each file in the list and return a list
+    # of array - each row containing the Hash type (MD5), the value and the (relative)
+    # file path.
+    def Check::calc_file_checksums list, precalc, pfff
       list.map { |fn|
-        ['MD5'] + `/usr/bin/md5sum #{fn}`.split
+        # First see if fn is in the precalculated list
+        fqn = File.expand_path(fn)
+        if precalc[fqn] and File.mtime(fqn) < precalc[fqn][:time]
+          $stderr.print "Precalculated ",fn,"\n"
+          rec = precalc[fqn]
+          [rec[:type],rec[:hash],fqn]
+        else
+          if pfff and File.size(fn) > 20_000_000 
+            ['PFFF'] + `#{pfff} #{fqn}`.split
+          else
+            ['MD5'] + `/usr/bin/md5sum #{fqn}`.split
+          end
+        end
       }
     end