Support :autoload option and autoload method for autoloading files and directories

This makes it possible to use the same +autoload+ call in all cases, and handle
four separate scenarios:

1. Autoload then reload: Fast development mode startup, loading the minimum
   number of files, but reloading if those files are changed
2. Autoload without reload: Useful for faster testing of a subset of an
   application, so the untested subsets is not loaded.
3. Require then reload: Slower development mode startup, but have entire
   application loaded before accepting requests
4. Require without reload: Normal production/testing mode with nothing autoloaded
   or reloaded

Author: jeremyevans

CHANGELOG

@@ -1,3 +1,7 @@
+= master
+
+* Support :autoload option and autoload method for autoloading files and directories (jeremyevans)
+
 = 2.0.0 (2022-06-23)
 
 * Fix TypeError being raised when requiring a file results in an error (jeremyevans)

README.rdoc

@@ -195,7 +195,7 @@ class.
 
 == Requiring
 
-Rack::Unreloader#require is a little different than require in that it takes
+Rack::Unreloader#require is a little different than Kernel#require in that it takes
 a file glob, not a normal require path.  For that reason, you must specify
 the extension when requiring the file, and it will only look in the current
 directory by default:
@@ -253,6 +253,45 @@ decide that instead of specifying the constants, ObjectSpace should be used to
 automatically determine the constants loaded. You can specify this by having the
 block return the :ObjectSpace symbol.
 
+=== Autoload
+
+To further speed things up in development mode, or when only running a subset of
+tests, it can be helpful to autoload files instead of require them, so that if
+the related constants are not accessed, you don't need to pay the cost of loading
+the related files.  To enable autoloading, pass the +:autoload+ option when
+creating the reloader:
+
+  Unreloader = Rack::Unreloader.new(autoload: true){App}
+
+Then, you can call +autoload+ instead of +require+:
+
+  Unreloader.autoload('models'){|f| File.basename(f).sub(/\.rb\z/, '').capitalize}
+
+This will monitor the models directory for files, setting up autoloads for each
+file.  After the file has been loaded, normal reloading will happen for the
+file. Note that for +autoload+, a block is required because the constant names
+are needed before loading the file to setup the autoload.
+
+If the <tt>reload: false</tt> option is given when creating the reloader,
+autoloads will still be setup by +autoload+, but no reloading will happen. This
+can be useful when testing subsets of an application.  When testing subsets of
+an application, you don't need reloading, but you can benefit from autoloading,
+so parts of the application you are not testing are not loaded.
+
+If you do not pass the +:autoload+ option when creating the reloader, then calls
+to +autoload+ will implicitly be transformed to calls to +require+.  This makes
+it possible to use the same +autoload+ call in all cases, and handle four
+separate scenarios:
+
+1. Autoload then reload: Fast development mode startup, loading the minimum
+   number of files, but reloading if those files are changed
+2. Autoload without reload: Useful for faster testing of a subset of an
+   application, so the untested subsets is not loaded.
+3. Require then reload: Slower development mode startup, but have entire
+   application loaded before accepting requests
+4. Require without reload: Normal production/testing mode with nothing autoloaded
+   or reloaded
+
 == Usage Outside Rack
 
 While +Rack::Unreloader+ is usually in the development of rack applications,

lib/rack/unreloader.rb

@@ -8,9 +8,12 @@ class Unreloader
     # Mutex used to synchronize reloads
     MUTEX = Monitor.new
 
-    # Reference to ::File as File would return Rack::File by default.
+    # Reference to ::File as File may return Rack::File by default.
     File = ::File
 
+    # Regexp for valid constant names, to prevent code execution.
+    VALID_CONSTANT_NAME_REGEXP = /\A(?:::)?([A-Z]\w*(?:::[A-Z]\w*)*)\z/.freeze
+
     # Given the list of paths, find all matching files, or matching ruby files
     # in subdirecories if given a directory, and return an array of expanded
     # paths.
@@ -41,11 +44,51 @@ def self.ruby_files(dir)
       files.sort
     end
 
+    # Autoload the file for the given objects.  objs should be a string, symbol,
+    # or array of them holding a Ruby constant name.  Access to the constant will
+    # load the related file.  A non-nil logger will have output logged to it.
+    def self.autoload_constants(objs, file, logger)
+      strings = Array(objs).map(&:to_s)
+      if strings.empty?
+        # Remove file from $LOADED_FEATURES if there are no constants to autoload.
+        # In general that is because the file is part of another class that will
+        # handle loading the file separately, and if that class is reloaded, we
+        # want to remove the loaded feature so the file can get loaded again.
+        $LOADED_FEATURES.delete(file)
+      else
+        logger.info("Setting up autoload for #{file}: #{strings.join(' ')}") if logger
+        strings.each do |s|
+          obj, mod = split_autoload(s)
+
+          if obj
+            obj.autoload(mod, file)
+          elsif logger
+            logger.info("Invalid constant name: #{s}")
+          end
+        end
+      end
+    end
+
+    # Split the given string into an array. The first is a module/class to add the
+    # autoload to, and the second is the name of the constant to be autoloaded.
+    def self.split_autoload(mod_string)
+      if m = VALID_CONSTANT_NAME_REGEXP.match(mod_string)
+        ns, sep, mod = m[1].rpartition('::')
+        if sep.empty?
+          [Object, mod]
+        else
+          [Object.module_eval("::#{ns}", __FILE__, __LINE__), mod]
+        end
+      end
+    end
+
     # The Rack::Unreloader::Reloader instead related to this instance, if one.
     attr_reader :reloader
 
     # Setup the reloader. Options:
     # 
+    # :autoload :: Whether to allow autoloading.  If not set to true, calls to
+    #              autoload will eagerly require the related files instead of autoloading.
     # :cooldown :: The number of seconds to wait between checks for changed files.
     #              Defaults to 1.  Set to nil/false to not check for changed files.
     # :handle_reload_errors :: Whether reload to handle reload errors by returning
@@ -59,12 +102,19 @@ def self.ruby_files(dir)
     #                match exactly, since modules don't have superclasses.
     def initialize(opts={}, &block)
       @app_block = block
+      @autoload = opts[:autoload]
+      @logger = opts[:logger]
       if opts.fetch(:reload, true)
         @cooldown = opts.fetch(:cooldown, 1)
         @handle_reload_errors = opts[:handle_reload_errors]
         @last = Time.at(0)
-        require_relative 'unreloader/reloader'
-        @reloader = Reloader.new(opts)
+        if @autoload
+          require_relative('unreloader/autoload_reloader')
+          @reloader = AutoloadReloader.new(opts)
+        else
+          require_relative('unreloader/reloader')
+          @reloader = Reloader.new(opts)
+        end
         reload!
       else
         @reloader = @cooldown = @handle_reload_errors = false
@@ -99,6 +149,24 @@ def require(paths, opts={}, &block)
       end
     end
 
+    # Add a file glob or array of file global to autoload and monitor
+    # for changes. A block is required.  It will be called with the
+    # path to be autoloaded, and should return the symbol for the
+    # constant name to autoload. Accepts the same options as #require.
+    def autoload(paths, opts={}, &block)
+      raise ArgumentError, "block required" unless block
+
+      if @autoload
+        if @reloader
+          @reloader.autoload_dependencies(paths, opts, &block)
+        else
+          Unreloader.expand_directory_paths(paths).each{|f| Unreloader.autoload_constants(yield(f), f, @logger)}
+        end
+      else
+        require(paths, opts, &block)
+      end
+    end
+
     # Records that each path in +files+ depends on +dependency+.  If there
     # is a modification to +dependency+, all related files will be reloaded
     # after +dependency+ is reloaded.  Both +dependency+ and each entry in +files+

lib/rack/unreloader/autoload_reloader.rb

@@ -0,0 +1,89 @@
+require_relative 'reloader'
+
+module Rack
+  class Unreloader
+    class AutoloadReloader < Reloader
+      def initialize(opts={})
+        super
+
+        # Files that autoloads have been setup for, but have not yet been loaded.
+        # Hash with realpath keys and values that are arrays with the
+        # a block that will return constant name strings that will autoload the 
+        # file, the modified time the file, and the delete hook.
+        @autoload_files = {}
+
+        # Directories where new files will be setup for autoloading.
+        # Uses same format as @monitor_dirs.
+        @autoload_dirs = {}
+      end
+
+      def autoload_dependencies(paths, opts={}, &block)
+        delete_hook = opts[:delete_hook]
+
+        Unreloader.expand_paths(paths).each do |file|
+          if File.directory?(file)
+            @autoload_dirs[file] = [nil, [], block, delete_hook]
+            check_autoload_dir(file)
+          else
+            # Comparisons against $LOADED_FEATURES need realpaths
+            @autoload_files[File.realpath(file)] = [block, modified_at(file), delete_hook]
+            Unreloader.autoload_constants(yield(file), file, @logger)
+          end
+        end
+      end
+
+      def remove_autoload(file, strings)
+        strings = Array(strings)
+        log("Removing autoload for #{file}: #{strings.join(" ")}") unless strings.empty?
+        strings.each do |s|
+          obj, mod = Unreloader.split_autoload(s)
+          # Assume that if the autoload string was valid to create the
+          # autoload, it is still valid when removing the autoload.
+          obj.send(:remove_const, mod)
+        end
+      end
+
+      def check_autoload_dir(dir)
+        subdir_times, files, block, delete_hook = md = @autoload_dirs[dir]
+        return if subdir_times && subdir_times.all?{|subdir, time| File.directory?(subdir) && modified_at(subdir) == time}
+        md[0] = subdir_times(dir)
+
+        cur_files = Unreloader.ruby_files(dir)
+        return if files == cur_files
+
+        removed_files = files - cur_files
+        new_files = cur_files - files
+
+        # Removed files that were never required should have the constant removed
+        # so that accesses to the constant do not attempt to autoload a file that
+        # no longer exists.
+        removed_files.each do |file|
+          remove_autoload(file, block.call(file)) unless @monitor_files[file]
+        end
+
+        # New files not yet loaded should have autoloads added for them.
+        autoload_dependencies(new_files, :delete_hook=>delete_hook, &block) unless new_files.empty?
+
+        files.replace(cur_files)
+      end
+
+      def reload!
+        (@autoload_files.keys & $LOADED_FEATURES).each do |file|
+          # Files setup for autoloads were autoloaded, move metadata to locations
+          # used for required files.
+          log("Autoloaded file required, setting up reloading: #{file}")
+          block, *metadata = @autoload_files.delete(file)
+          @constants_defined[file] = block
+          @monitor_files[file] = metadata
+          @files[file] = {:features=>Set.new, :constants=>Array(block.call(file))}
+        end
+
+        @autoload_dirs.each_key do |dir|
+          check_autoload_dir(dir)
+        end
+
+        super
+      end
+    end
+  end
+end

lib/rack/unreloader/reloader.rb

@@ -5,9 +5,6 @@ class Unreloader
     class Reloader
       File = ::File
 
-      # Regexp for valid constant names, to prevent code execution.
-      VALID_CONSTANT_NAME_REGEXP = /\A(?:::)?([A-Z]\w*(?:::[A-Z]\w*)*)\z/.freeze
-
       # Setup the reloader.  Supports :logger and :subclasses options, see
       # Rack::Unloader.new for details.
       def initialize(opts={})

spec/spec_helper.rb

@@ -89,9 +89,13 @@ def log_match(*logs)
 
   after do
     ru.reloader.clear! if ru.reloader
+    dirname = File.dirname(__FILE__)
+    keep_files = %w'unreloader_spec.rb spec_helper.rb'
+    $LOADED_FEATURES.delete_if{|f| f.start_with?(dirname) && !keep_files.include?(File.basename(f))}
     Object.send(:remove_const, :RU)
     Object.send(:remove_const, :App) if defined?(::App)
     Object.send(:remove_const, :App2) if defined?(::App2)
+    Object.send(:remove_const, :B) if defined?(::B)
     Dir['spec/app*.rb'].each{|f| File.delete(f)}
   end
 end