Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .rubocop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,9 @@ Metrics/MethodLength:
Metrics/BlockLength:
Exclude:
- 'spec/**/*'

Metrics/CyclomaticComplexity:
Max: 15

Metrics/PerceivedComplexity:
Max: 15
21 changes: 14 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ Why fastclone?
Doing lots of repeated checkouts on a specific machine?

| Repository | 1st Fastclone | 2nd Fastclone | git clone | cp -R |
| -----------|---------------|---------------|-----------|-------|
| angular.js | 8s | 3s | 6s | 0.5s |
| bootstrap | 26s | 3s | 11s | 0.2s |
| gradle | 25s | 9s | 19s | 6.2s |
| linux | 4m 53s | 1m 6s | 3m 51s | 29s |
| react.js | 18s | 3s | 8s | 0.5s |
| tensorflow | 19s | 4s | 8s | 1.5s |
| ---------- | ------------- | ------------- | --------- | ----- |
| angular.js | 8s | 3s | 6s | 0.5s |
| bootstrap | 26s | 3s | 11s | 0.2s |
| gradle | 25s | 9s | 19s | 6.2s |
| linux | 4m 53s | 1m 6s | 3m 51s | 29s |
| react.js | 18s | 3s | 8s | 0.5s |
| tensorflow | 19s | 4s | 8s | 1.5s |

Above times captured using `time` without verbose mode.

Expand Down Expand Up @@ -51,6 +51,8 @@ Usage
--lock-timeout N Timeout in seconds to acquire a lock on any reference repo.
Default is 0 which waits indefinitely.
--pre-clone-hook command An optional command that should be invoked before cloning mirror repo
--sparse-paths PATHS Comma-separated list of paths for sparse checkout.
Enables sparse checkout mode using git sparse-checkout.

Change the default `REFERENCE_REPO_DIR` environment variable if necessary.

Expand All @@ -66,6 +68,11 @@ The hook is invoked with given arguments:
1. path to the repo mirror location
1. attempt number, 0-indexed

Sparse checkout support
-----------------------

In passing `--sparse-paths`, git-fastclone will instead perform a sparse checkout, where the passed list of paths will be set up as patterns. This can be useful if you're interested only in a subset of paths in the repository.

How to test?
------------
Manual testing:
Expand Down
52 changes: 48 additions & 4 deletions lib/git-fastclone.rb
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ class Runner

attr_accessor :reference_dir, :prefetch_submodules, :reference_updated, :reference_mutex,
:options, :abs_clone_path, :using_local_repo, :verbose, :print_git_errors, :color,
:flock_timeout_secs
:flock_timeout_secs, :sparse_paths

def initialize
# Prefetch reference repos for submodules we've seen before
Expand Down Expand Up @@ -115,6 +115,8 @@ def initialize
self.color = false

self.flock_timeout_secs = 0

self.sparse_paths = nil
end

def run
Expand Down Expand Up @@ -172,6 +174,12 @@ def parse_options
'No-op when a file is missing') do |script_file|
options[:pre_clone_hook] = script_file
end

opts.on('--sparse-paths PATHS',
'Comma-separated list of paths for sparse checkout.',
'Enables sparse checkout mode using git sparse-checkout.') do |paths|
self.sparse_paths = paths.split(',').map(&:strip)
end
end.parse!
end

Expand Down Expand Up @@ -199,6 +207,16 @@ def parse_inputs
raise msg
end

# Validate that --branch is specified when using --sparse-paths
if sparse_paths && !options[:branch]
msg = "Error: --branch is required when using --sparse-paths\n" \
"Sparse checkouts need an explicit branch/revision to checkout.\n" \
'Usage: git-fastclone --sparse-paths <paths> --branch <branch> <url>'
raise msg.red if color

raise msg
end

self.reference_dir = ENV['REFERENCE_REPO_DIR'] || DEFAULT_REFERENCE_REPO_DIR
FileUtils.mkdir_p(reference_dir)

Expand Down Expand Up @@ -234,13 +252,23 @@ def clone(url, rev, src_dir, config)
clear_clone_dest_if_needed(attempt_number, clone_dest)

clone_commands = ['git', 'clone', verbose ? '--verbose' : '--quiet']
clone_commands << '--reference' << mirror.to_s << url.to_s << clone_dest
# For sparse checkouts, clone directly from the local mirror and skip the actual checkout process
# For normal clones, use --reference and clone from the remote URL
if sparse_paths
clone_commands.push('--no-checkout')
clone_commands << mirror.to_s << clone_dest
else
clone_commands << '--reference' << mirror.to_s << url.to_s << clone_dest
Copy link
Member

@gyfelton gyfelton Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if i do a full checkout then do it again with sparse checkout on the same clone_dest, what happens? for example will just one path get new updated version based on the branch, where the rest are the old version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the same clone destination already exists, then git-fastclone won't do the clone – that's how it already works. I think this is still the right way to handle things, rather than trying to deal with possibly updating an existing clone.

Example of doing two different checkouts of the same already-cached repo:

$ rm -rf bazel-sparse && be git fastclone --sparse-paths .bazelci -b master https://github.com/bazelbuild/bazel bazel-sparse; rm -rf bazel-full && be git fastclone -b master "https://github.com/bazelbuild/bazel" bazel-full

git-fastclone 1.6.0
Cloning bazel to /Users/tsutton/Development/git-fastclone/bazel-sparse
Checkout of bazel-sparse took 0.84033s

git-fastclone 1.6.0
Cloning bazel to /Users/tsutton/Development/git-fastclone/bazel-full
Checkout of bazel-full took 2.324625s

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right that makes total sense, i guess another small issue is if whoever used the new feature want to know if the checkout is full or not... but that's up to them to decide anyway

end
clone_commands << '--config' << config.to_s unless config.nil?
fail_on_error(*clone_commands, quiet: !verbose, print_on_failure: print_git_errors)

# Configure sparse checkout if enabled
perform_sparse_checkout(clone_dest, rev) if sparse_paths
end

# Only checkout if we're changing branches to a non-default branch
if rev
# Only checkout if we're changing branches to a non-default branch (for non-sparse clones)
if !sparse_paths && rev
fail_on_error('git', 'checkout', '--quiet', rev.to_s, quiet: !verbose,
print_on_failure: print_git_errors,
chdir: File.join(abs_clone_path, src_dir))
Expand All @@ -258,6 +286,22 @@ def clone(url, rev, src_dir, config)
end
end

def perform_sparse_checkout(clone_dest, rev)
puts 'Configuring sparse checkout...' if verbose

# Initialize sparse checkout with cone mode
fail_on_error('git', 'sparse-checkout', 'init', '--cone',
quiet: !verbose, print_on_failure: print_git_errors, chdir: clone_dest)

# Set the sparse paths
fail_on_error('git', 'sparse-checkout', 'set', *sparse_paths,
quiet: !verbose, print_on_failure: print_git_errors, chdir: clone_dest)

# Checkout the specified branch/revision
fail_on_error('git', 'checkout', '--quiet', rev.to_s,
quiet: !verbose, print_on_failure: print_git_errors, chdir: clone_dest)
end

def update_submodules(pwd, url)
return unless File.exist?(File.join(abs_clone_path, pwd, '.gitmodules'))

Expand Down
2 changes: 1 addition & 1 deletion lib/git-fastclone/version.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@

# Version string for git-fastclone
module GitFastCloneVersion
VERSION = '1.5.1'
VERSION = '1.6.0'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't have changelog.md huh

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hehe no, I looked for the same thing

end