This started as a way to quickly and natively ruby-ly parse gitignore files and find matching files. It's now gained an equivalent includes file functionality, ARGV awareness, and some shebang matching, while still being extremely fast, to be a one-stop file-list for your linter.
Filter a directory tree using a .gitignore file. Recognises all of the gitignore rules
FastIgnore.new(relative: true).sort == `git ls-files`.split("\n").sort
- Fast (faster than using
`git ls-files`.split("\n")
for small repos (because it avoids the overhead of``
)) - Supports ruby 2.5-3.1.x & jruby
- supports all gitignore rule patterns
- doesn't require git to be installed
- supports a gitignore-esque "include" patterns. (
include_rules:
/include_files:
) - supports an expansion of include patterns, expanding and anchoring paths (
argv_rules:
) - supports matching by shebang rather than filename for extensionless files:
#!:
- reads .gitignore in all subdirectories
- reads .git/info/excludes
- reads the global gitignore file mentioned in your git config
Add this line to your application's Gemfile:
gem 'fast_ignore'
And then execute:
$ bundle
Or install it yourself as:
$ gem install fast_ignore
FastIgnore.new.each { |file| puts "#{file} is not ignored by the .gitignore file" }
This yields paths that are not ignored by the gitignore, i.e. the paths that would be returned by git ls-files
.
A FastIgnore instance is an Enumerable and responds to all Enumerable methods:
FastIgnore.new.to_a
FastIgnore.new.map { |file| file.upcase }
Like other enumerables, FastIgnore#each
can return an enumerator:
FastIgnore.new.each.with_index { |file, index| puts "#{file}#{index}" }
Warning: Do not change directory (e.g. Dir.chdir
) in the block.
To check if a single path is allowed, use
FastIgnore.new.allowed?('relative/path')
FastIgnore.new.allowed?('./relative/path')
FastIgnore.new.allowed?('/absolute/path')
FastIgnore.new.allowed?('~/home/path')
Relative paths will be considered relative to the root:
directory, not the current directory.
This is aliased as ===
so you can use a FastIgnore instance in case statements.
@path_matcher ||= FastIgnore.new
case my_path
when @path_matcher
puts(my_path)
end
It's recommended to save the FastIgnore instance to a variable to avoid having to read and parse the gitignore file and gitconfig files repeatedly.
If your code already knows the path to test is/not a directory or wants to lie about whether it is/is not a directory, you can pass directory: true
or directory: false
as an argument to allowed?
(to have FastIgnore ask the file system, you can pass directory: nil
or nothing)
FastIgnore.new.allowed?('relative/path', directory: false) # matches `path` as a file
FastIgnore.new.allowed?('relative/path', directory: true) # matches `path` as a directory
FastIgnore.new.allowed?('relative/path', directory: nil) # matches path as whatever it is on the filesystem
FastIgnore.new.allowed?('relative/path) # or as a file if it doesn't exist on the file system
default: nil
If your code already knows the path to test is has a particular text content or wants to lie about the content, you can pass directory: true
or directory: false
as an argument to allowed?
(to have FastIgnore ask the file system, you can pass directory: nil
or nothing)
FastIgnore.new.allowed?('relative/path', content: "#!/usr/bin/env ruby\n\nputs 'hello'") # matches ruby shebang
FastIgnore.new.allowed?('relative/path', content: "#!/usr/bin/env bash\n\necho 'hello'") # matches bash shebang
FastIgnore.new.allowed?('relative/path', content: nil) # matches path as whatever content is on the filesystem
FastIgnore.new.allowed?('relative/path) # or as an empty file if it doesn't actually exist
default: nil
If your code already knows the path to test is has a particular text content or wants to lie about the content, you can pass directory: true
or directory: false
as an argument to allowed?
(to have FastIgnore ask the file system, you can pass directory: nil
or nothing)
FastIgnore.new.allowed?('relative/path', content: "#!/usr/bin/env ruby\n\nputs 'hello'") # matches ruby shebang
FastIgnore.new.allowed?('relative/path', content: "#!/usr/bin/env bash\n\necho 'hello'") # matches bash shebang
FastIgnore.new.allowed?('relative/path', content: nil) # matches path as whatever content is on the filesystem
FastIgnore.new.allowed?('relative/path) # or as an empty file if it doesn't actually exist
default: nil
If your code already knows the path to test exists or wants to lie about its existence, you can pass exists: true
or exists: false
as an argument to allowed?
(to have FastIgnore ask the file system, you can pass exists: nil
or nothing)
FastIgnore.new.allowed?('relative/path', exists: true) # will check the path regardless of whether it actually truly exists
FastIgnore.new.allowed?('relative/path', exists: false) # will always return false
FastIgnore.new.allowed?('relative/path', exists: nil) # asks the filesystem
FastIgnore.new.allowed?('relative/path) # asks the filesystem
default: false
By default a file must not be a directory for it to be considered allowed. This is intended to match the behaviour of git ls-files
which only lists files.
To match directories you can pass include_directories: true
to allowed?
FastIgnore.new.allowed?('relative/path', include_directories: true) # will test the path even if it's a directory
FastIgnore.new.allowed?('relative/path', include_directories: false) # will always return false if the path is a directory
FastIgnore.new.allowed?('relative/path) # will always return false if the path is a directory
Default: false
When relative: false
: FastIgnore#each will yield full paths.
When relative: true
: FastIgnore#each will yield paths relative to the root:
directory
FastIgnore.new(relative: true).to_a
Default: Dir.pwd ($PWD, the current working directory)
This directory is used for:
- the location of
.git/core/exclude
- the ancestor of all non-global automatically loaded
.gitignore
files - the root directory for array rules (
ignore_rules:
,include_rules:
,argv_rules:
) containing/
- the path that
relative:
is relative to - the ancestor of all paths yielded by
#each
- the path that
#allowed?
considers relative paths relative to - the ancestor of all
include_files:
andignore_files:
To use a different directory:
FastIgnore.new(root: '/absolute/path/to/root').to_a
FastIgnore.new(root: '../relative/path/to/root').to_a
A relative root will be found relative to the current working directory when the FastIgnore instance is initialized, and that will be the last time the current working directory is relevant.
Note: Changes to the current working directory (e.g. with Dir.chdir
), after initialising a FastIgnore instance, will not affect the FastIgnore instance. root:
will always be what it was when the instance was initialized, even as a default value.
Default: true
When gitignore: true
: the .gitignore file in the root:
directory is loaded, plus any .gitignore files in its subdirectories, the global git ignore file as described in git config, and .git/info/exclude. .git
directories are also excluded to match the behaviour of git ls-files
.
When gitignore: false
: no ignore files or git config files are automatically read, and .git
will not be automatically excluded.
FastIgnore.new(gitignore: false).to_a
This is a list of files in the gitignore format to parse and match paths against, not a list of files to ignore If you want an array of files use ignore_rules:
Additional gitignore-style files, either as a path or an array of paths.
You can specify other gitignore-style files to ignore as well.
Missing files will raise an Errno::ENOENT
error.
Relative paths are relative to the root:
directory.
Absolute paths also need to be within the root:
directory.
FastIgnore.new(ignore_files: 'relative/path/to/my/ignore/file').to_a
FastIgnore.new(ignore_files: ['/absolute/path/to/my/ignore/file', '/and/another']).to_a
Note: the location of the files will affect rules beginning with or containing /
.
To avoid raising Errno::ENOENT
when the file doesn't exist:
FastIgnore.new(ignore_files: ['/ignore/file'].select { |f| File.exist?(f) }).to_a
This can be a string, or an array of strings, and multiline strings can be used with one rule per line.
FastIgnore.new(ignore_rules: '.DS_Store').to_a
FastIgnore.new(ignore_rules: ['.git', '.gitkeep']).to_a
FastIgnore.new(ignore_rules: ".git\n.gitkeep").to_a
These rules use the root:
argument to resolve rules containing /
.
This is an array of files in the gitignore format to parse and match paths against, not a list of files to include. If you want an array of files use include_rules:
.
Building on the gitignore format, FastIgnore also accepts rules to include matching paths (rather than ignoring them). A rule matching a directory will include all descendants of that directory.
These rules can be provided in files either as absolute or relative paths, or an array of paths.
Relative paths are relative to the root:
directory.
Absolute paths also need to be within the root:
directory.
FastIgnore.new(include_files: 'my_include_file').to_a
FastIgnore.new(include_files: ['/absolute/include/file', './relative/include/file']).to_a
Missing files will raise an Errno::ENOENT
error.
To avoid raising Errno::ENOENT
when the file doesn't exist:
FastIgnore.new(include_files: ['include/file'].select { |f| File.exist?(f) }).to_a
Note: All paths checked must not be excluded by any ignore files AND each included by include file separately AND the include_rules:
AND the argv_rules:
. see Combinations for solutions to using OR.
Building on the gitignore format, FastIgnore also accepts rules to include matching paths (rather than ignoring them). A rule matching a directory will include all descendants of that directory.
This can be a string, or an array of strings, and multiline strings can be used with one rule per line.
FastIgnore.new(include_rules: %w{my*rule /and/another !rule}, gitignore: false).to_a
Rules use the root:
argument to resolve rules containing /
.
Note: All paths checked must not be excluded by any ignore files AND each included by include file separately AND the include_rules:
AND the argv_rules:
. see Combinations for solutions to using OR.
This is like include_rules:
with additional features meant for dealing with humans and ARGV
values.
It expands rules that are absolute paths, and paths beginning with ~
, ../
and ./
(with and without !
).
This means rules beginning with /
are absolute. Not relative to root:
.
Additionally it assumes all rules are relative to the root:
directory (after resolving absolute paths) unless they begin with *
(or !*
).
This can be a string, or an array of strings, and multiline strings can be used with one rule per line.
FastIgnore.new(argv_rules: ['./a/pasted/path', '/or/a/path/from/stdin', 'an/argument', '*.txt']).to_a
Warning: it will not expand e.g. /../
in the middle of a rule that doesn't begin with any of ~
,../
,./
,/
.
Note: All paths checked must not be excluded by any ignore files AND each included by include file separately AND the include_rules:
AND the argv_rules:
. see Combinations for solutions to using OR.
Sometimes you need to match files by their shebang/hashbang/etc rather than their path or filename
Rules beginning with #!:
will match whole words in the shebang line of extensionless files.
e.g.
#!:ruby
will match shebang lines: #!/usr/bin/env ruby
or #!/usr/bin/ruby
or #!/usr/bin/ruby -w
e.g.
#!:bin/ruby
will match #!/bin/ruby
or #!/usr/bin/ruby
or #!/usr/bin/ruby -w
Only exact substring matches are available, There's no special handling of * or / or etc.
These rules can be supplied any way regular rules are, whether in a .gitignore file or files mentioned in include_files:
or ignore_files:
or include_rules:
or ignore_rules:
or argv_rules:
FastIgnore.new(include_rules: ['*.rb', '#!:ruby']).to_a
FastIgnore.new(ignore_rules: ['*.sh', '#!:sh', '#!:bash', '#!:zsh']).to_a
Note: git considers rules like this as a comment and will ignore them.
In the simplest case a file must be allowed by each ignore file, each include file, and each array of rules. That is, they are combined using AND
.
To combine files using OR
, that is, a file may be matched by either file it doesn't have to be referred to in both:
provide the files as strings to include_rules:
or ignore_rules:
FastIgnore.new(include_rules: [File.read('/my/path'), File.read('/another/path')])).to_a
This does unfortunately lose the file path as the root for rules containing /
.
If that's important, combine the files in the file system and use include_files:
or ignore_files:
as normal.
To use the additional ARGV
handling of argv_rules:
on a file, read the file into the array.
FastIgnore.new(argv_rules: ["my/rule", File.read('/my/path')]).to_a
This does unfortunately lose the file path as the root /
and there is no workaround except setting the root:
for the whole FastIgnore instance.
-
Doesn't know what to do if you change the current working directory inside the
FastIgnore#each
block. So don't do that.(It does handle changing the current working directory between
FastIgnore#allowed?
calls) -
FastIgnore always matches patterns case-insensitively. (git varies by filesystem).
-
FastIgnore always outputs paths as literal UTF-8 characters. (git depends on your core.quotepath setting but by default outputs non ascii paths with octal escapes surrounded by quotes).
-
git has a system-wide config file installed at
$(prefix)/etc/gitconfig
, whereprefix
is defined for git at install time. FastIgnore assumes that it will always be/usr/local/etc/gitconfig
. if it's important your system config file is looked at, as that's where you have the core.excludesfile defined, use git's built-in way to override this by addingexport GIT_CONFIG_SYSTEM='/the/actual/location'
to your shell profile. -
Because git looks at its own index objects and FastIgnore looks at the file system there may be some differences between FastIgnore and
git ls-files
. To avoid these differences you may want to use thegit_ls
gem instead- Tracked files that were committed before the matching ignore rule was committed will be returned by
git ls-files
, but not by FastIgnore. - Untracked files will be returned by FastIgnore, but not by
git ls-files
- Deleted files whose deletions haven't been committed will be returned by
git ls-files
, but not by FastIgnore - On a case insensitive file system, with files that differ only by case,
git ls-files
will include all case variations, while FastIgnore will only include whichever variation git placed in the file system. - FastIgnore is unaware of submodules and just treats them like regular directories. For example:
git ls-files --recurse-submodules
won't use the parent repo's gitignore on a submodule, while FastIgnore doesn't know it's a submodule and will. - FastIgnore will only return the files actually on the file system when using
git sparse-checkout
.
- Tracked files that were committed before the matching ignore rule was committed will be returned by
Bug reports and pull requests are welcome on GitHub at https://github.com/robotdana/fast_ignore.
Some tools that may help:
bin/setup
: install development dependenciesbundle exec rspec
: run all testsbundle exec rake
: run all tests and lintersbin/console
: open apry
console with everything required for experimentingbin/ls [argv_rules]
: the equivalent ofgit ls-files
bin/prof/ls [argv_rules]
: ruby-prof report forbin/ls
bin/prof/parse [argv_rules]
: ruby-prof report for parsing root and global gitignore files and any arguments.bin/time [argv_rules]
: the average time for 30 runs ofbin/ls
This repo is too small to stress bin/time more than 0.01s, switch to a large repo and find the average time before and after changes.bin/compare
: compare the speed and output of FastIgnore andgit ls-files
. (suppressing differences that are because of known limitations)
The gem is available as open source under the terms of the MIT License.