Skip to content

Feat: do recursive glob matching for ignore_patterns#3539

Merged
georgesittas merged 1 commit intomainfrom
jo/glob_recursively_ignore_patterns
Dec 19, 2024
Merged

Feat: do recursive glob matching for ignore_patterns#3539
georgesittas merged 1 commit intomainfrom
jo/glob_recursively_ignore_patterns

Conversation

@georgesittas
Copy link
Copy Markdown
Collaborator

Fixes #3507

@georgesittas georgesittas requested a review from a team December 19, 2024 17:44
@georgesittas georgesittas force-pushed the jo/glob_recursively_ignore_patterns branch from a5a9f87 to db5fc28 Compare December 19, 2024 17:45
Comment thread sqlmesh/core/loader.py Outdated
@georgesittas georgesittas force-pushed the jo/glob_recursively_ignore_patterns branch from db5fc28 to ed0df51 Compare December 19, 2024 19:50
Comment thread sqlmesh/core/loader.py
Comment on lines +266 to +277
# We try to match both ignore_pattern itself and every file returned by glob,
# so that we will always ignore file names that do not appear in the latter.
ignored_filepaths = set(ignore_patterns) | {
ignored_path
for ignore_pattern in ignore_patterns
for ignored_path in glob.glob(str(self._context.path / ignore_pattern), recursive=True)
}
for filepath in path.glob(f"**/*{extension}"):
for ignore_pattern in ignore_patterns:
if filepath.match(ignore_pattern):
break
else:
yield filepath
if any(filepath.match(ignored_filepath) for ignored_filepath in ignored_filepaths):
continue

yield filepath
Copy link
Copy Markdown
Collaborator Author

@georgesittas georgesittas Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored to pre-compute the set of ignored file paths. See other thread re: performance implications.

@georgesittas georgesittas requested review from a team and tobymao December 19, 2024 19:53
@georgesittas georgesittas merged commit 2adf5a1 into main Dec 19, 2024
@georgesittas georgesittas deleted the jo/glob_recursively_ignore_patterns branch December 19, 2024 23:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ignore_patterns in config.yaml does not glob recursively

3 participants