Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexing steps are not marked stale when new indexable files are added #65

Closed
pydsigner opened this issue Nov 19, 2023 · 3 comments · Fixed by #59
Closed

Indexing steps are not marked stale when new indexable files are added #65

pydsigner opened this issue Nov 19, 2023 · 3 comments · Fixed by #59
Assignees
Labels
bug Something isn't working
Milestone

Comments

@pydsigner
Copy link
Owner

pydsigner commented Nov 19, 2023

The variation of processed files for most Steps consists in the files gathered by the containing Rule's Matcher, with any other dependencies remaining consistent from one run to another or defined within the entry-point file itself. For example, JinjaExtendedMarkdownStep will enter for each markdown file matched, then pull in either the default template from the Step configuration or the template defined in the markdown file's frontmatter. This explicit connection ensures that the output HTML will be regenerated any time either the markdown or the Jinja template changes.

In contrast, a Step that generates an index of files, like the CodeIndexStep proposed in #59, enters from the Jinja template itself, and establishes custody connections only to markdown files that exist when a fresh run occurs. If the only change that occurs is an added markdown file, the Step will not have a chance to gather the new file, leading to false up-to-date decisions.

@pydsigner pydsigner added the bug Something isn't working label Nov 19, 2023
@pydsigner pydsigner added this to the 1.0.0 milestone Nov 19, 2023
@pydsigner
Copy link
Owner Author

Possibilities for correction:

  • Add a way for Steps to include wildcard dependencies in their explicit custody information, so they can be marked dirty when new files appear
  • Add a method to Step so Steps can perform additional dirty checks
  • Workaround with an enumeration Step that matches each indexed file and appends it to a manifest file for the index Step to ingest, rather than embedding a Matcher in the index Step.

@pydsigner
Copy link
Owner Author

Perhaps the most sensible approach is to utilize the support for non-Path upstreams to produce a built-in glob upstream checker for the Custodian, which would be marked stale if the glob returned a different set of files.

@pydsigner
Copy link
Owner Author

For now, since the issue does not appear in core Steps, we can just address this in #59.

@pydsigner pydsigner linked a pull request Nov 19, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant