Create an "always stale" target that generates the hash table of parser functions #242
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
December 3, 2021 08:40
limnoliver
approved these changes
Dec 6, 2021
lib/src/generic_utils.R
Outdated
| #' @details this function is used to keep a dependency stale always, assuming this function is called whenever | ||
| #' that file is referenced and there is at least a paired target use of the file. | ||
| make_file_stale <- function(filepath){ | ||
| cat(file = filepath, paste0(format(Sys.time(), '%m/%d/%y %H:%M:%S; random ID: '), sample(1E6, 1))) |
There was a problem hiding this comment.
What's the purpose of the random ID (as opposed to just MM-DD-YYYY H:M:S) ?
There was a problem hiding this comment.
Thanks - I added a commit with more documentation explaining this. The random number is there because we want
to write something unique to the file, and there is a potential that this make_file_stale() function could be called several times in one second. Without the random ID added to it, the calls mad within the same second won't modify the file uniquely, and then it won't make the other targets stale.
added 2 commits
December 6, 2021 07:23
This was referenced Dec 8, 2021
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a partial for #231 , and focuses on solving a single problematic part of the pull: there is a target that generates the hash table of parser functions, but required manual intervention to trigger an update (the prior way was to edit the
dummystring). This was generating issues, because if you made an edit to a parser function or added a new one, you wouldn't necessarily send those changes on to the parsing of the files. You'd discover this when you got an error because there was a cooperator-added file that didn't have a parser ("but I added that file!") and then realize you need to go back and change the dummy argument. The worse case was when you made a bug fix to a parser and thought you propagated the changes withscmake()but the changes weren't recognized because they are effectively hidden fromremake(the parser files are referenced by a directory of files, not the individual files in the recipe).This is a
scipiper-specific fix which does introduce some new codes and concepts even though I think we're doing most of the new stages and refactors withtargets. But I found this issue to be too error-prone to have it keep biting us, so this could be a solution moving forward or could be replaced if the coop section of this pipeline is refactored withtargetsdue to the stage's continued use/development and the other brittle components that still exist and are or will be documented in #231What this solution does is force
7a_temp_coop_munge/tmp/parser_files.ymlandcoop_parsersto both be stale on every single build. These are quick targets to generate, so this doesn't cost much. If neither of these targets actually change after rebuild, the result of the task table isn't rebuilt too, so that works as expected: