Update main from HRQB-17 and HRQB-18 #30
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Why these changes are being introduced: When working on a single task, it can be helpful to run the task in the context of a pipeline, but only that task. How this addresses that need: * Adds --start-task option to pipeline run CLI command * Further documents and error handles for running pipeline tasks Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-17
Why these changes are being introduced: When bulk upserting records, Quickbase will respond with every field error, for all records. If a single field is incorrectly handled across all records, this can result in thousands of log lines. How this addresses that need: * Refactor logging errors in Quickbase upsert response * Group and count error types for logging Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-17
Why these changes are being introduced: hrqb.tasks.pipelines should be dedicated to HRQBPipelineTasks. How this addresses that need: * Moves the helper functions to run tasks via luigi to a dedicated file Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-17
Why these changes are being introduced: It is helpful when developing on a task to have the ability to programatically remove the output target data from only that task. This should also apply when including the --cleanup flag for a pipeline run, if a single task is specified, where only the target data for that task would be removed. How this addresses that need: * Normalize --task argument on remove-data and run CLI commands * Updates docstrings and README to match Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-17
Why these changes are being introduced: When running sub-commands for the pipeline command, e.g. remove-data or run, you can specifiy a target task to focus on from the pipeline. Formerly this was a CLI arg on the sub-commands, but this could be hoisted into the parent pipeline command and normalize behavior. How this addresses that need: * Moves --task CLI arg to pipeline group command Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-17
HRQB 17 - pipeline task isolation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR #27 was supposed to have its target branch changed to
main, but was merged before that happened. The PR was approved, so merging tomainnow via this PR.