HRQB 15 - Load static data from HR #34
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghukill
force-pushed
the
HRQB-15-libhr-table
branch
from
1ce8711 to
fb08645
Compare
Why these changes are being introduced: HRQBClient needs to support a unique pipeline that will load static data from HR into a Quickbase table. This pipeline will be run quite rarely, and would be suitable to do so manually from the CLI locally; an initial load and potentially bulk updates from HR. After that, this table will be managed directly in QB. The data in this table is utilized by other tasks and pipelines. How this addresses that need: * Creates new UpdateLibHRData pipeline and associated tasks Side effects of this change: * Ability to load and update LibHREmployeeAppointments table via HRQBClient Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-15
Why these changes are being introduced: Ideally, whenever data is pulled for use in a transform task, this will occur via an extract task for separation of concerns. How this addresses that need: * Moves QBClient for Departments data into extract task Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-15
ghukill
force-pushed
the
HRQB-15-libhr-table
branch
from
fb08645 to
9a69f4f
Compare
ghukill
commented
May 30, 2024
Comment on lines
+55
to
+56
| libhr_df = self.named_inputs["ExtractLibHREmployeeAppointments"].read() | ||
| departments_df = self.named_inputs["ExtractQBDepartments"].read() |
There was a problem hiding this comment.
This is first time this has been used. This task requires two parent tasks, therefore we cannot use self.single_input_dataframe. This is utilizing the self.named_inputs to get the target data from the parent tasks by those task names.
ehanson8
approved these changes
May 30, 2024
| """Pipeline to load Library HR employee appointment data from static CSV file. | ||
| This pipeline loads the table 'LibHR Employee Appointments', which contains | ||
| information known only by Library HR, that we cannot get from the data warehouse, |
There was a problem hiding this comment.
You might capitalize data warehouse for consistency
jonavellecuerdo
approved these changes
May 31, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose and background context
This PR introduces a new pipeline
UpdateLibHRData. UnlikeFullUpdate, this pipeline is anticipated to get run very infrequently, but is needed. This pipeline will load static data provided by HR into a table calledLibHR Employee Appointments. This pipeline will be used for an initial load of data, and potentially for bulk updates requested by HR, but this table will primarily be managed directly in Quickbase.How can a reviewer manually see the effects of these changes?
Still in discussion with HR about credentials and access, so it may be some time before we have readily accessible ways to test run these pipelines for everyone.
That said, here is an example CLI command to invokes it, passing a parameter for a local CSV file:
And the output:
Includes new or updated dependencies?
NO
Changes expectations for external applications?
NO
What are the relevant tickets?
Developer
Code Reviewer(s)