Skip to content
This repository has been archived by the owner on Oct 2, 2023. It is now read-only.

modify postgres archiving and add ceqr school template #388

Merged
merged 18 commits into from
Apr 26, 2023

Conversation

damonmcc
Copy link
Member

@damonmcc damonmcc commented Apr 21, 2023

resolves #370, resolves #371

changes

  • modify ingest.py logic when destination in postgres:
    • create a schema for the dataset if it doesn't exist
    • write the data as a table named after the version
    • update the "latest" table to be a view of the new data
  • modify related tests files:
    • still use nypl_libraries data in tests but rename it to test_nypl_libraries for clarity and to avoid conflicts
    • update tests to expect new schema/versioned tables postgres approach
    • improve tests to use less hardcoded strings

notes

@damonmcc damonmcc marked this pull request as draft April 21, 2023 18:08
@damonmcc damonmcc assigned damonmcc and unassigned damonmcc Apr 23, 2023
@damonmcc damonmcc added enhancement New feature or request data update Related to a data product update labels Apr 24, 2023
@damonmcc damonmcc marked this pull request as ready for review April 25, 2023 14:30
@mbh329 mbh329 self-requested a review April 25, 2023 14:47
@damonmcc damonmcc changed the title add ceqr school templates modify postgres archiving and add ceqr school template Apr 25, 2023
@@ -0,0 +1,32 @@
dataset:
name: &name sca_e_projections
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might make sense if we tested one of these datasets on different machine? Do you have the raw files handy and want to send one over?

Copy link
Contributor

@mbh329 mbh329 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took a look at EDM-DATA cluster and the new files used for ceqr are available in the EDM-DATA/recipes DB

@damonmcc damonmcc merged commit 3803454 into main Apr 26, 2023
1 check passed
@damonmcc damonmcc deleted the dm-add-ceqr-school-templates branch May 1, 2023 20:37
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
data update Related to a data product update enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create issue template for sca_e_pct for ceqr app Create issue template for sca_e_projections for ceqr app
3 participants