Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dimension Upsert #74

Open
Tracked by #90
Thejagd27 opened this issue May 2, 2023 · 1 comment
Open
Tracked by #90

Dimension Upsert #74

Thejagd27 opened this issue May 2, 2023 · 1 comment
Assignees
Labels
backlog This issue is part of the development backlog

Comments

@Thejagd27
Copy link

Thejagd27 commented May 2, 2023

@ChakshuGautam Please review the approach and suggest if any changes required
Issue with Dimension Upsert

If we upsert the dimension data using /ingestion/dimension API , the data in the DB will be updated but that updated data will not be there in CSV file
So if we re run the yarn cli ingest then all the old data will be lost and we will not have updated data
 
Proposed Approach for Dimension Upsert

Create a API /ingestion/dimension, accepts csv file as input, the name of the csv file should be same as dimension name
Validates all the required columns for that table
If there is any error in column then sends the appropriate error and stops the ingestion
If there is any error in the records then those records will be added to error file and uploaded to ingestion_error
Else Insertion will be done on conflict updation will be done to the table
Once all the records are upserted we need to get all the records from the table, create a CSV file write all the records to the CSV file
Upload that CSV file to processed_input->dimensions folder
NOTE-> Assuming we will get data for all the columns for that dimension table except the auto generated ID column

@techsavvyash techsavvyash added the backlog This issue is part of the development backlog label Oct 7, 2023
@techsavvyash techsavvyash self-assigned this Oct 7, 2023
@techsavvyash
Copy link
Collaborator

#101 implemented event upsertion which lays the foundation for this issue as well, port the implementation in #101 to include dimensions as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backlog This issue is part of the development backlog
Projects
None yet
Development

No branches or pull requests

2 participants