Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transform raw data to prod to be used in features later #21

Closed
VMois opened this issue May 5, 2024 · 0 comments · Fixed by #24
Closed

Transform raw data to prod to be used in features later #21

VMois opened this issue May 5, 2024 · 0 comments · Fixed by #24
Assignees
Labels
epic Kinda like milestone. Collection of PRs, other issues, etc. related to a common topic

Comments

@VMois
Copy link
Owner

VMois commented May 5, 2024

BQ has powerful functions that work with GEOGRAPHY data. I will need them to calculate the average wait time on a stop.
To use those functions, I need to convert raw latitude and longitude coordinates to BQ native Geography Point. In particular, I care about two tables: vehicle_position and stops. vehicle_position contains bus GPS coordinates + timestamp,
stops contains the GPS coordinates of the stops.

The aim of this epic is to design DAGs that transform lat and lon columns in vehicle_position and stops tables to BQ Geography Point. Transformed data will be saved as new tables in the prod stage (together with not-transformed data). This aligns with the second column in the diagram from the data flow article - https://vmois.dev/data-flow-bigquery-miwaitway/. Later, using those prod tables, I will be able to calculate average wait time, etc.

@VMois VMois added the epic Kinda like milestone. Collection of PRs, other issues, etc. related to a common topic label May 5, 2024
@VMois VMois self-assigned this May 5, 2024
@VMois VMois closed this as completed in #24 May 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
epic Kinda like milestone. Collection of PRs, other issues, etc. related to a common topic
Projects
Status: ✅ Done
Development

Successfully merging a pull request may close this issue.

1 participant