Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move realtime gathering of data from Airflow to a separate small container that runs 24/7 #8

Closed
Tracked by #7
VMois opened this issue Apr 18, 2024 · 0 comments · Fixed by #16
Closed
Tracked by #7

Move realtime gathering of data from Airflow to a separate small container that runs 24/7 #8

VMois opened this issue Apr 18, 2024 · 0 comments · Fixed by #16
Assignees
@VMois
Labels
enhancement New feature or request

Comments

@VMois
Copy link
Owner

VMois commented Apr 18, 2024

If Airflow is redeployed, real-time data collection will be interrupted. In addition, wasting Airflow resources might be unnecessary, as Airflow is not designed for running tasks 24/7.

Making such a service in Golang might be a good option for memory and CPU efficiency. Collecting data and saving it to GCS will be enough.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VMois VMois

Labels
enhancement New feature or request
Projects
MiWaitWay
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add realtime GTFS vehicle position ingestor VMois/miwaitway
1 participant
@VMois