🚀 Optimize Menno data download and processing workflow#1939
Conversation
There was a problem hiding this comment.
Pull request overview
This PR aims to optimize the Menno data download and processing workflow by adding a file existence check to avoid redundant downloads, moving the CSV path construction earlier in the function, and cleaning up commented-out code.
Key changes:
- Added file existence check before processing to skip download if CSV already exists
- Moved CSV path construction earlier in the function flow
- Removed commented-out code and unnecessary comments
| if _, err := os.Stat(csvPath); err == nil { | ||
| log.Printf("CSV file already exists, skipping download: %s", csvPath) | ||
| return // Skip download if the file exists | ||
| } | ||
|
|
There was a problem hiding this comment.
This early return skips the entire data processing pipeline when the CSV file exists. The function should not only download the file but also process it and populate the database. The file existence check at line 128 returns early, preventing the CSV from being read and the database from being populated (lines 139-244).
The retrieveMennoData function already handles file existence checks appropriately (line 75), so this additional check is redundant and breaks the intended workflow. If the goal is to avoid re-downloading, the existing logic in retrieveMennoData already provides this optimization.
| if _, err := os.Stat(csvPath); err == nil { | |
| log.Printf("CSV file already exists, skipping download: %s", csvPath) | |
| return // Skip download if the file exists | |
| } | |
| // (File existence is handled in retrieveMennoData; do not return early here.) |
|
@copilot open a new pull request to apply changes based on the comments in this thread |
No description provided.