Repository navigation
Proposal: Move Generated Dataset Files Out of Source-Controlled Branches #306
Replies: 6 comments 3 replies
|
Hi, Thanks for the proposal and the tests. Give me some days to take a look at it and I will come back to you. Indeed, I'm a little busy currently. |
|
Hi, I read your proposal and I'm aligned with the global ideas and remarks. We can move the generated content to a released artefact named as you proposed YYYY-MM-DD but with the prefix statistics-dataset-. About the following point: For me, we can indeed move the creation of the file input.csv to the script gather_data.py to minimize the code in the workflow. So drop the creation from the workflow YAML file. The important change will be in the script that generate the statistics page that will need to use the latest available data release. Just a remark about this: These files are produced automatically by our scheduled workflows and are not manually maintained source files. Because their contents change each month, they frequently cause merge conflicts across branches despite not representing any actual code changes. The update is performed once a month so conflit are easy to resolve when occur (rare). It is important keep in mind that this project (OSHP) is a documentation project (content must be stable) so there is no change every day so conflit are rare and easy to solve. It is not a software with commit every days. In addition, the README of the statistics project will need to be updated. |
|
Will raise a PR shortly on the same. |
|
🙏 A huge thanks for the proposal and the PR. ✅ I reviewed and merged it. 🔬 I launched the workflow to create the statistic data source to ensure that all is ok after the merge. When the job will be finished I will launch the workflow to update the statistics page to ensure that the complete flow is ok. 🤝 A big big thanks again for your help on the project |


Uh oh!
There was an error while loading. Please reload this page.
Background
Our repository currently stores two automatically generated files:
subprojects/statistics/data/data.dbsubprojects/statistics/data/input.csvThese files are produced automatically by our scheduled workflows and are not manually maintained source files. Because their contents change each month, they frequently cause merge conflicts across branches despite not representing any actual code changes.
I explored
.gitattributesmerge strategies as a mitigation, but these require local Git configuration on every developer's machine and don't eliminate all conflict scenarios. Storing binary blobs (.db) in Git history also leads to unnecessary repository bloat over time.Proposed Change
Move these generated dataset files out of normal development branches entirely and store them as versioned GitHub Release assets instead.
The proposed flow:
data.dbandinput.csvas it does today.gather_data.pycontains a Python fallback that automatically downloads and generates input.csv if the file is not found locally, making local developer runs fully self-contained without any manual setup.Benefits
I have tested this end-to-end on my fork and everything looks good:
✅ Release assets: https://github.com/chinmaypant21/www-project-secure-headers/releases/tag/dataset-2026-06-24
✅ Draft PR on my fork (for reference): chinmaypant21#2
Looking forward to any feedback on feasibility, tradeoffs, or suggestions on the same.
All reactions