Skip to content

Add build-resources branch for reviewable generated markdown files #462

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
richarddushime merged 6 commits into from
Oct 29, 2025
Merged

Add build-resources branch for reviewable generated markdown files #462

richarddushime merged 6 commits into from
Oct 29, 2025

Conversation

Copy link
Contributor

Copilot AI commented Oct 27, 2025 •
edited
Loading

Generated markdown files (resources, tenzing, glossary) were only available as workflow artifacts, making them invisible for review and untrackable over time.

Changes

.github/workflows/data-processing.yml

  • Added workflow_dispatch input parameter regenerate_glossary (boolean) for controlled glossary updates
  • Added conditional glossary generation step (only runs when manually triggered with regenerate_glossary: true)
  • New step commits generated files to build-resources branch after artifact upload
  • Implements retry logic with --force-with-lease (3 attempts with rebase)
  • Uses dynamic branch references instead of hardcoded names

.github/BUILD_RESOURCES_BRANCH.md

  • Documents automated branch management
  • Clarifies dual approach: artifacts for builds (primary), branch for review (secondary)
  • Documents conditional glossary regeneration process and manual trigger instructions

Architecture

# Daily runs (automatic)
- Generate files (tenzing, resources only)
- Upload artifacts → Used by deploy workflows (unchanged)
- Commit to build-resources → Reviewable on GitHub (new)

# Manual runs with regenerate_glossary: true
- Generate files (tenzing, resources, glossary)
- Upload artifacts → Used by deploy workflows (unchanged)
- Commit to build-resources → Reviewable on GitHub (new)

Files committed to build-resources:

  • content/curated_resources/*.md - Resources from Google Sheets (not in master) - Updated daily
  • content/contributors/tenzing.md - Contributor list (already in master, now trackable) - Updated daily
  • content/glossary/*/ - Glossary terms (already in master, now trackable) - Updated only on manual trigger

Glossary Regeneration:
Glossary files are NOT regenerated on daily runs due to source stability concerns. To regenerate glossary:

  1. Actions → "Data Processing" workflow
  2. "Run workflow" → Check "Regenerate glossary files" → Run

Build process unchanged—continues using artifacts. Branch provides visibility and version history without affecting deployments.

Original prompt

This section details on the original issue you should resolve

<issue_title>Change data processing issue to commit markdown version</issue_title>
<issue_description>Currently, the data processing returns in artefacts that are passed into hugo, so that we never see the current md files for resources, tenzing or the glossary. It would be easier if they were reviewable - so could we maybe instead commit the outputs to a branch, and then use them from there (though we would then still need to combine the generated folders from that branch with the PR sources for previews). @richarddushime @flavioazevedo let's discuss the best way here.</issue_description>

<agent_instructions>Make a proposal for this - maybe the resources should just be committed to a build-resources branch but still be used as is (from artefacts) during builds</agent_instructions>

Comments on the Issue (you are @copilot in this section)

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits October 27, 2025 13:08
@LukasWallrich
Add build-resources branch workflow and documentation
6e957c1 
Co-authored-by: LukasWallrich <60155545+LukasWallrich@users.noreply.github.com>
@LukasWallrich
Improve build-resources workflow with retry logic and dynamic branch …
39de595 
…handling

Co-authored-by: LukasWallrich <60155545+LukasWallrich@users.noreply.github.com>
Copilot AI changed the title [WIP] Change data processing to commit markdown version Add build-resources branch for reviewable generated markdown files Oct 27, 2025
Copilot finished work on behalf of LukasWallrich October 27, 2025 13:19
Copilot AI requested a review from LukasWallrich October 27, 2025 13:19
@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 27, 2025

@copilot the glossary should not be rebuilt daily as the sources are less stable - but can this be triggered on workflow dispatch with a specific parameter?

@github-actions github-actions bot added the cicd Relevant to GitHub workflows label Oct 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 27, 2025

👍 All image files/references (if any) are in webp format, in line with our policy.

@LukasWallrich
Make glossary regeneration conditional on manual trigger
719155d 
Co-authored-by: LukasWallrich <60155545+LukasWallrich@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Oct 27, 2025

@copilot the glossary should not be rebuilt daily as the sources are less stable - but can this be triggered on workflow dispatch with a specific parameter?

Done! Glossary regeneration is now conditional and won't run on daily scheduled runs.

Changes in commit 719155d:

  • Added workflow_dispatch input parameter: regenerate_glossary (boolean, default: false)
  • Glossary generation step now only runs when regenerate_glossary: true
  • Build-resources commit conditionally includes glossary files based on the parameter
  • Updated documentation with instructions

To regenerate glossary:

  1. Actions → "Data Processing" workflow
  2. "Run workflow" → Check "Regenerate glossary files" → Run

Daily runs will update resources and contributors but skip glossary generation.

Copilot finished work on behalf of LukasWallrich October 27, 2025 13:40
@LukasWallrich LukasWallrich marked this pull request as ready for review October 27, 2025 13:57
@LukasWallrich LukasWallrich requested a review from a team as a code owner October 27, 2025 13:57
@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 27, 2025

@richarddushime please have a look, I think ready to merge

@LukasWallrich LukasWallrich mentioned this pull request Oct 27, 2025
Open
@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 27, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

8 similar comments
@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 27, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 28, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 28, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 28, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 28, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 29, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 29, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 29, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 29, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

1 similar comment
@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 29, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

@LukasWallrich
Copy link
Contributor

LukasWallrich commented Oct 29, 2025

✅ This PR has been deployed to staging as part of an aggregated deployment. View at: https://staging.forrt.org

richarddushime
richarddushime approved these changes Oct 29, 2025
View reviewed changes
Copy link
Contributor

@richarddushime richarddushime left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks Good .
Next : TESTING and we see the results

@richarddushime richarddushime merged commit 7b87ba0 into master Oct 29, 2025
14 checks passed
@richarddushime richarddushime deleted the copilot/change-data-processing-commit-md branch October 29, 2025 21:55
richarddushime added a commit that referenced this pull request Nov 13, 2025
@LukasWallrich @richarddushime
Add build-resources branch for reviewable generated markdown files (#462
4a4d130 
)

* Initial plan

* Add build-resources branch workflow and documentation

Co-authored-by: LukasWallrich <60155545+LukasWallrich@users.noreply.github.com>

* Improve build-resources workflow with retry logic and dynamic branch handling

Co-authored-by: LukasWallrich <60155545+LukasWallrich@users.noreply.github.com>

* Make glossary regeneration conditional on manual trigger

Co-authored-by: LukasWallrich <60155545+LukasWallrich@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: LukasWallrich <60155545+LukasWallrich@users.noreply.github.com>
Co-authored-by: Lukas Wallrich <lukas.wallrich@gmail.com>
Co-authored-by: Richard Dushime <45734838+richarddushime@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@richarddushime richarddushime richarddushime approved these changes

@LukasWallrich LukasWallrich Awaiting requested review from LukasWallrich

Assignees

@LukasWallrich LukasWallrich

Copilot code review Copilot

Labels

cicd Relevant to GitHub workflows

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Change data processing issue to commit markdown version

3 participants

@LukasWallrich @richarddushime