Skip to content

Publish reduced orso files as part of nightly builds #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Mar 5, 2025
Merged

Conversation

nvaytet
Copy link
Member

@nvaytet nvaytet commented Mar 3, 2025

Fixes #123

@nvaytet nvaytet requested a review from jl-wynen March 4, 2025 13:48
@nvaytet nvaytet marked this pull request as ready for review March 4, 2025 13:48
Copy link
Member

@jl-wynen jl-wynen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a solution for accessing the artefact now?

@nvaytet
Copy link
Member Author

nvaytet commented Mar 4, 2025

I think the solution is to create a github account and use github cli, unless you can think of another place where we can publish the artifacts (somewhere else on github?)

@jl-wynen
Copy link
Member

jl-wynen commented Mar 4, 2025

Let me rephrase, do you / we have a tested solution that we know works?

@MridulS
Copy link
Member

MridulS commented Mar 5, 2025

Just to unblock @mEngstroem while he is writing the integration test I have started a cron job in a server that fetchs the newest run and uploads the reduced data to https://pub-6c25ef91903d4301a3338bd53b370098.r2.dev/amor_reduced_iofq.ort
We can change the URL downstream while we test things out.

@nvaytet
Copy link
Member Author

nvaytet commented Mar 5, 2025

@MridulS in this PR, we will make both latest and stable artifacts.
Is your service able to fetch both and post them both?

Also, there may be more files in the future, not just a single file.
Would probably be better to publish a zip archive?

@MridulS
Copy link
Member

MridulS commented Mar 5, 2025

The current service fetches only from nightly_at_main workflow, but yes it can fetch and upload from multiple runs. The bash file running is available here https://gist.github.com/MridulS/79d7de68b27276da6fafa4f6c7bc9645

@nvaytet nvaytet merged commit 412fb1d into main Mar 5, 2025
7 checks passed
@nvaytet nvaytet deleted the cron-orso-files branch March 5, 2025 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create a cron nightly job that publishes reduced orso files
3 participants