Output checkpoint

[bohendo]

Uploads the output directory every 60 seconds while the job is running, so if a run crashes we still have something to show for it.

Also helpful to get WIP output.

resolves #26

Summary by CodeRabbit

• New Feature: Implemented upload_output function in user_data.sh.tmpl for periodic uploads of files to an S3 bucket during job execution, enhancing data persistence and reliability.
• Improvement: Updated cleanup function to include file upload to S3, ensuring data is stored upon job completion.
• Change: Modified GetLatestCompletedJob in state.go to return an error when no completed jobs are found, improving error handling and API robustness.

[github-actions]

CodeRabbit

Walkthrough

A new feature for periodic uploads to an S3 bucket has been integrated into the user_data.sh.tmpl script, enhancing data synchronization capabilities. Additionally, error handling in state.go has been improved to handle cases where no completed jobs are found.

Changes

File(s)	Summary
cmd/cloudexec/user_data.sh.tmpl	Introduced upload_output function for S3 uploads, invoked on cleanup and at regular intervals controlled by sync_heartbeat.
pkg/state/state.go	Modified GetLatestCompletedJob to return an error when no completed jobs are detected.

🐇 A hop of code, a leap of function,
To the cloud it goes, with swift instruction.
No job left behind, not one to shun,
For every cycle done, a new upload begun.
With each tick and tock, the heartbeat syncs,
🚀 Up to the S3, our payload links.
And should you query for a job complete,
An error returns if none meet the feat.
So here we commit, with joyous elation,
A rabbit's touch on cloud automation. 🌥️✨

---

Uplevel your code reviews with CodeRabbit Pro

CodeRabbit Pro

If you like this project, please support us by purchasing the Pro version. The Pro version has advanced context, superior noise reduction and several proprietary improvements compared to the open source version. Moreover, CodeRabbit Pro is free for open source projects.

[github-actions]

CodeRabbit

Commits

Files that changed from the base of the PR and between 530f2fa and 1d5099b commits.

Files selected (2)

• cmd/cloudexec/user_data.sh.tmpl (4)
• pkg/state/state.go (1)

Review comments generated (5)

• Review: 4
• LGTM: 1

---

Tips

Chat with CodeRabbit Bot (@coderabbitai)

• Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
• Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

• The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
• You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

• Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.

[github-actions]

CodeRabbit

Commits

Files that changed from the base of the PR and between 1d5099b and a29d1ae commits.

Files selected (2)

• cmd/cloudexec/user_data.sh.tmpl (5)
• pkg/state/state.go (2)

Review comments generated (7)

• Review: 4
• LGTM: 3

---

Tips

Chat with CodeRabbit Bot (@coderabbitai)

• Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
• Invite the bot into a review comment chain by tagging @coderabbitai in a reply.

Code suggestions

• The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
• You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.

Pausing incremental reviews

• Add @coderabbitai: ignore anywhere in the PR description to pause further reviews from the bot.