Fix cron job to update PR embedding cache #3493
Conversation
- Add safeguards to prevent embedding all PRs when cache download fails - Add --force-rebuild flag for explicit full rebuilds - Add --allow-full-rebuild flag for first-time setup - Update cronjob to use cache_refresh.py wrapper
- Merge cache_refresh.py functionality into build_cache.py - Add --download-from-s3 flag for cronjob use - Add --s3-bucket and --s3-prefix flags for S3 configuration - Update cronjob to use unified script with --download-from-s3 - Remove redundant cache_refresh.py wrapper
dominik3141
changed the title
dominik3141
changed the title
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| raise RuntimeError(f"Failed to upload cache to S3: {e}") from e | ||
|
|
||
|
|
||
| def _get_api_key() -> str: |
There was a problem hiding this comment.
Can we use the impl in softmax/?
There was a problem hiding this comment.
To get the API key? It's just three lines of code, I don't see the point in wrapping the simple call to boto3 (assuming I understand you correctly).
| git remote set-url origin https://github.com/Metta-AI/metta.git | ||
| git fetch --unshallow origin main 2>/dev/null || git fetch origin main | ||
| git checkout -f origin/main | ||
| uv pip install numpy google-genai boto3 |
There was a problem hiding this comment.
This works but we can also add these as a dependency group inside project.toml and install through uv
nishu-builder
left a comment
nishu-builder
left a comment
There was a problem hiding this comment.
Any reason not to put all this in the softmax directory and then use the softmax directory as your working when you do UV sync? That should avoid a lot of UV re-downloading packages hell.
I have solved that problem by the uv --no-sync, but I agree that there should be a better way to do that |
Resolved conflicts by: - Keeping our separate helm chart structure (dashboard-cronjob and pr-similarity-cache-cronjob) - Incorporating PR #3493 changes to pr-similarity-cache-cronjob command (bash script with git checkout, uv run --no-sync, and --download-from-s3 flag) - Removing old softmax-cronjob anchor system that was deleted in our branch - Updated helmfile.yaml to use new chart-based approach instead of YAML anchors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Two sets of changes
uv runwith--no-syncto make sure that we don't install hundreds of large dependencies that we never need.Asana Task