Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: remote builder #1631

Merged
merged 18 commits into from
Apr 2, 2024
Merged

refactor: remote builder #1631

merged 18 commits into from
Apr 2, 2024

Conversation

18alantom
Copy link
Member

@18alantom 18alantom commented Mar 26, 2024

This PR is to be a refactor of #1475.

While the previous implementation works, it has a few issues that make it unsuitable for production and long term:

  • Improper error handling, if docker build fails, press mostly just receives a CalledProcessError with irrelevant traceback. One needs to dive into agent logs to figure out what went wrong.
  • Divergence from press build code, feat: remote docker image build support agent#83 is almost entirely a duplicate of deploy_candidate.py this is not a feasible approach, the code between the two has already diverged.

Agent side: frappe/agent#94

Note: this PR is incomplete, remote builder still isn't entirely ready, but since this PR has grown to above 1.2k line changes, I'm merging it as checkpoint.

TODO (will be addressed in a future PR):

  • Proper error handling on agent side build failures (need to test it)
  • Checking if it all works properly
  • Method to clear bench app cache on remote builds
  • Higher freq agent polling for build jobs
  • Report to run command in docker cache

Things completed:

  • Refactor handling of build and push output parsing
  • Move output parsing from agent to press (prevent duplication and divergence)
  • Add remote build related steps
  • Improve debugging DX by linking Error Logs on Agent Jobs and Deploy Candidates
  • Break remote build into 3 steps (previously it was only 1)
  • Ensure that press build works properly

@18alantom 18alantom changed the title chore: gen AJ fixtures for remote builder refactor: remote builder Mar 26, 2024
Copy link

codecov bot commented Mar 26, 2024

Codecov Report

Attention: Patch coverage is 45.85492% with 209 lines in your changes are missing coverage. Please review.

Project coverage is 38.69%. Comparing base (3308910) to head (524c8c5).
Report is 1 commits behind head on master.

Files Patch % Lines
...press/doctype/deploy_candidate/deploy_candidate.py 26.06% 122 Missing ⚠️
.../doctype/deploy_candidate/docker_output_parsers.py 62.96% 70 Missing ⚠️
press/agent.py 25.00% 9 Missing ⚠️
press/press/doctype/agent_job/agent_job.py 70.58% 5 Missing ⚠️
...s/doctype/deploy_candidate/deploy_notifications.py 0.00% 2 Missing ⚠️
press/press/doctype/app_patch/app_patch.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1631      +/-   ##
==========================================
+ Coverage   38.66%   38.69%   +0.03%     
==========================================
  Files         316      317       +1     
  Lines       25172    25357     +185     
==========================================
+ Hits         9733     9813      +80     
- Misses      15439    15544     +105     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pass dc_name when uploading
@18alantom 18alantom marked this pull request as ready for review April 2, 2024 05:21
@phot0n
Copy link

phot0n commented Apr 2, 2024

Improve debugging DX by linking Error Logs on Agent Jobs and Deploy Candidates

thanks for this ❤️ - been so annoyed by having to constantly keep an eye on error log.

@18alantom
Copy link
Member Author

I am 75% sure the failing tests are unrelated to my PR (they're failing elsewhere too). And so I'm merging it, this really shouldn't be FC workflow, but it is. Will address it soon.

@18alantom 18alantom merged commit 8c13f1a into master Apr 2, 2024
4 of 7 checks passed
@18alantom 18alantom deleted the refactor-remote-builder branch April 2, 2024 10:09
18alantom added a commit that referenced this pull request Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants