Skip to content

Conversation

jescalada
Copy link
Contributor

@jescalada jescalada commented Sep 7, 2025

Fixes #985. The profiling screenshots are in the issue discussion.

This PR reduces push times by around 50% by using the depth: 1 and singleBranch: true options in pullRemote when cloning. It also adds the stream: true option to the express-http-proxy configuration, which should help with larger pushes.

I've experimented with other optimizations to the chain actions but it seems that the majority of time is spent cloning the target repository (and therefore limited by download speeds).

I'd say storing the target repository so we only clone the entire repo once is the best option, the tradeoff being that we might need to store several GB of data for larger repos. Perhaps we could have a config option to toggle this so admins can do as they see fit?

Copy link

netlify bot commented Sep 7, 2025

Deploy Preview for endearing-brigadeiros-63f9d0 canceled.

Name Link
🔨 Latest commit 3a60eba
🔍 Latest deploy log https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/68bd6ac32801ad0008e40877

Copy link

netlify bot commented Sep 7, 2025

Deploy Preview for endearing-brigadeiros-63f9d0 canceled.

Name Link
🔨 Latest commit be57ca4
🔍 Latest deploy log https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/68cc08b09b832a0008c0d928

@jescalada jescalada changed the title feat: pullRemote optimizations feat: push speed optimizations Sep 7, 2025
Copy link

codecov bot commented Sep 13, 2025

Codecov Report

❌ Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 83.72%. Comparing base (c29b56c) to head (be57ca4).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
src/proxy/processors/push-action/pullRemote.ts 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1189      +/-   ##
==========================================
- Coverage   83.74%   83.72%   -0.03%     
==========================================
  Files          67       67              
  Lines        2892     2888       -4     
  Branches      366      366              
==========================================
- Hits         2422     2418       -4     
  Misses        410      410              
  Partials       60       60              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jescalada jescalada marked this pull request as ready for review September 13, 2025 06:36
@jescalada jescalada requested review from kriswest, coopernetes and a team September 13, 2025 06:36
Copy link
Contributor

@coopernetes coopernetes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, one minor point on the singleBranch arg.

@jescalada jescalada merged commit 50a1761 into finos:main Sep 19, 2025
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

optimize pullRemote for large repos
3 participants