connectors-ci: get modified files from PR API #26889
Conversation
octavia-squidington-iii
added
area/connectors
alafanechere
force-pushed
the
augustin/connectors-ci/modified-files-from-pr
branch
from
762b28a
to
04b17f5
Compare
This reverts commit 1d6f5bd.
| @@ -24,6 +24,7 @@ | |||
|
|
|||
| if TYPE_CHECKING: | |||
| from ci_connector_ops.pipelines.contexts import ConnectorContext | |||
| from github import PullRequest | |||
There was a problem hiding this comment.
I was expecting some curl/API work in this PR. I guess this just handles it!
There was a problem hiding this comment.
This lib is pretty good at wrapping the GitHub API for typed interaction.
| git_branch: str, git_revision: str, diffed_branch: str, is_local: bool, ci_context: CIContext, pull_request: PullRequest | ||
| ) -> List[str]: | ||
| """Get the list of modified files in the current git branch. | ||
| If the current branch is master, it will return the list of modified files in the current commit. |
There was a problem hiding this comment.
Suggestion: Maybe clarify that this "current commit" is the squash commit of the branch whose merge kicked off the run
The reason I mention this is that if we decided to switch from squash commits to a full history (e.g. rebase instead of merge), I'm not sure if this "last commit" logic would work anymore. Its an edge case not worth writing more code for, but a reminder of the context that allows us to do it this way may be helpful.
There was a problem hiding this comment.
I improved the docstring, thank you for the suggestion.
True, but we can easily fallback on the |
What
In our connectors CI pipeline we originally get the list of modified files with
git diffoperation.These operations are quite expensive, especially in the CI because we run the
git diffoperation in a dagger container that is checkouting airbyte's repo.In a pull request context the GitHub API provides the list of modified files in the pull request.
We can use this list instead of running
git diffto speed up pipelines running on PRs.The current pull request is an attempt at doing so.
I also did an additional operation:
We previously alway ran two different git diff commands: one to get the list of modified files in branch, the other one to get the list of modified files in the last commit.
I slightly changed this logic: we get the list of modified files in the last commits only when we're on master. When we're on a branch we git diff the full branch.
Before on source-pokeapi: total run duration 3min
After change on source-pokeapi: total run duration 2mn
How
get_modified_filesUser impact
Faster test run on PR: expect 1 minute speed gain.