Skip to content

Proposal: Support git providers other than GitHub in prow #10146

Closed as not planned

Description

@wbrefvem

What would you like to be added:

Some sort of generic git service in prow that normalizes incoming webhooks into a generic format that hook and other plugins can understand, as well as provides a generic interface for performing git-related actions such as commenting on PRs, etc.

My initial proposal is that this git service essentially be another plugin that implements a number of generic git-related APIs, and that the name/location of this service be configurable so that we can support out-of-tree providers. In addition, it's probably worthwhile to think about decomposing the notions of code review and issue tracking into separate provider APIs, since not all providers implement these functions, although presumably prow will only ever be useful in combination with a git provider that at least provides basic code review features. But it would be nice to support, e.g., code review tool X in combination with issue tracker Y. We currently have something like this with the loose coupling of the Gerrit adapter and reporter. It would be great if we could make this sort of thing generic and extensible.

We discussed this briefly on the sig-testing call of November 6, and my hope is that this issue will be an asynchronous continuation of that discussion. I am also working on a more formal proposal that I plan to present on the weekly sig-testing call some time in the near future.

Why is this needed:

Currently, prow is tightly coupled to GitHub. Making it more generic would give the various k8s projects more flexibility in how they craft their workflows, as well as making prow more accessible to projects in the broader open source community and beyond.

/kind feature

Activity

  1. BenTheElder commented on Nov 14, 2018

    @BenTheElder
    Member

    cc @cjwagner @krzyzacy
    /area prow

  2. yastij commented on Nov 15, 2018

    @yastij
    Member

    we might want to think about an interface that providers might want to implement.

  3. wbrefvem commented on Nov 20, 2018

    @wbrefvem
    ContributorAuthor

    @yastij Yeah, there seems to be some consensus around this.

  4. wbrefvem commented on Nov 20, 2018

    @wbrefvem
    ContributorAuthor

    Update

    Here are some more concrete ideas on how to tackle this:

    • Implement a git service as a freestanding gRPC server
    • At least one webhook normalization method that would transform an incoming webhook into a generic event that hook can dispatch
    • Individual methods for various actions such as AssignIssue, CommentOnPR, etc.
    • Ship a default service that supports popular providers such as, but not necessarily limited to, GitHub, GitHub Enterprise, GitLab, Bitbucket.
    • Make protobuf package and .proto files available for users who want to implement their own service.
  5. midnightconman commented on Feb 9, 2019

    @midnightconman
    Contributor

    This PR #11075 was just merged, it gets us closer to this goal.

  6. fejta-bot commented on May 10, 2019

    @fejta-bot

    Issues go stale after 90d of inactivity.
    Mark the issue as fresh with /remove-lifecycle stale.
    Stale issues rot after an additional 30d of inactivity and eventually close.

    If this issue is safe to close now please do so with /close.

    Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
    /lifecycle stale

  7. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on May 10, 2019
  8. yastij commented on May 10, 2019

    @yastij
    Member

    /remove-lifecycle stale
    /lifecycle frozen

  9. added
    lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to staleness.
    and removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on May 10, 2019
  10. townsendb commented on May 28, 2019

    @townsendb

    Hi @BenTheElder . Is this proposal on the radar? I'm using GitLab and trying to use the next gen serverless features of Jenkins-x but blocked by this issue in prow.
    Many thanks.

  11. McShauno commented on Jun 12, 2019

    @McShauno

    Ok, fine ... I'll write it. Give me a couple of weeks.

    ¯_(ツ)_/¯

  12. BenTheElder commented on Jun 12, 2019

    @BenTheElder
    Member

    @townsendb Hi, Sorry I missed this comment in all my github notifications somehow... I don't really work on prow currently (kind mainly) though I'd love to see this happen. I'm not sure where this is at.

    @cjwagner & @stevekuznetsov are good current points of contact for prow.

  13. 11 remaining items

  14. abdennour commented on Jan 10, 2021

    @abdennour

    we love prow. But, we are using bitbucket cloud.
    Any change ?

  15. cjwagner commented on Jan 11, 2021

    @cjwagner
    Member

    @abdennour No change, Alvaro's comment is still accurate. PRs are welcome if you'd like to contribute to this!

  16. spiffxp commented on Aug 9, 2021

    @spiffxp
    Contributor

    /sig testing

  17. added
    sig/testingCategorizes an issue or PR as relevant to SIG Testing.
    on Aug 9, 2021
  18. vsoch commented on Apr 24, 2023

    @vsoch

    I'm definitely interested in this use case (and new to prow but looking to try it out!) At the labs we use GitLab heavily, and it would be great to have this integration. The degree to which I could help will probably come down to if it's easy to set up a development environment I think.

  19. vsoch commented on Apr 24, 2023

    @vsoch

    Also the README says it was going to be deleted in February but... it still lives!! cue Frankenstein monster noises
    image

  20. BenTheElder commented on Apr 24, 2023

    @BenTheElder
    Member

    There really need to be more concrete proposals, this thread is pretty vague.

    prow has many aspects and most of them are inherently GitHub specific (eg everything related to webhook handling and label based chatops) which theoretically could be abstracted away, but at this point would take a lot of work and require a more detailed proposal

  21. vsoch commented on Apr 24, 2023

    @vsoch

    I'm not familiar with how proposals are done here - akin to a KEP?

    I was reading here: https://docs.prow.k8s.io/docs/life-of-a-prow-job/

    And at least from a high level, the abstractions like hooks, comments, are things that can be mapped to other code review places too. I definitely don't disagree about it being a lot of work, but (at face value) doesn't seem impossible?

    @BenTheElder are these the best docs https://docs.prow.k8s.io/docs/build-test-update/ for a developer to get started trying things out?

  22. BenTheElder commented on Apr 24, 2023

    @BenTheElder
    Member

    I'm not familiar with how proposals are done here - akin to a KEP?

    Depends on the scope of the change, typically a lightweight google doc is enough to start discussion, which should be shared to the SIG in the mailinglist and slack.

    And at least from a high level, the abstractions like hooks, comments, are things that can be mapped to other code review places too. I definitely don't disagree about it being a lot of work, but (at face value) doesn't seem impossible?

    It's not impossible, but it's a lot of work and as-is we're having difficulty getting much more scoped changes the Kubernetes project itself would benefit from reviewed / approved promptly, there's limited bandwidth from the current prow OWNERS.

    @BenTheElder are these the best docs https://docs.prow.k8s.io/docs/build-test-update/ for a developer to get started trying things out?

    I don't work on prow itself often anymore and when I do I make small fixes that don't require spinning up an instance.

  23. vsoch commented on Apr 24, 2023

    @vsoch

    Interesting - I would guess there is a long history behind this, but why does prow have to live under Kubernetes here proper? If it’s holding the project back and adding extra churn for developers that doesn’t make a lot of sense.

  24. BenTheElder commented on Apr 24, 2023

    @BenTheElder
    Member

    Interesting - I would guess there is a long history behind this, but why does prow have to live under Kubernetes here proper?

    Prow exists because Kubernetes needed some CI tooling, which started as a webhook system for commands against jenkins and then a merge robot. It eventually turned to running CI workloads on Kubernetes itself.

    Usage for other purposes beyond Kubernetes came much later. It still primarily exists for Kubernetes and supports workflows used by the project.

    If it’s holding the project back and adding extra churn for developers that doesn’t make a lot of sense.

    That's not what I meant. I meant that the existing owners are currently already struggling to keep up with incoming feature work so I'm warning that it may not be a great time to propose a very large change.

  25. vsoch commented on Apr 24, 2023

    @vsoch

    Gotcha, and thank you for the details! It could be then that a better avenue to pursue would be creating something like prow, but separate from it and with a focus on more general abstractions for components.

    If you (or anyone in this thread) know if other tools in this space that are for creating your own CI/CD on Kubernetes please share! When you do a general Google search it typically shows articles about CI/CD with respect to continuous deployment to Kubernetes, and not making your own system (that could work with it).

  26. kfox1111 commented on Apr 24, 2023

    @kfox1111

    There was a fork of Prow I think a while back that eventually became Lighthouse? https://github.com/jenkins-x/lighthouse/

    I wanna say it was intentionally done to try and add more then github support while not burdening the k8s developers with features they didn't need.

    Might be what your looking for?

  27. BenTheElder commented on May 30, 2024

    @BenTheElder
    Member

    I don't think we've had the bandwidth for this, but in any case prow is now split out to github.com/kubernetes-sigs/prow and future feature discussions should be held there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/prowIssues or PRs related to prowkind/featureCategorizes issue or PR as related to a new feature.lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to staleness.sig/testingCategorizes an issue or PR as relevant to SIG Testing.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions