Add paddle operator proposal to kubeflow community.

[tizhou86]

Add paddle operator proposal to kubeflow community.

[terrytangyuan]

This makes sense. What about the implementation of the operator? Could you leverage https://github.com/kubeflow/common's interface that's used for other operators?

[tizhou86]

We are using kube-builder as the skeleton for the paddle-operator, part of the kubeflow common components are not necessary for our project, like job controller etc. We'll see whether we can leverage on kubeflow common for advanced features.

[kuizhiqing]

Thanks for your reply, we are aware of your concern, technically, we are building our operator with the newest version of kubebuilder which take care of almost all the staff like Informer/Indexer/clienset etc. It leaves out the Reconcile function to be implemented, it also provides CRUD operations directly on resources with context, which means that the kubeflow/common may not necessary in this circumstances.

[terrytangyuan]

cc @kubeflow/wg-training-leads @kubeflow/wg-automl-leads

[terrytangyuan]

This LGTM from me. Please take another look @kubeflow/wg-training-leads @kubeflow/wg-automl-leads as well.

@tizhou86 @kuizhiqing Are you planning to transfer the existing repo https://github.com/PaddleFlow/paddle-operator to Kubeflow org once this is approved? @Bobgy @theadactyl Any questions or concerns on this?

[terrytangyuan]

/hold

[PatrickXYS]

Awesome!

Only nits comment

[PatrickXYS]

All link expired

[tizhou86]

Since we've just merged the new refactoring, I'll fix it soon.

[Jeffwan]

We briefly talked about this story in AutoML and training bi-weekly meeting. People are interested in support plan as well. For example, if there's active maintainers for this project. If it want to be part of kubeflow release, etc.

[tizhou86]

This LGTM from me. Please take another look @kubeflow/wg-training-leads @kubeflow/wg-automl-leads as well.

@tizhou86 @kuizhiqing Are you planning to transfer the existing repo https://github.com/PaddleFlow/paddle-operator to Kubeflow org once this is approved? @Bobgy @theadactyl Any questions or concerns on this?

Yes, Paddle-operator is one of our ecosystem projects that we want to transfer to kubeflow org for maintenance, It's approved by the Paddle team already, we think the open governance will be good for the Paddle ecosystem projects.

[tizhou86]

We briefly talked about this story in AutoML and training bi-weekly meeting. People are interested in support plan as well. For example, if there's active maintainers for this project. If it want to be part of kubeflow release, etc.

Great! If possible, we can do a presentations on kubeflow weekly meeting, and then we can discuss about the future plan in the meeting.

[gaocegege]

/assign @Jeffwan

/cc @zw0610

[google-oss-robot]

@gaocegege: GitHub didn't allow me to request PR reviews from the following users: zw0610.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/assign @Jeffwan

/cc @zw0610

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Jeffwan]

@gaocegege Last WG-training meeting was CN time friendly and I miss it. Any conclusion?

[gaocegege]

We think we can accept it. If we decide to merge into one controller, we can refactor the paddle operator later.

WDYT

[Jeffwan]

We think we can accept it. If we decide to merge into one controller, we can refactor the paddle operator later.

WDYT

Agree. Let's kick off the transfer process.

/lgtm
/approve

/cc @Bobgy @theadactyl

@gaocegege @terrytangyuan and I help review the proposal from WG-Training side and it looks good to us. Can you help take a review?

[gaocegege]

/cc @Bobgy

Could you please help us create the repository for the paddle-operator?

[tizhou86]

Hi Bob, please let us know if anything should be done before the transfer process, thanks!

[tizhou86]

Hi Bob, please let us know if anything should be done before the transfer process, thanks!

The original repository is: https://github.com/PaddleFlow/paddle-operator , the new repository I suppose will be : https://github.com/kubeflow/paddle-operator , we've recently added more user guides and documentations for developers.

[tizhou86]

/cc @Bobgy for fear that this thread may have escaped from attention. :-)

[Bobgy]

Sorry for the delay, @kubeflow/wg-training-leads.

I discussed with @kubeflow/project-steering-group and we would suggest merging all training operators into one monorepo. How do feel about that?

The rationale:
we need to do a complex release process each time a training operator repo is created. It seems that the number of frameworks will only keep increase in the future. However, if I understand correctly, most operators share quite some stuff in common. So I think it might be a lower effort thing for both of us to create one central training operators repo once, and keep adding framework-specific operators to it.

[Bobgy]

I think the proposal also relates to #512

[terrytangyuan]

@Bobgy Please see our discussion on merging the operators in kubeflow/common#103. Although I think this should be discussed separately and should not block the acceptance of paddle-operator. Release is also worth a separate discussion as it's the release process in general that's complex but not specific to training operators.

[Bobgy]

@terrytangyuan regardless of whether operators are merged, what about using a monorepo?
I'm not talking about the engineering process for release, but actually Google's internal release process, any new repo in Kubeflow org needs to go through Google release process and we want to avoid this by using a monorepo (anyway, only in case when monorepo makes sense for Training WG).

Looks like we can wait for kubeflow/common#103 (comment) to see more updates?

[Jeffwan]

any new repo in Kubeflow org needs to go through Google release process

em. Since this is internal process and we don't have any clues, could you help us understand the efforts? It helps WG leads to make deliberate decision at the time we have these kind of repo requests.

[tizhou86]

Yeah, we are really looking forward to have paddle operator to integrated into Kubeflow community, some of our end users are building the PaddlePaddle deep learning system leveraging on Kubeflow and paddle operator, we want to make it more straightforward to our users. And with the impact of the PaddlePaddle community(15.4k for the paddle main repo and 50k+for the paddle org projects), we think it will be a win-win solution for both community.

[Bobgy]

Sorry for the long delay, let me give an update:

1. I have learned what the process for adopting a new repo roughly looks like, but I'll have to do it once to understand how much efforts we need each time. Due to policy issues, I cannot share more details.
2. I had some discussions within PSG, but haven't reached a final conclusion yet. I'll update again next week.

[Bobgy]

Hi all, @kubeflow/project-steering-group agreed to accept this proposal and start a new repo for paddle-operator without waiting for potential merge of Training WG operators.

@kubeflow/wg-training-leads may I confirm that you are still willing to accept the proposal too, right? because it's been a while.

If everyone agrees, I can start working with @tizhou86 on the process to adopt the paddle-operator repo.

[terrytangyuan]

+1 from me. @kubeflow/wg-training-leads Any concerns?

[johnugeorge]

+1

[gaocegege]

SGTM

[tizhou86]

Great, thanks @Bobgy ! Please let me know if anything is needed from PaddlePaddle team.

[Bobgy]

/lgtm
/approve
Let's get the proposal in first.

[google-oss-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bobgy, gaocegege, Jeffwan, terrytangyuan, tizhou86

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Bobgy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Bobgy]

/unhold

UPDATE: for adoption progress, refer to #520