Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uber Issue: Manifests WG Steps for release 1.3 #1735

Closed
yanniszark opened this issue Feb 5, 2021 · 5 comments
Closed

Uber Issue: Manifests WG Steps for release 1.3 #1735

yanniszark opened this issue Feb 5, 2021 · 5 comments
Assignees

Comments

@yanniszark
Copy link
Contributor

yanniszark commented Feb 5, 2021

Intro

I know that many are eager to track the progress of the manifests working group,
so I'm opening this issue to document the plan, as was presented in the
wg-manifests meeting on 2021/02/05
(recording).

More details about the mission of wg-manifests can be found in its charter:
https://github.com/kubeflow/community/blob/master/wg-manifests/charter.md

Plan

This plan will be updated regularly.

Step 1: Migrate Manifests to a new Folder Structure

Currently, the manifests repo is a flat namespace with everything mangled.
We plan to reorganize the folder structure into sections that will reflect the
following properties:

  • Applications with their source code in upstream KF maintained by Kubeflow WGs (e.g., notebook-controller)
  • Common services (Dex, Istio, Cert-Manager, KNative), maintained by the
    Manifests WG.
  • Applications contributed by community members, not owned by any Kubeflow WG.
  • Distribution-specific manifests (kfdef, stacks, aws, gcp, etc.). The goal is
    for this folder to become empty in subsequent releases.

Issue: #1738

Step 2: Consolidate manifests development upstream

Consolidate manifests development upstream. Move all manifests from
kubeflow/manifests to their respective upstream repos.

To give an example from the notebook-controller, the diagram shows the current
situation. Manifests are in two repos, blurring separation of responsibilities
and making things harder for application developers.

diagram_1

Instead, we will copy all manifests from the manifests repo back to each
upstream's repo. From there, they will be imported in the manifests repo. The
following diagram presents the desired state:

diagram_2

Issue: #1740

Step 3: Upgrade Istio

The manifests working group is responsible for certain common services (Dex,
Istio, Cert-Manager, KNative). Out of those, we plan to provide a newer version
of Istio, to leverage the new AuthorizationPolicy API that will be supported
across the board in 1.3.

The work to refactor every component to use Istio AuthorizationPolicy is tracked
in kubeflow/kubeflow#5313 (comment).

Issue: #1753

Step 4: Consolidate manifests with multiple versions into 1

Some manifests (e.g., notebooks controller, katib) have multiple versions. Take
the example of notebook-controller:

diagram_3

There are the following versions of manifests:

  • The ones living in the upstream repo, used in development and testing.
  • The base and overlays folders, in kubeflow/manifests. These were
    optimized to work with kfctl's kustomization combination magic.
  • The base_v3 folder, designed to eschew vars and be buildable with
    kustomize. It imports files from base.

There is no need for such complexity and duplication. We will provide an example
(of the notebook-controller) and guide for how to consolidate them into a
proposed structure.

Deadlines

Date Steps
2021/02/09 Steps 1-3

Common Questions

  • Q: Will kfctl be supported?
    A: Kubeflow applications publish kustomize packages. Any tool that can
    work with kustomizations should work. Manifests are not tied to any specific
    deployment tool and more information on kfctl development can be found in
    the deployments-wg.

  • Q: What Istio version will be included?
    A: Will be decided as part of step (3) of the plan above. In principle,
    you can use any version of your choosing, as long as it supports
    AuthorizationPolicy (1.4+).

cc @kubeflow/wg-manifests-leads

@Bobgy
Copy link
Contributor

Bobgy commented Feb 11, 2021

This is absolutely awesome! Thank you so much for driving these! @yanniszark

yanniszark added a commit to arrikto/kubeflow that referenced this issue Feb 15, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Notebook Controller`
from path `apps/jupyter/notebook-controller/upstream` of kubeflow/manifests to path
`components/notebook-controller/config` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
yanniszark added a commit to arrikto/kubeflow that referenced this issue Feb 15, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Central Dashboard`
from path `apps/centraldashboard/upstream` of kubeflow/manifests to path
`components/centraldashboard/manifests` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
yanniszark added a commit to arrikto/kubeflow that referenced this issue Feb 15, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Profiles + KFAM`
from path `apps/profiles/upstream` of kubeflow/manifests to path
`components/profile-controller/config` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
yanniszark added a commit to arrikto/kubeflow that referenced this issue Feb 15, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Profiles + KFAM`
from path `apps/profiles/upstream` of kubeflow/manifests to path
`components/profile-controller/config` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
yanniszark added a commit to arrikto/kubeflow that referenced this issue Feb 15, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `PodDefaults Webhook`
from path `apps/admission-webhook/upstream` of kubeflow/manifests to path
`components/admission-webhook/manifests` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
yanniszark added a commit to arrikto/katib that referenced this issue Mar 4, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Katib`
from path `apps/katib/upstream` of kubeflow/manifests to path
`manifests/v1beta1` of the upstream repo (https://github.com/kubeflow/katib).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
yanniszark added a commit to arrikto/katib that referenced this issue Mar 4, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Katib`
from path `apps/katib/upstream` of kubeflow/manifests to path
`manifests/v1beta1` of the upstream repo (https://github.com/kubeflow/katib).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
google-oss-robot pushed a commit to kubeflow/kubeflow that referenced this issue Mar 4, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Jupyter Web App`
from path `apps/jupyter/jupyter-web-app/upstream` of kubeflow/manifests to path
`components/crud-web-apps/jupyter/manifests` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
yanniszark added a commit to arrikto/katib that referenced this issue Mar 6, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Katib`
from path `apps/katib/upstream` of kubeflow/manifests to path
`manifests/v1beta1` of the upstream repo (https://github.com/kubeflow/katib).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
yanniszark added a commit to arrikto/katib that referenced this issue Mar 6, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Katib`
from path `apps/katib/upstream` of kubeflow/manifests to path
`manifests/v1beta1` of the upstream repo (https://github.com/kubeflow/katib).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
google-oss-robot pushed a commit to kubeflow/katib that referenced this issue Mar 6, 2021
* manifests: Move manifests development upstream

As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Katib`
from path `apps/katib/upstream` of kubeflow/manifests to path
`manifests/v1beta1` of the upstream repo (https://github.com/kubeflow/katib).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>

* manifests: Fold base, overlays into components

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
juliusvonkohout pushed a commit to juliusvonkohout/kubeflow that referenced this issue Mar 7, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Central Dashboard`
from path `apps/centraldashboard/upstream` of kubeflow/manifests to path
`components/centraldashboard/manifests` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
juliusvonkohout pushed a commit to juliusvonkohout/kubeflow that referenced this issue Mar 7, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `PodDefaults Webhook`
from path `apps/admission-webhook/upstream` of kubeflow/manifests to path
`components/admission-webhook/manifests` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
juliusvonkohout pushed a commit to juliusvonkohout/kubeflow that referenced this issue Mar 7, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Profiles + KFAM`
from path `apps/profiles/upstream` of kubeflow/manifests to path
`components/profile-controller/config` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
juliusvonkohout pushed a commit to juliusvonkohout/kubeflow that referenced this issue Mar 7, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Jupyter Web App`
from path `apps/jupyter/jupyter-web-app/upstream` of kubeflow/manifests to path
`components/crud-web-apps/jupyter/manifests` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
yanniszark added a commit to arrikto/kubeflow-mxnet-operator that referenced this issue Mar 7, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `MXNet Operator`
from path `apps/mxnet-job/upstream` of kubeflow/manifests to path
`manifests` of the upstream repo (https://github.com/kubeflow/mxnet-operator).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
google-oss-robot pushed a commit to kubeflow/training-operator that referenced this issue Mar 8, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `TFJob Operator`
from path `apps/tf-training/upstream` of kubeflow/manifests to path
`manifests` of the upstream repo (https://github.com/kubeflow/tf-operator).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
Subreptivus pushed a commit to equinor/kubeflow that referenced this issue Mar 10, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Notebook Controller`
from path `apps/jupyter/notebook-controller/upstream` of kubeflow/manifests to path
`components/notebook-controller/config` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
Subreptivus pushed a commit to equinor/kubeflow that referenced this issue Mar 10, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Central Dashboard`
from path `apps/centraldashboard/upstream` of kubeflow/manifests to path
`components/centraldashboard/manifests` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
Subreptivus pushed a commit to equinor/kubeflow that referenced this issue Mar 10, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `PodDefaults Webhook`
from path `apps/admission-webhook/upstream` of kubeflow/manifests to path
`components/admission-webhook/manifests` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
Subreptivus pushed a commit to equinor/kubeflow that referenced this issue Mar 10, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Profiles + KFAM`
from path `apps/profiles/upstream` of kubeflow/manifests to path
`components/profile-controller/config` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
Subreptivus pushed a commit to equinor/kubeflow that referenced this issue Mar 10, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Jupyter Web App`
from path `apps/jupyter/jupyter-web-app/upstream` of kubeflow/manifests to path
`components/crud-web-apps/jupyter/manifests` of the upstream repo (https://github.com/kubeflow/kubeflow).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
google-oss-robot pushed a commit to kubeflow/kubebench that referenced this issue Mar 23, 2021
As part of the work of wg-manifests for 1.3
(kubeflow/manifests#1735), we are moving manifests
development in upstream repos. This gives the application developers full
ownership of their manifests, tracked in a single place.

This commit copies the manifests for application `Kubebench`
from path `apps/kubebench/upstream` of kubeflow/manifests to path
`manifests` of the upstream repo (https://github.com/kubeflow/kubebench).

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>
@yanniszark
Copy link
Contributor Author

All steps completed, closing issue.

Needs Triage automation moved this from To Do to Closed Apr 14, 2021
@kubeflow-bot kubeflow-bot removed this from Closed in Needs Triage Apr 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants