From 0ca0445cb4a0d084fd61c5c2d87bf590a369630d Mon Sep 17 00:00:00 2001 From: Jake Date: Thu, 6 May 2021 11:45:13 +0100 Subject: [PATCH 01/12] add multi-repo profile proposal --- docs/proposal-01-multi-profile-repo.md | 271 +++++++++++++++++++++++++ 1 file changed, 271 insertions(+) create mode 100644 docs/proposal-01-multi-profile-repo.md diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md new file mode 100644 index 00000000..9ad8c909 --- /dev/null +++ b/docs/proposal-01-multi-profile-repo.md @@ -0,0 +1,271 @@ + + +# Multi-profile repos with versioning + + + +## Authors + +Jake Klein @aclevername + +## Status + +WIP + + + +## Table of Contents + +- [Summary](#summary) +- [Design Details](#design-details) +- [Drawbacks (Optional)](#drawbacks-optional) +- [Open Questions / Known Unknowns](#open-questions--known-unknowns) + + +## Summary + + +The purpose of this doc is to outline the proposed implementation of multi-profile repositories, and how to handle tagging profiles in this scenario. +Currently you are limited to 1 profile per repository and you can only reference a branch in a repository. This document proposes that +we introduce the ability to have multiple profiles in a repository, and follow a pattern of tagging used in [kustomize](https://github.com/kubernetes-sigs/kustomize/tags) +to distinguish what tag is used for each profile. + + +## Design Details +### Repository Layout + +An example repository `weaveworks-profiles` containing two profiles, `foo` and `bar` would be laid out as follows: +```bash +$ ls weaveworks-profiles +foo/ +bar/ +``` + +Inside `foo` and `bar` it would contain the `profiles.yaml` and any local artifacts, for example: +```bash +$ tree weaveworks-profile/foo +├── charts +│   └── nginx +│   └── Chart.yaml +├── kustomize +│   └── web-app +│   └── deployment.yaml +└── profile.yaml + +``` +The `profile.yaml` would contain a reference to the artifacts relative to profile directory, for example: +```bash +$ cat foo/profile.yaml +apiVersion: profiles.fluxcd.io/v1alpha1 +kind: Profile +metadata: + name: foo +spec: + description: Profile for deploying local nginx chart and web-app + version: v0.0.2 + artifacts: + - name: nginx-server + path: charts/nginx + kind: HelmChart + - name: web-app + path: kustomize/web-app + kind: Kustomize +``` + +### Tagging +[kustomize](https://github.com/kubernetes-sigs/kustomize/tags) is an example of a repository that contains multiple "things" that are all versioned +independently. For example if you take a look at some of the tags it has: +``` +kyaml/v0.10.19 +kyaml/v0.10.18 +api/v0.8.9 +api/v0.8.8 +cmd/config/v0.9.11 +cmd/config/v0.9.10 +``` + +The prefix before the semver version corresponds to the directory in which the product lives, with the last directory also being the name of the component. For example the repository contains: +``` +tree kustomize/ +├── api + └── main.go + ... +├── cmd +│   └── config + └── main.go + ... +└── kyaml + └── main.go + ... +``` + +If you wanted to get version `v0.8.9` of `api` you would checkout to tag `api/v0.8.9` and inside the `api/` directory it would contain the desired code. +The case of `cmd/config` is slightly less clean, its easier to write automation around detecting new profiles if the profile directory is always at the top +level directory. Support for sub-directories is in the [Open Questions / Known Unknowns](#open-questions--known-unknowns) section below. + +This approach could be used for versioning profiles in a repository. + +### Profile Subscription modifications +Currently we only support referencing a repository as follows: +``` +apiVersion: weave.works/v1alpha1 +kind: ProfileSubscription +metadata: + name: nginx-profile-test + namespace: default +spec: + profileURL: https://github.com/weaveworks/nginx-profile + branch: main +``` + +We want to maintain support for `branch` workflows as its useful for development, while also introducing a new `tags` approach. There are two possible approaches: + +#### Approach 1 + +``` +apiVersion: weave.works/v1alpha1 +kind: ProfileSubscription +metadata: + name: foo-test + namespace: default +spec: + profileURL: https://github.com/weaveworks/weaveworks-profiles + tag: v0.1.2 + profile: foo +``` + +``` +apiVersion: weave.works/v1alpha1 +kind: ProfileSubscription +metadata: + name: bar-test + namespace: default +spec: + profileURL: https://github.com/weaveworks/weaveworks-profiles + branch: main + profile: bar +``` + +This approach hides the fact that the git tag contains the profile name, and leaves it up to the profiles controller to concatinate the `profile` and `tag` value together. +This introduces a common `profile` (or `profileName`) field that is shared across the two for us to know which profile in the repository we are referencing and its directory (they must be equal). + +#### Approach 2 + +``` +apiVersion: weave.works/v1alpha1 +kind: ProfileSubscription +metadata: + name: foo-test + namespace: default +spec: + profileURL: https://github.com/weaveworks/weaveworks-profiles + tag: foo/v0.1.2 +``` + +``` +apiVersion: weave.works/v1alpha1 +kind: ProfileSubscription +metadata: + name: bar-test + namespace: default +spec: + profileURL: https://github.com/weaveworks/weaveworks-profiles + branch: main + path: bar/ +``` + +This approach makes it clear upon inspection what tag is actually being referenced in the repository, but is less clear what directory/profile inside the repository is being referenced. +It also maintains two completely different approach for using `branch` vs `tag` + +## Drawbacks (Optional) + +- This approach to tagging repositories that contain multiple "things" is not widely used, users will likely not be familiar with it + +## Open Questions / Known Unknowns + +Support for specifying sub-directories in the tag allow more granular repository layouts, but make parsing tags less clean. For example if my repository is as follows: + +```bash +$ ls weaveworks-profiles +foo/ +bar/ +``` + +``` +bar/v0.10.19 +bar/v0.10.18 +foo/v0.1.1 +foo/v0.1.0 +``` +Its clear that it has two profiles `foo` and `bar`, and that they have there respective tags. If we added support for sub-directories we introduce more complexity, for example: + +```bash +$ ls weaveworks-profiles +foo/ +bar/ +sub/ + path/ + foo/ +``` + +``` +bar/v0.10.19 +bar/v0.10.18 +foo/v0.1.1 +foo/v0.1.0 +sub/path/foo/v1.1.0 +sub/path/foo/v1.0.0 +``` + +Here we have to different profiles, both called `foo` who are only distinguishable by there paths. If we were to write automation to parse profile repositories we might find it awkward +to handle such scenarios. It would also result in more obscure subscription definitions, for example to adapt approach 1 would look like: +``` +apiVersion: weave.works/v1alpha1 +kind: ProfileSubscription +metadata: + name: foo-test + namespace: default +spec: + profileURL: https://github.com/weaveworks/weaveworks-profiles + tag: v0.1.2 + profile: foo + path: sub/path +``` + +or + +``` +apiVersion: weave.works/v1alpha1 +kind: ProfileSubscription +metadata: + name: foo-test + namespace: default +spec: + profileURL: https://github.com/weaveworks/weaveworks-profiles + tag: v0.1.2 + profile: sub/path/foo +``` From b46fff92105d0cf0b2ec6fb62da390b6d5437b83 Mon Sep 17 00:00:00 2001 From: Jake Date: Thu, 6 May 2021 12:09:11 +0100 Subject: [PATCH 02/12] update --- docs/proposal-01-multi-profile-repo.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index 9ad8c909..14ce8554 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -78,7 +78,7 @@ $ tree weaveworks-profile/foo ``` The `profile.yaml` would contain a reference to the artifacts relative to profile directory, for example: -```bash +```yaml $ cat foo/profile.yaml apiVersion: profiles.fluxcd.io/v1alpha1 kind: Profile @@ -146,7 +146,7 @@ We want to maintain support for `branch` workflows as its useful for development #### Approach 1 -``` +```yaml apiVersion: weave.works/v1alpha1 kind: ProfileSubscription metadata: @@ -158,7 +158,7 @@ spec: profile: foo ``` -``` +```yaml apiVersion: weave.works/v1alpha1 kind: ProfileSubscription metadata: @@ -175,7 +175,7 @@ This introduces a common `profile` (or `profileName`) field that is shared acros #### Approach 2 -``` +```yaml apiVersion: weave.works/v1alpha1 kind: ProfileSubscription metadata: @@ -186,7 +186,7 @@ spec: tag: foo/v0.1.2 ``` -``` +```yaml apiVersion: weave.works/v1alpha1 kind: ProfileSubscription metadata: @@ -243,7 +243,7 @@ sub/path/foo/v1.0.0 Here we have to different profiles, both called `foo` who are only distinguishable by there paths. If we were to write automation to parse profile repositories we might find it awkward to handle such scenarios. It would also result in more obscure subscription definitions, for example to adapt approach 1 would look like: -``` +```yaml apiVersion: weave.works/v1alpha1 kind: ProfileSubscription metadata: @@ -258,7 +258,7 @@ spec: or -``` +```yaml apiVersion: weave.works/v1alpha1 kind: ProfileSubscription metadata: From 7243c2e0976e0b9f9392bca2241498ec8f43a900 Mon Sep 17 00:00:00 2001 From: Jake Klein Date: Thu, 6 May 2021 13:30:48 +0100 Subject: [PATCH 03/12] Update docs/proposal-01-multi-profile-repo.md Co-authored-by: Claudia --- docs/proposal-01-multi-profile-repo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index 14ce8554..6adcd562 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -170,7 +170,7 @@ spec: profile: bar ``` -This approach hides the fact that the git tag contains the profile name, and leaves it up to the profiles controller to concatinate the `profile` and `tag` value together. +This approach hides the fact that the git tag contains the profile name, and leaves it up to the profiles controller to concatenate the `profile` and `tag` value together. This introduces a common `profile` (or `profileName`) field that is shared across the two for us to know which profile in the repository we are referencing and its directory (they must be equal). #### Approach 2 From 2ee7b56d71cfbab094629df3e0ad68211be9b3c7 Mon Sep 17 00:00:00 2001 From: Jake Klein Date: Thu, 6 May 2021 13:30:55 +0100 Subject: [PATCH 04/12] Update docs/proposal-01-multi-profile-repo.md Co-authored-by: Claudia --- docs/proposal-01-multi-profile-repo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index 6adcd562..2cdc7208 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -241,7 +241,7 @@ sub/path/foo/v1.1.0 sub/path/foo/v1.0.0 ``` -Here we have to different profiles, both called `foo` who are only distinguishable by there paths. If we were to write automation to parse profile repositories we might find it awkward +Here we have two different profiles, both called `foo` which are only distinguishable by their paths. If we were to write automation to parse profile repositories we might find it awkward to handle such scenarios. It would also result in more obscure subscription definitions, for example to adapt approach 1 would look like: ```yaml apiVersion: weave.works/v1alpha1 From a9c049e88252d7fc221b7fd5d63aa73e70ccce95 Mon Sep 17 00:00:00 2001 From: Jake Klein Date: Thu, 6 May 2021 13:30:59 +0100 Subject: [PATCH 05/12] Update docs/proposal-01-multi-profile-repo.md Co-authored-by: Claudia --- docs/proposal-01-multi-profile-repo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index 2cdc7208..42dea3ce 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -142,7 +142,7 @@ spec: branch: main ``` -We want to maintain support for `branch` workflows as its useful for development, while also introducing a new `tags` approach. There are two possible approaches: +We want to maintain support for `branch` workflows as it's useful for development, while also introducing a new `tags` approach. There are two possible approaches: #### Approach 1 From a04c39807d11f1be58da4c09e9644300c9c7b4e1 Mon Sep 17 00:00:00 2001 From: Jake Klein Date: Thu, 6 May 2021 13:31:05 +0100 Subject: [PATCH 06/12] Update docs/proposal-01-multi-profile-repo.md Co-authored-by: Claudia --- docs/proposal-01-multi-profile-repo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index 42dea3ce..00785a98 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -199,7 +199,7 @@ spec: ``` This approach makes it clear upon inspection what tag is actually being referenced in the repository, but is less clear what directory/profile inside the repository is being referenced. -It also maintains two completely different approach for using `branch` vs `tag` +It also maintains two completely different approaches for using `branch` vs `tag` ## Drawbacks (Optional) From 23f4d2995bb95d98145a326e8dd4c67b68b48e75 Mon Sep 17 00:00:00 2001 From: Jake Klein Date: Thu, 6 May 2021 13:31:10 +0100 Subject: [PATCH 07/12] Update docs/proposal-01-multi-profile-repo.md Co-authored-by: Gergely Brautigam --- docs/proposal-01-multi-profile-repo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index 00785a98..49311032 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -207,7 +207,7 @@ It also maintains two completely different approaches for using `branch` vs `tag ## Open Questions / Known Unknowns -Support for specifying sub-directories in the tag allow more granular repository layouts, but make parsing tags less clean. For example if my repository is as follows: +Support for specifying sub-directories in the tag allow more granular repository layouts, but makes parsing tags less clean. For example if my repository is as follows: ```bash $ ls weaveworks-profiles From beae932854e5cc4b3e70dbdc881734f27a1f65c9 Mon Sep 17 00:00:00 2001 From: Jake Date: Thu, 6 May 2021 13:33:20 +0100 Subject: [PATCH 08/12] rename tags -> version --- docs/proposal-01-multi-profile-repo.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index 49311032..d8c105f3 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -154,7 +154,7 @@ metadata: namespace: default spec: profileURL: https://github.com/weaveworks/weaveworks-profiles - tag: v0.1.2 + version: v0.1.2 profile: foo ``` @@ -183,7 +183,7 @@ metadata: namespace: default spec: profileURL: https://github.com/weaveworks/weaveworks-profiles - tag: foo/v0.1.2 + version: foo/v0.1.2 ``` ```yaml @@ -251,7 +251,7 @@ metadata: namespace: default spec: profileURL: https://github.com/weaveworks/weaveworks-profiles - tag: v0.1.2 + version: v0.1.2 profile: foo path: sub/path ``` @@ -266,6 +266,6 @@ metadata: namespace: default spec: profileURL: https://github.com/weaveworks/weaveworks-profiles - tag: v0.1.2 + version: v0.1.2 profile: sub/path/foo ``` From a9a3387e62ae943bb353e6e534d686b3e703341b Mon Sep 17 00:00:00 2001 From: Jake Klein Date: Thu, 6 May 2021 13:34:53 +0100 Subject: [PATCH 09/12] Update docs/proposal-01-multi-profile-repo.md Co-authored-by: Claudia --- docs/proposal-01-multi-profile-repo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index d8c105f3..a4b6a227 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -124,7 +124,7 @@ tree kustomize/ ``` If you wanted to get version `v0.8.9` of `api` you would checkout to tag `api/v0.8.9` and inside the `api/` directory it would contain the desired code. -The case of `cmd/config` is slightly less clean, its easier to write automation around detecting new profiles if the profile directory is always at the top +The case of `cmd/config` is slightly less clean, it's easier to write automation around detecting new profiles if the profile directory is always at the top level directory. Support for sub-directories is in the [Open Questions / Known Unknowns](#open-questions--known-unknowns) section below. This approach could be used for versioning profiles in a repository. From ab9b82360ade651ed814adefaebc2b73ef054ace Mon Sep 17 00:00:00 2001 From: Jake Klein Date: Thu, 6 May 2021 13:34:58 +0100 Subject: [PATCH 10/12] Update docs/proposal-01-multi-profile-repo.md Co-authored-by: Claudia --- docs/proposal-01-multi-profile-repo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index a4b6a227..e2f767ab 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -221,7 +221,7 @@ bar/v0.10.18 foo/v0.1.1 foo/v0.1.0 ``` -Its clear that it has two profiles `foo` and `bar`, and that they have there respective tags. If we added support for sub-directories we introduce more complexity, for example: +It's clear that it has two profiles `foo` and `bar`, and that they have their respective tags. If we added support for sub-directories we introduce more complexity, for example: ```bash $ ls weaveworks-profiles From d5695e0421b4afe9d09ac39fd6cfaf496fa2d542 Mon Sep 17 00:00:00 2001 From: Jake Date: Thu, 6 May 2021 13:43:19 +0100 Subject: [PATCH 11/12] update --- docs/proposal-01-multi-profile-repo.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index e2f767ab..d484fbeb 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -215,6 +215,7 @@ foo/ bar/ ``` +with tags: ``` bar/v0.10.19 bar/v0.10.18 @@ -232,6 +233,7 @@ sub/ foo/ ``` +with tags: ``` bar/v0.10.19 bar/v0.10.18 From f155bfa9990d7027a396c88e0ea04a0e6780ce60 Mon Sep 17 00:00:00 2001 From: Jake Date: Fri, 7 May 2021 13:53:50 +0100 Subject: [PATCH 12/12] update --- docs/proposal-01-multi-profile-repo.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/docs/proposal-01-multi-profile-repo.md b/docs/proposal-01-multi-profile-repo.md index d484fbeb..b894a210 100644 --- a/docs/proposal-01-multi-profile-repo.md +++ b/docs/proposal-01-multi-profile-repo.md @@ -24,7 +24,7 @@ Jake Klein @aclevername ## Status -WIP +Approved