Continue to add support for git urls as dependency repositories #9482

yxxhero · 2021-03-14T13:47:56Z

Continue the PR #6734 work.
related issue: #9461

If applicable:

this PR contains documentation
this PR contains unit tests
this PR has been tested for backwards compatibility

jdolitsky

Can you please implement the Getter interface for the git:// protocol? This will allow it to be used for helm install as well.

Here is an example PR for how this was done for oci://: #8843

yxxhero · 2021-03-16T00:35:51Z

ok, I will do this work as soon as I can. Thanks for your reply. @jdolitsky

BlueCog · 2021-04-02T06:54:13Z

Hi @yxxhero , is there anything i can help you with?

And one more question: is it also possible to give a contextDir with a chart in the Chart.yaml? For the usecase when the chart is not directly in the root of the Git repository (see example below)

# Chart.yaml
dependencies:
- name: nginx
  version: "1.2.3"
  repository: "ssh://git@git.host:7999/helm-dep/helm-dep.git"
  contextDir: "helm/chart"

yxxhero · 2021-04-05T11:59:44Z

Hi @yxxhero , is there anything i can help you with?

And one more question: is it also possible to give a contextDir with a chart in the Chart.yaml? For the usecase when the chart is not directly in the root of the Git repository (see example below)
# Chart.yaml
dependencies:
- name: nginx
  version: "1.2.3"
  repository: "ssh://git@git.host:7999/helm-dep/helm-dep.git"
  contextDir: "helm/chart" 

I'm busy recently. I'll try to finish it Thanks very much.

yxxhero · 2021-04-24T10:13:31Z

@jdolitsky ping

yxxhero · 2021-04-26T06:10:17Z

@technosophos @mattfarina @bacongobbler @jdolitsky Look forward to your reply. Thanks very much.

BlueCog · 2021-04-29T08:19:57Z

Hey @yxxhero great work!

From the inline docs i get that it is not possible to use subdirectories (contextdir) with this Git functionality:

* The helm chart and 'Chart.yaml' must be in the root of the git repo.
The chart cannot be loaded from a subdirectory.

Is this a design choice? Or is it functionality what is possible to add? For example we (my team) have more files and directories in our helm-chart-repo's. Files needed for CICD pipelines for example.

So our repo would be:

\chart-cicd\helm-chart
           \cicd-files

So when referencing the helm chart in the dependencies we would like to reference the subdir helm-chart from the repo chart-cicd

yxxhero · 2021-04-29T09:14:59Z

Hey @yxxhero great work!

From the inline docs i get that it is not possible to use subdirectories (contextdir) with this Git functionality:
* The helm chart and 'Chart.yaml' must be in the root of the git repo.
The chart cannot be loaded from a subdirectory.
Is this a design choice? Or is it functionality what is possible to add? For example we (my team) have more files and directories in our helm-chart-repo's. Files needed for CICD pipelines for example.

So our repo would be:
\chart-cicd\helm-chart
           \cicd-files 
So when referencing the helm chart in the dependencies we would like to reference the subdir helm-chart from the repo chart-cicd

The current design is like this. Of course you can change it and work together to do it. @BlueCog

yxxhero · 2021-04-29T10:47:12Z

I'm going to add contextDir features in my own repo. If it is done. You can test it. Thanks very much. @BlueCog

yxxhero · 2021-05-06T07:13:28Z

@bacongobbler @jdolitsky Does this feature require design documentation? Thanks very much.

cmd/helm/dependency.go

BlueCog · 2021-05-06T09:44:30Z

@yxxhero

FYI: i've build a Helm client with your PR and tested the functionality (private and public repo's). It works without any issue.

Greets

yxxhero · 2021-06-02T03:32:13Z

@jdolitsky ping

pkg/getter/gitgetter.go

pkg/gitutil/gitutil.go

internal/resolver/resolver.go

pkg/downloader/chart_downloader.go

pkg/downloader/manager.go

pkg/getter/gitgetter.go

cmd/helm/dependency.go

rblaine95 · 2021-06-23T12:51:14Z

I built the last commit from this PR and am getting the following error when trying to helmy dep up

» helmy version
version.BuildInfo{Version:"v3.6.1-yxx-1+unreleased", GitCommit:"eef224055ea6457c5a628a76815b0de5b0efe2c1", GitTreeState:"clean", GoVersion:"go1.16.5"}

» helmy dep up
Hang tight while we grab the latest from your chart repositories...
[... trim ...]
Update Complete. ⎈Happy Helming!⎈
Saving 2 charts
Downloading helm-example from repo git://git@bitbucket.org:example/helm-example.git
Save error occurred:  could not download git://git@bitbucket.org:example/helm-example.git: mkdir /tmp/helm1319646873: not a directory
Deleting newly downloaded charts, restoring pre-update state
Error: could not download git://git@bitbucket.org:example/helm-example.git: mkdir /tmp/helm1319646873: not a directory

Another issue I'm getting is, if the name of the dependency doesn't match the name of the repo, I get this error:

dependencies:
  - name: library
    version: 1.2.3
    repository: git://git@bitbucket.org:example/helm-library.git

» helmy dep up
Hang tight while we grab the latest from your chart repositories...
[... trim ...]
Update Complete. ⎈Happy Helming!⎈
Error: The name of dependency git://git@bitbucket.org:example/helm-library.git should be helm-library

In this case, the name of the chart is library but the repo is named helm-library.
The name field in the Chart.yaml is also library.

yxxhero · 2021-06-23T15:12:49Z

I built the last commit from this PR and am getting the following error when trying to helmy dep up

» helmy version
version.BuildInfo{Version:"v3.6.1-yxx-1+unreleased", GitCommit:"eef224055ea6457c5a628a76815b0de5b0efe2c1", GitTreeState:"clean", GoVersion:"go1.16.5"}

» helmy dep up
Hang tight while we grab the latest from your chart repositories...
[... trim ...]
Update Complete. ⎈Happy Helming!⎈
Saving 2 charts
Downloading helm-example from repo git://git@bitbucket.org:example/helm-example.git
Save error occurred:  could not download git://git@bitbucket.org:example/helm-example.git: mkdir /tmp/helm1319646873: not a directory
Deleting newly downloaded charts, restoring pre-update state
Error: could not download git://git@bitbucket.org:example/helm-example.git: mkdir /tmp/helm1319646873: not a directory

Another issue I'm getting is, if the name of the dependency doesn't match the name of the repo, I get this error:

dependencies:
  - name: library
    version: 1.2.3
    repository: git://git@bitbucket.org:example/helm-library.git

» helmy dep up
Hang tight while we grab the latest from your chart repositories...
[... trim ...]
Update Complete. ⎈Happy Helming!⎈
Error: The name of dependency git://git@bitbucket.org:example/helm-library.git should be helm-library

In this case, the name of the chart is library but the repo is named helm-library.
The name field in the Chart.yaml is also library.

mkdir /tmp/helm1319646873: not a directory fix. I used os.CreateTemp by mistake.
you shuld change like below:

dependencies:
  - name: helm-library
    version: 1.2.3
    repository: git://git@bitbucket.org:example/helm-library.git

rblaine95 · 2021-06-24T08:19:34Z

Great, rebuilt the latest commit (416efce) and changed the name of the dependency to helm-library to match the name of the repo.

» helmy version
version.BuildInfo{Version:"v3.6.1-yxx-1+unreleased", GitCommit:"416efce3c2859d182ceaa6f739018bd46ab13259", GitTreeState:"clean", GoVersion:"go1.16.5"}

» helmy dep up
Hang tight while we grab the latest from your chart repositories...
[... trim ...]
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading helm-library from repo git://git@bitbucket.org:example/helm-library.git
Deleting outdated charts

One question, does the name of the dependency have to match the name of the repo?
I ask this because, in my org, we use the helm- prefix to distinguish between applications and helm charts that deploy said applications (or libraries).
We named our internal library chart libs, but the repo helm-libs.
Just like scala-libs or jenkins-libs

I've noticed in umbrella/master charts if the name/version of a dependency doesn't match the name/version of the downloaded chart, helm values don't propagate downwards.

I'll craft a reproducible example and post it here a little later, maybe this issue is outside of the scope of this PR.

Edit: I'll see what I can do regarding the example, I may be mistaken due to having used a build of the old git PR and possible issues in that version.

yxxhero · 2021-06-24T08:25:15Z

Great, rebuilt the latest commit (416efce) and changed the name of the dependency to helm-library to match the name of the repo.
» helmy version
version.BuildInfo{Version:"v3.6.1-yxx-1+unreleased", GitCommit:"416efce3c2859d182ceaa6f739018bd46ab13259", GitTreeState:"clean", GoVersion:"go1.16.5"}

» helmy dep up
Hang tight while we grab the latest from your chart repositories...
[... trim ...]
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading helm-library from repo git://git@bitbucket.org:example/helm-library.git
Deleting outdated charts
One question, does the name of the dependency have to match the name of the repo?
I ask this because, in my org, we use the helm- prefix to distinguish between applications and helm charts that deploy said applications (or libraries).
We named our internal library chart libs, but the repo helm-libs.
Just like scala-libs or jenkins-libs

I've noticed in umbrella/master charts if the name/version of a dependency doesn't match the name/version of the downloaded chart, helm values don't propagate downwards.

I'll craft a reproducible example and post it here a little later, maybe this issue is outside of the scope of this PR.

@rblaine95 I will find a way to remove this restriction. wait for your example. I will use your example to test it.

yxxhero · 2021-07-31T09:48:29Z

@jdolitsky @technosophos ping

rblaine95 · 2021-10-06T14:02:36Z

Any news regarding this PR?
We're very excited to use this feature 🚀

yxxhero · 2021-10-07T00:52:57Z

The PR is already available. I have written HIP and will actively promote the merger of PR. @rblaine95 @BlueCog @pmahoney-raise
Thank you for your attention.

rblaine95 · 2021-10-18T08:54:52Z

Setting version to a non-semantic version (for example: a branch name like feat/refactor-ingress) fails:

» helm version
version.BuildInfo{Version:"v3.7+unreleased", GitCommit:"edeeaabb294ea99aecd6753823010d2508c0fd02", GitTreeState:"clean", GoVersion:"go1.17.2"}

» HELM_EXPERIMENTAL_GIT=1 helm dep up
Hang tight while we grab the latest from your chart repositories...
[...]
Update Complete. ⎈Happy Helming!⎈
Error: dependency "zenlibs" has an invalid version/constraint format: improper constraint: feat/refactor-ingress

dependencies:
  - name: zenlibs
    version: feat/refactor-ingress
    repository: git://git@bitbucket.org:zenaptix/helm-zenlibs.git

yxxhero · 2021-10-18T09:03:18Z

@rblaine95 This is the current default behavior. Do you have any idea?

rblaine95 · 2021-10-18T09:05:08Z

I thought this PR supports branch names and tags?

yxxhero · 2021-10-18T09:07:53Z

I'll consider your suggestion. @rblaine95 Thanks very much.

rblaine95 · 2021-10-18T09:09:54Z

I'll consider your suggestion. @rblaine95 Thanks very much.

It says here

When using a 'git://' repository, the 'version' must be a valid semantic tag or branch
name for the git repo. For example 'master'.

Edit:
internal/resolver/resolver.go:L127-130

if !found {
  return nil, fmt.Errorf(`dependency %q is missing git branch or tag: %s.
When using a "git://" type repository, the "version" should be a valid branch or tag name`, d.Name, d.Version)
}

pkg/getter/gitgetter.go:L67-69

if version == "" {
  return nil, fmt.Errorf("the version must be a valid tag or branch name for the git repo, not nil")
}

It looks like your PR already does support branch name, but there's something that checks the version constraint before the git checks which causes it to bomb out.

yxxhero · 2021-10-18T09:21:12Z

@rblaine95 cool, you are right.

yxxhero · 2022-01-22T14:27:20Z

@jdolitsky @technosophos

yxxhero · 2022-03-28T01:17:00Z

@jdolitsky @technosophos ping

Signed-off-by: yxxhero <aiopsclub@163.com>

Signed-off-by: Jeff Valore <rally25rs@yahoo.com>

Signed-off-by: yxxhero <aiopsclub@163.com>

Signed-off-by: yxxhero <aiopsclub@163.com> add license headers for gitutils Signed-off-by: yxxhero <aiopsclub@163.com>

Signed-off-by: yxxhero <aiopsclub@163.com>

jackivanov · 2022-08-23T12:28:06Z

In the tendency to GitOps it's sad to see this PR being closed. The maintains don't seem to appreciate the community effort at all.

yxxhero · 2022-08-23T12:32:21Z

@jdolitsky oh. sorry. It's my fault. I will create a PR. Thanks very much.

helm-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 14, 2021

jdolitsky requested changes Mar 15, 2021

View reviewed changes

yxxhero marked this pull request as draft March 16, 2021 11:39

helm-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 19, 2021

yxxhero marked this pull request as ready for review April 24, 2021 10:12

yxxhero requested a review from jdolitsky April 24, 2021 10:12

yxxhero mentioned this pull request Apr 24, 2021

Working with unpackaged charts as dependency #9461

Closed

BlueCog reviewed May 6, 2021

View reviewed changes

cmd/helm/dependency.go Show resolved Hide resolved

yxxhero changed the title ~~Support git url repository~~ Continue to add support for git urls as dependency repositories May 8, 2021

helm-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 2, 2021

cndoit18 reviewed Jun 2, 2021

View reviewed changes

pkg/getter/gitgetter.go Outdated Show resolved Hide resolved

pkg/gitutil/gitutil.go Outdated Show resolved Hide resolved

internal/resolver/resolver.go Outdated Show resolved Hide resolved

jdolitsky requested changes Jun 4, 2021

View reviewed changes

yxxhero requested a review from jdolitsky June 5, 2021 10:32

helm-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jul 24, 2021

yxxhero requested a review from technosophos July 24, 2021 07:11

bacongobbler mentioned this pull request Aug 23, 2021

Add support for git urls as dependency repositories #6734

Closed

3 tasks

yxxhero mentioned this pull request Jan 22, 2022

HIP for git support helm/community#214

Closed

patrick-stephens mentioned this pull request Mar 10, 2022

Helm chart tarball in release fluent/fluent-operator#230

Closed

yxxhero and others added 10 commits May 21, 2022 20:21

fix conflic in resolver package

ca7120c

Signed-off-by: yxxhero <aiopsclub@163.com>

fix 'helm dep build' command when chart contains 'git:' dependencies

4c86566

Signed-off-by: Jeff Valore <rally25rs@yahoo.com>

Implement the Getter interface for the git:// protocol.

0c58557

Signed-off-by: yxxhero <aiopsclub@163.com>

add unittest for git:// protocol

10c39e1

Signed-off-by: yxxhero <aiopsclub@163.com>

move GITGetter to GitGetter

23d91bb

Signed-off-by: yxxhero <aiopsclub@163.com>

fix TestCompressDirToTgz unittest

698c228

Signed-off-by: yxxhero <aiopsclub@163.com>

move gitutil to vcs

e195cda

Signed-off-by: yxxhero <aiopsclub@163.com>

update resolver unittest for git

23153d9

Signed-off-by: yxxhero <aiopsclub@163.com> add license headers for gitutils Signed-off-by: yxxhero <aiopsclub@163.com>

fix oci issue

417cc17

Signed-off-by: yxxhero <aiopsclub@163.com>

fix confict

16a45a6

Signed-off-by: yxxhero <aiopsclub@163.com>

yxxhero closed this by deleting the head repository Aug 23, 2022

yxxhero mentioned this pull request Aug 23, 2022

Support git url repository #11258

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Continue to add support for git urls as dependency repositories #9482

Continue to add support for git urls as dependency repositories #9482

yxxhero commented Mar 14, 2021 •

edited

Loading

jdolitsky left a comment

yxxhero commented Mar 16, 2021

BlueCog commented Apr 2, 2021

yxxhero commented Apr 5, 2021

yxxhero commented Apr 24, 2021

yxxhero commented Apr 26, 2021

BlueCog commented Apr 29, 2021

yxxhero commented Apr 29, 2021 •

edited

Loading

yxxhero commented Apr 29, 2021

yxxhero commented May 6, 2021

BlueCog commented May 6, 2021

yxxhero commented Jun 2, 2021

rblaine95 commented Jun 23, 2021

yxxhero commented Jun 23, 2021

rblaine95 commented Jun 24, 2021 •

edited

Loading

yxxhero commented Jun 24, 2021

yxxhero commented Jul 31, 2021

rblaine95 commented Oct 6, 2021

yxxhero commented Oct 7, 2021

rblaine95 commented Oct 18, 2021

yxxhero commented Oct 18, 2021

rblaine95 commented Oct 18, 2021

yxxhero commented Oct 18, 2021

rblaine95 commented Oct 18, 2021 •

edited

Loading

yxxhero commented Oct 18, 2021

yxxhero commented Jan 22, 2022

yxxhero commented Mar 28, 2022

jackivanov commented Aug 23, 2022

yxxhero commented Aug 23, 2022

Continue to add support for git urls as dependency repositories #9482

Continue to add support for git urls as dependency repositories #9482

Conversation

yxxhero commented Mar 14, 2021 • edited Loading

jdolitsky left a comment

Choose a reason for hiding this comment

yxxhero commented Mar 16, 2021

BlueCog commented Apr 2, 2021

yxxhero commented Apr 5, 2021

yxxhero commented Apr 24, 2021

yxxhero commented Apr 26, 2021

BlueCog commented Apr 29, 2021

yxxhero commented Apr 29, 2021 • edited Loading

yxxhero commented Apr 29, 2021

yxxhero commented May 6, 2021

BlueCog commented May 6, 2021

yxxhero commented Jun 2, 2021

rblaine95 commented Jun 23, 2021

yxxhero commented Jun 23, 2021

rblaine95 commented Jun 24, 2021 • edited Loading

yxxhero commented Jun 24, 2021

yxxhero commented Jul 31, 2021

rblaine95 commented Oct 6, 2021

yxxhero commented Oct 7, 2021

rblaine95 commented Oct 18, 2021

yxxhero commented Oct 18, 2021

rblaine95 commented Oct 18, 2021

yxxhero commented Oct 18, 2021

rblaine95 commented Oct 18, 2021 • edited Loading

yxxhero commented Oct 18, 2021

yxxhero commented Jan 22, 2022

yxxhero commented Mar 28, 2022

jackivanov commented Aug 23, 2022

yxxhero commented Aug 23, 2022

yxxhero commented Mar 14, 2021 •

edited

Loading

yxxhero commented Apr 29, 2021 •

edited

Loading

rblaine95 commented Jun 24, 2021 •

edited

Loading

rblaine95 commented Oct 18, 2021 •

edited

Loading