Support for hosted Helm repos

[derkoe]

• What are you trying to do?
  Creating a hosted Helm repository.

• What feature or behavior is this required for?
  To publish inhouse Helm Charts

• Anything else?
  Is there any plan to support hosted (and probably proxy) Helm repositories?

[DarthHater]

I'd like to do this, yeah! I have some of it in progress (I use an internal repo so I can move a bit quicker) but far from complete. Would you be into working with me on it @derkoe ?

[tomdavidson]

I'm not a Java developer, but anything I can help with?

[uliul-carpatin]

@DarthHater it would be nice to have a feature branch open with the current development..

[DarthHater]

@uliul-carpatin I'll see if I can push it up with a bit more progress soon. One of those things I don't want to do half baked.

@tomdavidson help test it once I/we get something up!

[ntwrkguru]

@DarthHater I'd like to test as well. Took me about a week to get this working as a built-in to Dockerized Nexus. It was then that I realized it only supported proxy. Guess I should have read the docs more thoroughly...stupid Y chromosome.

[blackdog0403]

@DarthHater This feature looks great. My team and I have been considering many ways to give authorization for helm charts repository but this looks better than others.
I would like to test too.

[DarthHater]

I'll try and give this a bit more focus this week y'all!

[DarthHater]

Made some progress on this today, gonna noodle a bit more and then see if I can get it out here on a branch for y'all. Just trying to figure out the best way to regenerate the index.yaml file each time a chart gets uploaded. Took some fun thinking as you can do it any number of ways, it's a question of which way will be the best for y'all to use, hopefully I took the right approach with some good input from @jlstephens89

[DarthHater]

Should have something for y'all to test next week. Thanks for keeping on me about this, I'll be stoked to see if it works to y'alls satisfaction!

[DarthHater]

Aight @blackdog0403 @ntwrkguru @uliul-carpatin @derkoe @tomdavidson I got something that will generate the index.yaml file. I need to do a few more tweaks and then I'll publish it for y'all to test. What I've done, just to give you an idea is:

• Allow for uploads of .tgz files (Chart packages) via a PUT
• Every 60 seconds (configurable) a service runs that will generate the index.yaml file which contains a list of charts, this is trigged by new artifacts being uploaded, and delayed by the configured amount of time

This means that your repo at current time would have everything under /, so no folders or anything akin. That can be done, but it'd take more lifting than I have time for at the moment. I have a few more things to iron out with generating a valid index.yaml file, but I can see the light at the end of the tunnel, so hang on :)

The design for all of this mimics quite a bit of what we do for Yum createrepo in Nexus Repo 3, and thanks to @jlstephens89 and @doddi for guiding me down the path!

[tomdavidson]

Nice!

A pipeline that releases to the repo will need to wait for the index.yaml before trying to deploy or have a quite small interval so I have a few questions to help me understand the implications:

1. What is the overhead impact of regen on a timer? With large catalogs do we risk the regen taking too long or hogging resources?
2. Is a race possible with Nexus webhook on a new artifact or can the webhook wait for the index.yaml too?
3. Nexus doesn't give an event for the new artifact?

[DarthHater]

@tomdavidson we chose this approach for Helm because we expected a lot of releases, versus with R we generate the metadata on each request (which can fall down at scale, but we didn't expect a LOT of releases on an R hosted repository).

1. Same as it would be doing generation on the fly, which is why we scurried it off into a service, also so that we can only regenerate when new assets/artifacts are added. We potentially risk that, yes, which would cause us to take deeper dives into it, where we append to an already existing file, or something else like that.
2. Webhooks would be sent out with each new artifact uploaded, and I believe a webhook will also be sent out when the index.yaml is created. I don't see what could cause a race condition here unless you were generating your own index.yaml externally? If it's because you'd like the index.yaml available as soon as a new artifact as deployed, well, that's magic sir :), or you are given the alternative of building it on each request, which can fall down at scale.
3. I'm using Events for the code I've written, we subscribe to these to know when to schedule rebuilding the metadata. The 60 second delay after receiving the event is configurable, I just have mine set to that. I realized I missed mentioning that in my last reply, updated it!

Hope that helps!

[tomdavidson]

oh yes, give me some magic!

Thank you for explaining how it works to me - this is prob obvious to many others. The race I was concerned about is the new artifact webhook triggering a deploy job and the job can't find the artifact because the index.yml is not created it.

[j-s-3]

The configurable delay was something we added for Yum to prevent multiple successive rebuilds when someone was uploading a LOT of RPMs. The helm implementation can get everything it needs from the database and doesn't need to crack open the charts so the default could / should be a lot lower than 60 seconds (say 1 second). We should also benchmark it with a lot of traffic as it might be unnecessary for Helm.

…

On Wed, Aug 1, 2018 at 10:20 PM Tom Davidson ***@***.***> wrote: oh yes, give me some magic! Thank you for explaining how it works to me - this is prob obvious to many others. The race I was concerned about is the new artifact webhook triggering a deploy job and the job can't find the artifact because the index.yml is not created it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANEL8xjxd3jKXiOp9l8-_2_Di9-IJyE7ks5uMlPZgaJpZM4TyhZv> .

-- Joseph Stephens Software Developer Sonatype, Inc. 8161 Maple Lawn Blvd., Suite 250 Fulton, MD 20759 jstephens@sonatype.com <mben-soula@sonatype.com> o: 3 <https://app.onsip.com/app/call?n=Sue%20Jasmin&a=sjasmin%40sonatype.onsip.com> 01.684.8080 x628

[DarthHater]

The artifact should be available as soon as it's published by the standard URL, so as long as something is calling that, it'll be fine. The index.yaml will be available with updated metadata in 60 seconds, or that can be tuned down. As @jlstephens89 said, we can likely change the configured time down, or just abandon it entirely if it's unecesssary (or better yet y'all can send an opinionated PR that does that :) ). I'd like y'all to test this once I get it out and let me know what seems to fit best. I anticipated quite a bit of publishing with helm, but maybe I was a bit overly enthusiastic. The format seems like one given CI/CD where you might publish a lot of charts, but I don't fully get when you'd make a change to a chart (could be infrequent), versus when you'd make a change to a Docker image (which seems to be very frequent).

[DarthHater]

Gotta get through a PR process, but today I got it to save what I think is a valid index.yaml file after uploading three charts:

apiVersion: '1.0'
entries:
  moodle:
  - created: {}
    description: Moodle is a learning platform designed to provide educators, administrators
      and learners with a single robust, secure and integrated system to create personalised
      learning environments
    digest: 543f7db011209a9be45a42e72299065c9a6e9358758bc546e6df38a9c5b1a7a0
    icon: https://bitnami.com/assets/stacks/moodle/img/moodle-stack-110x117.png
    maintainers:
    - email: containers@bitnami.com
      name: Bitnami
    name: moodle
    sources:
    - https://github.com/bitnami/bitnami-docker-moodle
    urls:
    - moodle-0.1.4.tgz
    version: 0.1.4
  mongodb:
  - created: {}
    description: NoSQL document-oriented database that stores JSON-like documents
      with dynamic schemas, simplifying the integration of data in content-driven
      applications.
    digest: 979d36b208be9b266c70860d4fe1f9e5130d9d60b3bcbd893132452648dfe27f
    icon: https://bitnami.com/assets/stacks/mongodb/img/mongodb-stack-220x234.png
    maintainers:
    - email: containers@bitnami.com
      name: Bitnami
    name: mongodb
    sources:
    - https://github.com/bitnami/bitnami-docker-mongodb
    urls:
    - mongodb-0.4.9.tgz
    version: 0.4.9
  - created: {}
    description: NoSQL document-oriented database that stores JSON-like documents
      with dynamic schemas, simplifying the integration of data in content-driven
      applications.
    digest: 76d2253cafd97a026d578c8696f369b37e3cb073c5e3e3b1368018eaa54d451f
    icon: https://bitnami.com/assets/stacks/mongodb/img/mongodb-stack-220x234.png
    maintainers:
    - email: containers@bitnami.com
      name: Bitnami
    name: mongodb
    sources:
    - https://github.com/bitnami/bitnami-docker-mongodb
    urls:
    - mongodb-4.0.4.tgz
    version: 4.0.4

Pretty stoked! Still need to get the created DateTime right (fun times with snakeyaml), but it's looking pretty good to me.

[DarthHater]

Okie dokie, I just pushed this out to master, I'll tag a release as well to make it obvious.

I have NOT added documentation related to this yet, because I would like some people to test it first to let me know what they run into before I make it explicitly known it's out.

You'll want to setup a helm-hosted repo, and then upload charts to it.

I've been using curl to test thus far, an example of such is:

curl -u admin:admin123 http://localhost:8081/repository/helm-internal/ --upload-file mongodb-0.4.9.tgz -v

Once you upload charts, they should get processed by Nexus Repo, and in 1 second a new index.yaml should get created at the root. Adding this repo should be very similar to how you would add a proxy repo to helm, take a gander at that!

Please let me know what issues you run into, create new ones here, etc... and I'll dive in to them. Thanks for the patience!

CC @blackdog0403 @ntwrkguru @uliul-carpatin @derkoe @tomdavidson @JPWKU

[DarthHater]

Stuff I want to know about:

• Does the index.yaml generate correctly, as you'd expect, work with your version of Helm, etc...
• Does it generate in a timely manner?
• Do the charts get stored as you'd expect?

Etc....

I am a helm user but pretty light overall, so I'd love to know more about your use, as I am sure it will eclipse mine!

[ntwrkguru]

Testing now. Re: Your Dockerfile; I use a multi-stage build. I'm sure it can be improved, but it seems to work. The key parts are the initial FROM alpine:3.6 where I use alpine to clone the repository and run the maven clean package

Below that, in the main image build steps, I COPY --from=0 to copy the jar. This way the all of the build residue isn't in the main container.

Something like this:

FROM alpine:3.6
RUN apk add --no-cache maven git
WORKDIR /build/
RUN git clone https://github.com/sonatype-nexus-community/nexus-repository-helm.git \
    && cd nexus-repository-helm \
    && mvn clean package



FROM	centos:centos7
LABEL MAINTAINER=goaway@nospam.com

ENV NEXUS_VERSION=3.12.1-01
ARG NEXUS_DOWNLOAD_URL=https://download.sonatype.com/nexus/3/nexus-${NEXUS_VERSION}-unix.tar.gz
ARG NEXUS_DOWNLOAD_SHA256_HASH=8f3137471cdb002b6cf9d1874bbddb6ad280d4d7a492e363cfc5f54fbd45b5d9

RUN yum install -y \
  curl tar java-1.8.0-openjdk \
  && yum clean all


# configure nexus runtime
ENV SONATYPE_DIR=/opt/sonatype
ENV NEXUS_HOME=${SONATYPE_DIR}/nexus \
  NEXUS_DATA=/nexus-data \
  NEXUS_CONTEXT='' \
  SONATYPE_WORK=${SONATYPE_DIR}/sonatype-work \
  DOCKER_TYPE='docker'

# install nexus
RUN mkdir -p ${NEXUS_HOME} \
  && curl --fail --silent --location --retry 3 \
    ${NEXUS_DOWNLOAD_URL} \
  | gunzip \
  | tar x -C ${NEXUS_HOME} --strip-components=1 nexus-${NEXUS_VERSION} \
  && chown -R root:root ${NEXUS_HOME}

COPY --from=0 \
      /build/nexus-repository-helm/target/nexus-repository-helm-0.0.1.jar \
  ${NEXUS_HOME}/system/org/sonatype/nexus/plugins/nexus-repository-helm/0.0.1/nexus-repository-helm-0.0.1.jar

# configure nexus
RUN sed -e '/^nexus-context/ s:$:${NEXUS_CONTEXT}:' -i ${NEXUS_HOME}/etc/nexus-default.properties

RUN sed -e '/<feature prerequisite/i\ \ \ \ \ \ \ \ <feature prerequisite="false" dependency="false">nexus-repository-helm<\/feature>' -e '/<\/features>/i \ \ \ \ <feature name="nexus-repository-helm" description="org.sonatype.nexus.plugins:nexus-repository-helm" version="0.0.1">\n        <details>org.sonatype.nexus.plugins:nexus-repository-helm<\/details>\n        <bundle>mvn:org.sonatype.nexus.plugins/nexus-repository-helm/0.0.1<\/bundle>\n    <\/feature>' -i \
    ${NEXUS_HOME}/system/org/sonatype/nexus/assemblies/nexus-core-feature/${NEXUS_VERSION}/nexus-core-feature-${NEXUS_VERSION}-features.xml

RUN useradd -r -u 200 -m -c "nexus role account" -d ${NEXUS_DATA} -s /bin/false nexus \
  && mkdir -p ${NEXUS_DATA}/etc ${NEXUS_DATA}/log ${NEXUS_DATA}/tmp ${SONATYPE_WORK} \
  && ln -s ${NEXUS_DATA} ${SONATYPE_WORK}/nexus3 \
  && chown -R nexus:nexus ${NEXUS_DATA}

EXPOSE 8081

WORKDIR ${NEXUS_HOME}

ENV INSTALL4J_JAVA_HOME="/usr/bin/java"
ENV INSTALL4J_ADD_VM_PARAMS="-Xms1200m -Xmx1200m -XX:MaxDirectMemorySize=2g -Djava.util.prefs.userRoot=${NEXUS_DATA}/javaprefs"

USER nexus

CMD ["bin/nexus", "run"]

[DarthHater]

Neato, thanks for sharing! Feel free to create a new issue if you think we can improve the one we have out there, or send a PR!

[ntwrkguru]

Yours is cleaner, for sure, so maybe I'll submit a PR to add a build stage to the one you have. :-)

[ntwrkguru]

DOH!!! Ignore me...I'm completely out of it today...that Dockerfile totally uses a build stage.

[DarthHater]

Hah yeah I was wondering! No worries, my head is a bit in the clouds today :)

[DarthHater]

These screenshots are from my testing:

This was after uploading three different charts, and the resulting index.yaml file. This was tested against Nexus Repo 3.14.0-SNAPSHOT (but it's configured to work against Nexus Repo 3.13.0, as an FYI).

[DarthHater]

FTR I turned the 60 second rebuild time down to 1 second, and it's still configurable, if you want to play with different intervals.

[ntwrkguru]

For some reason, it builds fine on my Mac, but not on my Ubuntu 14.04 server. Pardon the garbled junk...it's Ansible's output:

[INFO] Total time: 0', u'4:02 min\\n', u'[INFO] Finished at: 2018-08-21T20:34:34Z\\n[INFO] ------------------------------------------------', u'------------------------\\n', u'[ERROR] Failed t', u'o execute goal o', u'rg.apa',u'che.mav', u'en.plu', u'g', u'i', u'ns:maven-surefire-plugin:2.20:test (default-test) on project nexus-repository-helm: T', u'here are test failures.\\n', u'[ERROR] \\n[ERROR] Plea', u'se refer to /nexus-repository-helm/target/surefire-reports for the individual test results.\\n[ERROR]', u' Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.\\n[ERROR] -> [Help 1]\\n', u'[ERROR] \\n[ERROR] To', u' see the full stack trace of the errors, re-run Maven with the -e switch.\\n[ERROR] Re-run Maven using the -X s', u'witch to enable full deb', u'ug logging.\\n[ERROR] \\n[ERRO', u'R] For more information about the errors and possible solutions, please read the following articles:\\n[ERROR] [Hel', u'p ', u'1', u'] http://cwiki.apache.org/confluence/dis', u'pl', u'ay/MAVEN/MojoFail', u'ureEx', u'ception\\n', u'Removing intermediate container a187fe480e46\\n']"}

[j-s-3]

@ntwrkguru Does your stack trace include which test fails?

[ntwrkguru]

Got it to work with some "magic". It's probably not your code, rather my implementation. I can indeed create a hosted helm repo, but am I correct in assuming that this only accepts uploads via curl at the moment? I attempted to add the repo using helm repo add ... but was unsuccessful. Just wanted to see what the current limitations were so that I could verify that I was doing things correctly.

[DarthHater]

@ntwrkguru , you likely won't be able to add a repo until after you've uploaded artifacts to it, as the index.yaml won't exist until after something is there. I believe helm first checks for that file when you run helm repo add

You can upload to the repo via anything that does HTTP PUT, so curl, wget, your own code, etc...

[dtshepherd]

@DarthHater does the hosted helm repo still support proxying to an external repo?

[ghost]

@DarthHater
Looks like fixed. Added missed fields in Chart.yaml (like icon, sources and maintainers)

[dtshepherd]

@brezblock See my PR #5

[jonathansloman]

Hi, keen to use this, but can't get it to work. I'm running nexus 3.13.0-01. I've successfully published a helm chart to the repo, and I can see that it has created an index.yaml file - it shows up in the browse view, and lists a file size and blob reference etc. But when I try to actually download it, I get a 404 (eg from http://my-nexus-hostname:8081/nexus/repository/my-helm/index.yaml which is the url listed 'Path' in the summary view when browsing the repo).

I've tried regenerating the index.yaml but still get the same behaviour.

I can see the contents of the file in the blob store on the disk, and it looks correct.
I don't think it's a permissions issue - I can retrieve the actual chart from the same repository via http://my-nexus-hostname:8081/nexus/repository/my-helm/my-chart-0.1.0.tgz and that works fine.

Any idea what's going on here? Have I done something stupid or does something not quite work right?
I installed the plugin using the 'most permanent' method.

[dtshepherd]

@jonathansloman I noticed this bug as well. I’ve been meaning to open an issue. Restart nexus and it should see the index.yaml. You only need to do this once for the initial index create.

[DarthHater]

Hey y'all! Sorry, I was out on vacation. Appreciate the feedback, I'll see what I can get merged, etc... today!

[jonathansloman]

@jonathansloman I noticed this bug as well. I’ve been meaning to open an issue. Restart nexus and it should see the index.yaml. You only need to do this once for the initial index create.

Thanks, can confirm this fixes it. One the one hand, glad it wasn't something stupid I did, on the other hand, should have thought of restarting to try to fix it myself..

[DarthHater]

That seems super odd that you two needed to restart (I've never experienced this). I'll see if I can spin things up and recreate it!

[dtshepherd]

Yeah our dev and prod nexus server exhibited the same issue. You have to push a single chart, then restart nexus the 2nd time. After that, its good for new indexes.

[DarthHater]

SUPER weird. Lemme play with it. I left out some functionality that would allow you to generate the index.yaml manually (rather than automagically like now), maybe I can add that in for trouble shooting, but I'd love to know why it's not working right out of the gate for y'all.

[dtshepherd]

Maybe the nexus frontend is caching something about the store and the restart clears the cache making the index.yaml available?

[g0blin79]

Hey guys, need some informations because as I correctly intend, the "hosted" repo was a really recent improvement. I created an "hosted helm repo" on my Nexus instance. Now in the list of repositories I have helm repo too, with the relative URL (ex: https://nexus.mydomain.com/repository/charts/).

I uploaded a chart package in the way in which @DarthHater did here #2 (comment) (I don't know if there is another way to do that).

Now I tried to add this new repo to my helm client then I did:

helm repo add --username my-nexus-user --password my-nexus-pass nexusrepo https://nexus.mydomain.com/repository/charts/

but I receive:

Error: Looks like "https://nexus.mydomain.com/repository/charts/" is not a valid chart repository or cannot be reached: Failed to fetch https://nexus.mydomain.com/repository/charts/index.yaml : 502 Bad Gateway

I don't know if it is the best place to ask this, but I saw that this is the issue about "hosted helm" feature and wanted to avoid duplication. In case tell me where repost.

Thank you for your support

[jonathansloman]

hi @g0blin79 - read my comment above, it's the same issue. The index.yaml file, though it exists, is not visible after you add your first chart. Try restarting your nexus, you should then find the index.yaml is visible and you can do the helm repo add.

[g0blin79]

@jonathansloman yes, sure: thank you. Now it works.

[DarthHater]

Ok y'all, after some talking with @jlstephens89 I think we figured out why the index.yaml might not "show up" until after a restart, and that was because I was adding a NegativeCacheHandler to Hosted repos. I went ahead and got rid of that, and tagged a new release: https://github.com/sonatype-nexus-community/nexus-repository-helm/releases/tag/v0.0.4

Give that a whirl and let me know your experience! What I think was happening was that a request for the index.yaml was cached as 404, and that cache wasn't busted until you restarted Nexus Repo. Removing the negative cache handler SHOULD alleviate that, I believe.

[DarthHater]

I'm gonna play a bit more with the exact commands y'all are running too, got some free time today!

[g0blin79]

wow now it works like a charm.

But (don't hate me...)...

It's not clear for me if there is a way more "standard" to upload a chart via command-line.
I tried with "curl" command and it works, but it is not so simple to use (and to put in a CI system for example).
Could be really amazing if we can use the "helm push" command born for chartmuseum. I tried with that but probably it does not work, because I have an "Error 405: Method not allowed".

Anyway, any other solution could be really appreciated. Meantime a "curl" could be enough 😄

[dtshepherd]

@DarthHater Ok I just tested out 0.0.5 and things seem to be working.

[dtshepherd]

@g0blin79 I haven't been able to get this in its own repo, but I created a simple helm plugin to helm push charts: https://gist.github.com/dtshepherd/9cd8e2b649775e548a35cb57ca02e62a

[DarthHater]

That's neat @dtshepherd ! If you rewrote it in Golang (maybe I can help you) maybe we can get it accepted to helm itself?

[dtshepherd]

There's already a stale ticket here: helm/helm#3564

Also, the push interface may be too specific for helm itself. The plugin should probably be called push-nexus or something like that. Seems like the plugin approach is probably the most appropriate. App Registry actually did the same thing: https://github.com/app-registry/appr-helm-plugin

The github plugin is written in bash: https://github.com/sagansystems/helm-github/blob/master/github.sh. I kind of like using bash because it doesn't require more binary packaging, but I can understand wanting it to be in Go.

Not sure what repo to host this code in yet, but I do find it useful for integration with your nexus helm plugin.

[DarthHater]

Yeah I love it! If you write it in Golang you'll probably get more helm people to pitch in on it at some point. I'd gladly help you on that (I love writing Go).

If you want, we can create a repo here on sonatype-nexus-community, something like helm-nexus-push or something akin if you'd like and I'll help you get it setup and everything?

[dtshepherd]

Yeah, I'd be on board with that. I can write Go as well, I don't think it will be much effort to port it.

[DarthHater]

@dtshepherd send me an email at jhesse at sonatype dot com and I'll help you get some stuff put together for it, already created the repo/added you as an org member, etc...

[wu105]

Has any one uploaded a .tgz.prov file along with a .tgz file?
This is for signed charts.
The Helm document suggests that the helm repo only need to provide this .prov file when requested for the chart signing to work, thus hope this plugin would support chart signing without much complications. Myself would be able to test in a few days.

[DarthHater]

@wu105 see #19 , I've got a branch kinda cooking on it, but there are some fun things to be done to get it working

[drbeka]

hi,
have tested version 0.0.6 and it works for standard tgz file.
Is it possible to build a tree structure inside the helm repository?
Have tried with
curl https://mynexus/repository/myhelm/sub1/sub2 --upload-file mychart-0.1.0.tgz
but no effect, only found 404 in request.log

Is it possible to have a folder structure?

[marcoreni]

hi,
have tested version 0.0.6 and it works for standard tgz file.
Is it possible to build a tree structure inside the helm repository?
Have tried with
curl https://mynexus/repository/myhelm/sub1/sub2 --upload-file mychart-0.1.0.tgz
but no effect, only found 404 in request.log

Is it possible to have a folder structure?

Hi all,

has anything been done regarding the above post?

Chartmuseum supports nested folders by defining a "depth" parameter. Can something like that be added here or do we have to create multiple repositories?

[vosmax]

I’ve installed version 0.0.13 on Nexus 3.19 but have an issue with ‘helm push’ - server returns 500. Was it fixed or not?

[digitalray]

@vosmax no need to use helm push, you just need to upload the package to the repo somehow. I use curl:
curl -v -F file=@somehelm-0.0.1.tgz -u username:password https://nexusendpointaddress.com/service/rest/v1/components?repository=helmrepo-name
Once the package is uploaded, nexus will regenerate the index.yaml file