RFC: Git Repo Artifact Type

[ethanfrogers]

As we work to make Spinnaker easier to use, we have encountered a couple of cases where the current artifact abstractions can't adequately represent the data we need to do certain things. For example, I was recently involved in the effort to bring Kustomize to Rosco as a Bake (Manifest) rendering engine. Kustomize works by running a command-line tool against a set of files on disk, usually checked in as part of a repository, to produce a fully rendered Kubernetes manifest. In most cases this is fine because Kustomize can be run as part of a CI process. However, in order to make the experience as "native" as possible, we had to get creative in how we collected all of the dependencies required to render the manifest. While this approach works, it is challenging to write, maintain, and explain

Along the same lines, the current Helm Chart rendering support requires that users package their Helm Charts as .tar.gz archives before being able to run them through the Bake (Manifest) stage. We've had users request support for eliminating this step and sourcing their Helm templates directly from source (Helm support rendering charts from source in a directory as well as a tarball).

In both cases, we've found that having the ability to reference an entire collection of source files not only makes sense in the user's mind but also makes supporting these types of features easier for us, the developers.

My simple proposal is a generic git/repo artifact type which would represent a collection of files stored in a Git repository. When used, this artifact would clone a Git repo, compress it's contents into a .tar.gz and transmit the archive back to the caller. The caller would then handle decompressing the archive into a location where it can operate on it as necessary.

As an example, if this artifact type existed today, I would have done the following for Kustomize:

1. User supplies 2 bits of information - the git/repo artifact which represents their source code & a path to their kustomization.yml within that repo.
2. When the request hits Rosco, the KustomizeBakeManifestService would fetch the artifact, decompress and extract it to a temporary directory.
3. kustomize build would be run against the temporary directory.
4. Once the manifest has been rendered, the temporary directory would be cleaned up.

I've built an example implementation here using JGit.

[ethanfrogers]

cc @ezimanyi

[dbyron0]

+ such a big number to this. The extra step of publishing versioned tarballs for helm charts adds a ton of friction to our workflow.

[ezimanyi]

This looks reasonable to me; looks from the sample implementation that it's using bare git and would thus support any hosted git service (vs. our single-file artifacts which are separate for github, bitbucket, etc.)?

[ethanfrogers]

Correct, which is a big plus. We won't have to support multiple implementations. Based on the reference implementation - what are your thoughts on cloning the repo to disk and then reading it into memory before sending it off? I wasn't a huge fan of this but wasn't sure of a better way ti implement that. Alternatively, most hosted services support the option to fetch repositories as tarballs via their API which we could fetch and stream back, skipping the disk write all together.

[ezimanyi]

Yeah, storing the tarball to disk is not great...though maybe that's better if it means being compatible with anything using git protocol.

[ethanfrogers]

Well, the consumer of the artifact will have to write it to disk, at least in the case of Rosco where we run those type of CLI tools (Kustomize, Helm, etc). The only reason I'm writing the initial clone to disk is because I don't think JGit has any option to virtualize the filesystem for that operation. If it does, then I'm all for cutting out that step.

Would it be helpful if I wrote up a small design doc?

[ezimanyi]

Yeah, I'd think writing up a design doc would make sense. Seems like a reasonable amount of work so it's probably something that could get implemented in the reasonably near future.

[ethanfrogers]

@ezimanyi Here's a brief design doc - https://docs.google.com/document/d/10rPfhbaYuLpPpIfGysEq21Jp0l8i1vNwSHoiN6nvwvI/edit?usp=sharing

I basically just consolidated the above into a formal doc. The implementation seems pretty straightforward to me but I tried to address some of the tradeoffs and decisions we need to decide on. LMK if you have any questions or see areas that need more clarification.

[ethanfrogers]

@ezimanyi I've added some comments on the design doc, mostly just responding to points that you made. Most of the comments/concerns are around size which I think we can alleviate with things like sparse and subpath checkout. If we're good with this being the initial design I'll prioritize it and start working on it soon.

[ezimanyi]

I'm happy with the state of the design document; 👍. Excited to see this move forward!

[ethanfrogers]

Adding the accepted label as this is landing in 1.17.