WIP: add step to produce KIND images for EKS-D #125
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: chanwit The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Hi @chanwit. Thanks for your PR. I'm waiting for a aws member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
| @@ -0,0 +1,8 @@ | |||
| FROM kindest/node:v1.18.8 | |||
There was a problem hiding this comment.
Can you be sure to build this image from source?
There was a problem hiding this comment.
Hi Micah,
I'm still looking for the best approach that we could tackle this problem together.
This is a point I'd like to discuss too.
Would you like it to be built directly from the KIND base image?
or
Would you like it to go from a ready-to-use KIND node image, and just simply replace the binaries from EKS-D?
My POC was working great using the latter case, but I'm very open to any approach.
There was a problem hiding this comment.
Build a KIND node image from source means we'll need to re-compile binaries again from the whole Kubernetes source tree.
But I'd like to use the compiled binaries from the previous steps to make sure that we actually ship exactly the same binaries.
There was a problem hiding this comment.
Also according to the underlying implementation of the KIND build command,
https://github.com/kubernetes-sigs/kind/blob/master/pkg/build/nodeimage/build_impl.go#L90
- copy required binaries to the container
- chmod +x them
- create the version file
- and just commit to create the new image.
The Dockerfile in this PR does just that.
There was a problem hiding this comment.
For EKS Distro, we don't distribute images or binaries we have not built from source.
Would you like it to be built directly from the KIND base image?
or
Would you like it to go from a ready-to-use KIND node image, and just simply replace the binaries from EKS-D?
We'd want to build the kind binary and image from source, so neither of these options quite work. Looking at the Kind docs on the node image, I think we'd need to build the base image, and then invoke kind build node-image referencing our base image.
Looking at the kind source, we'll probably (independently from this change) want to have kind build node-image --type support the a new option (Maybe some YAML config file?) that satisfies the Builder and Bits interfaces. EKS-D builds the binary images and image tars, so there is no need for kind build nodeimage itself to need to build K8s from source.
There was a problem hiding this comment.
That's clarified a lot! Thank you @mhausenblas.
I'd like to re-summarize these into actionable items:
- Patch
kind buildto support a new option to put binaries (and tars) into the kind node image. (Binaries are OK, but I'm not sure we need to put image tars there as we can let user specify custom images via a KIND config - also see here where I embed a similar config to make a CLI work to bootstrap an EKS-D cluster). - Build
kindbinary from source - which means we'll have "projects/kind" in the source tree. - Use the
kindbinary to build the base image. - Use the base image to build the node image
There was a problem hiding this comment.
Right!
- Yes! I don't think it would be a heavy lift to have a YAML config-based
Builder()that could be specified from the command line. I think there would be advantage in adding the OCI tars we provide, as kind will side-load the images and use those by default - Lets go with our convention of
/projects/<github org>/<repo>, soprojects/kubernetes-sigs/kind/ - The current kind base image builds from Ubuntu and our preference would be to go with Amazon Linux 2. It doesn't look like it would be a heavy lift to move to an AL2 image and install containerd/runc from
amazon-linux-extras.crictlmight be the only thing in that base image that neither Amazon Linux nor EKS-distro (yet) builds today. - Correct, although we'll have to figure out a solution for this too as EKS-Distro build jobs don't have Docker available and instead use buildkit for container images.
There was a problem hiding this comment.
I created #130 to track this
|
I close this PR and will create a new one after we have a good solution to implement KinD-based images. |
Issue #, if available:
Description of changes:
This PR adds a sub-project to produce KIND-based images for EKS-D, by
[ ] copying binaries from the previous build steps (kubernetes/_output) into the sub-project directory
[ ] changing the version of KIND image to match with the version-build of EKS-D.
[ ] build the image, and push to the target repository
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.