Flexible and contextualized image selection process

[mhausenblas]

Currently, images used for launching distributed processes are determined by kubed-sh-internal environment variables in a static and global manner. Having a way to overwrite/set the image per individual process would enable a number of use cases (scripting, etc.) and make kubed-sh more flexible.

This issue came out of a discussion with @jbeda, thanks a lot for this valuable input.

[mhausenblas]

OK, I think I've got a basic design proposal together for this: in a nutshell, extend the environment concept to provide 'isolated' execution spaces. I don't want to call it namespace as we have already Linux namespaces and K8s namespaces and that would add to the confusion. So the interaction would look as follows:

1. One starts out in the global env.
2. Create a new env and switch to it automatically with env new $name:

[context]$ env new foo
[foo@context]$ env
BINARY_IMAGE=alpine:3.7
...

3. The new env foo inherits the existing env vars from the global env which can be overwritten.
4. Switch between envs using env select $name (assuming here that test.example.com already exists):

[foo@context]$ env select test.example.com
[test.example.com@context]

5. Iff the env name is a FQDN then it will be used for exposing long-running distributed processes cluster-externally via an Ingress controller and respective resources, for example:

[test.example.com@context] node tc/myapp.js &
[test.example.com@context] curl test.example.com/myapp
Hello from Node.js!

WDYT @jbeda?

[jbeda]

Haven't had a chance to play quite with it in real life quite yet :)

The idea of "environment" is an interesting one and is something that we've been introducing into ksonnet.

I do worry that there are too many 1:1 pairing of concepts. That is a pattern that can often lead to some great easy usability wins but cause problems later. For example, if env ties to an external DNS name, what happens if you have more than one long running process in an env? What about microservice type things? Do you expect that they'd be in different envs?

I'd think about how env ties to namespace. Also, is there a config file for describing the default envs? Is the server you are talking about part of the env? Does modifying an env temporarily modify the current session or does it write out to the imaginary config file?

The big piece that is missing for me still is how you specify all the other important bits of a pod beyond the image and the program you want to run? What about volumes, limits, configmaps, etc?

[mhausenblas]

Thanks for your feedback, Joe, I appreciate your time! I think you're raising excellent points here. Let me try responding to some points right away and others I need to mull over.

I do worry that there are too many 1:1 pairing of concepts. That is a pattern that can often lead to some great easy usability wins but cause problems later.

Fair point. I'm willing to experiment here a bit but the result should be sustainable, agreed.

For example, if env ties to an external DNS name, what happens if you have more than one long running process in an env?

Hmm, that one I thought I've addressed with the design above? If you're in the env test.example.com and launch a binary or script with the name bar then the resulting URL would be test.example.com/bar, allowing for multiple long-running processes.

I'd think about how env ties to namespace.

Yup, I did think about it but just realized I failed to actually spell it out: namespaces are more granular, that is, a namespace can have many environments (technical an environment would just be a bunch of labels on resources). The reason for this is twofold: 1. allows for efficient sharing of resources and admin policies while being simple for com patterns, and 2. there can be platforms where you either can't create namespaces and/or default is not available (yes, I'm talking about OpenShift Online, for example ;)

The big piece that is missing for me still is how you specify all the other important bits of a pod beyond the image and the program you want to run? What about volumes, limits, configmaps, etc?

This is the hard part I don't have a good suggestion ATM but some ideas I'd like to explore. Please hold the line …

[mhausenblas]

Now, kubed-sh supports create, select, delete, and list for envs. The resources created (deployments and services) are labelled with the environment. When doing a ps, only dprocs in the current, active env are listed, and a ps all shows all envs in the current context.