How to create a single stack that provides resources to N clusters #7382
-
|
I've hit a bit of a wall with regards to how I'm setting up my infrastructure. As a short background, I'm creating pretty heavy SaaS instances wherein each customer gets its dedicated environment, which could span multiple clusters around the world (and all clusters are dedicated to that customer). As such, I've created two primary pulumi programs: The important part is that with the current setup, each I've chosen to differentiate the two primarily because kubernetes workloads are updated more frequently and dynamically than the underlying infrastructure. Both stacks are designed similarly, though the Essentially, each index.ts refers to one or more (4-8 or so) ComponentResources which are each responsible for creating a type of component and all their dependencies. For The main issue I am facing right now is that my To start with - I had an original Initially, the conflict was due to my naïve naming of pulumi k8s resources: const influxHelm = new k8s.helm.v3.Chart(`influx`,{
chart: "influxdb",
fetchOpts: {
repo: args.helmRepos.influxdata.url
},
values: influxValues
},{
provider: args.providers.kubernetes,
parent: this
}) This obviously conflicted when the ComponentResource ("Database") which contained this was looped on the kubernetes providers.. for (region in infra.regions) {
const dbs = new Database(``${region}-Database`, {
helmRepos: ...
providers: {
kubernetes: infra.regions[region].kube.provider
}
})
} So I attempted to ensure that the name of the influxdb helm chart was region-aware, and setting the alias to the old name to prevent re-creation: const influxHelm = new k8s.helm.v3.Chart(`${args.environment.region}-influx`,{
chart: "influxdb",
fetchOpts: {
repo: args.helmRepos.influxdata.url
},
values: influxValues
},{
provider: args.providers.kubernetes,
parent: this,
aliases: [{name: "influx"}]
})
This also did not work. I believed it might be because both the regions would be pointing towards the same alias, so I picked another of my My desperation now was in that I really needed to make sure I didn't destroy any of the resources created in the single-region instances while ensuring that I make things multi-region capable. With that in mind, I've created a gist that summarizes the way I try to set up my baseline-k8s - and I need to know if this is ever possible to make work with pulumi. Essentially if this makes sense - then I can continue on with my fixing - but if it doesn't - I'd love to know:
If you've read this far - thanks for your time - and if you have a bit extra - please have a look at this gist :) https://gist.github.com/headconnect/dde82d304b508bfb8f8c15faab3dbe16 |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
A bit of an update: Found one thing that definetely creates a conflict.. so using const keyCloakCRDInjection = new k8s.yaml.ConfigGroup(`${args.environment.region}-keycloak-crd-install`, {
files: `${pathToKeyCloakOper}/crds/*.yaml`
},{
provider: authK8sProvider,
parent: this,
})Results in: And this only occurs when running it multi-region. As you can see, the URN contains no uniquenes.. Workaround for this is moderately easy though tedious and messy - simply create Second part - seems helm really is a culprit here, even if I set the pulumi name safely, if the "fullNameOverride" is set simply (as in this case, it was set to "redis"), then there is still no uniqueness because even though the different resources are actually created in different clusters - it's all the same to pulumi: const redis = new k8s.helm.v3.Chart(`${args.environment.region}-redis`,{
chart: "redis",
version: "13.0.1",
fetchOpts: {
repo: args.helmRepos.bitnami.url
},
values: redisValues
},{
provider: args.providers.kubernetes,
parent: this
})Causing: |
Beta Was this translation helpful? Give feedback.
-
|
Yeah no, I spoke too soon about the ConfigFile stuff: const keyCloakOperator = new k8s.yaml.ConfigFile(`${args.environment.region}-keycloak-operator`, {
file: `${pathToKeyCloakOper}/operator.yaml`
},{
parent: this,
provider: authK8sProvider
})Begets: |
Beta Was this translation helpful? Give feedback.
-
|
Ok - here's a workaround that seems to actually.. you know.. seem to work!! I warn you that this is a bit ugly though, but at least it works well on both helm charts as well as yaml.Config* stuff. Firstly you need to set some sort of string which is unique per cluster (and wouldn't normally come up in any of your kubernetes yamls whatsoever). For me, that's the name of the cluster. Also, it needs to be able to form a valid url (i.e comply with RFC 1034/1035) - hence my replacemed of _ with - below. const keyCloakOperator = new k8s.yaml.ConfigFile(`${args.environment.region}-keycloak-operator`, {
file: `${pathToKeyCloakOper}/operator.yaml`,
resourcePrefix: `${args.environment.kubename.replace("_","-")}`,
transformations: [
(obj: any, opts: pulumi.CustomResourceOptions) => {
obj = untransformer(obj, `${args.environment.kubename.replace("_","-")}-`)
}
]
},{
parent: this,
provider: authK8sProvider
})The important bit is the resourcePrefix (adds uniqueness) - but in combination with the transformation (removes uniqueness before actual deployment). This also works fine for Also, /**
* The untransformer
*
* This function will take an object as its input, and deeply
* traverse its structure while checking all values.
*
* All values of type string will have the `removePrefix` removed
*/
export const untransformer = (obj: any, removePrefix : string) : any => {
let keys = Object.keys(obj);
keys.forEach((key) => {
let value = obj[key];
if (typeof value === 'string') {
obj[key] = obj[key].replace(removePrefix, "")
} else if (typeof value === 'object') {
untransformer(obj[key], removePrefix)
}
})
}Did I say it was a bit ugly? This is incredibly ugly. Then again - the final result seems to work, so.. EDIT: Added trailing "-" to the replacestring as the prefix adds it by default ;) |
Beta Was this translation helpful? Give feedback.
Ok - here's a workaround that seems to actually.. you know.. seem to work!!
I warn you that this is a bit ugly though, but at least it works well on both helm charts as well as yaml.Config* stuff. Firstly you need to set some sort of string which is unique per cluster (and wouldn't normally come up in any of your kubernetes yamls whatsoever). For me, that's the name of the cluster. Also, it needs to be able to form a valid url (i.e comply with RFC 1034/1035) - hence my replacemed of _ with - below.