Programmatically set the default providers

[lukehoban]

From @geekflyer on Slack:

Hi, is there a way to programmatically set a global default provider or (programmatically set a root resource which contains this provider)??
The concrete issue is the following: Currently in my stacks I’m programmatically creating a k8s provider that should be used as the deployment target for all resources in that stack. In order to use this dynamically created provider, however, I have pass it explicitly to each k8s resource that is created in that stack OR I have to explicitly set a common parent on each resource in that stack and set the k8s provider on that parent. If I accidentally forget to explicitly set the provider or parent on one of the resources, pulumi seems to create an implicit k8s provider based on my local kubectl config / context. Depending on what my current local kubectl context contains, pulumi may end up silently deploying resources to a different cluster than what I actually want. It would be very helpful if I could disable or override this “implicit default provider” programmatically in the entrypoint of my program. (edited)

[lukehoban]

Currently there is not a mechanism to set the default. The closest available option today is to put all resources inside a component (and pass parent: this) - though as you rightly note this can be error prone.

We definitely want to improve this - I can imagine a couple ways we might be able to do it;

• expose the stack object (the implicit parent of unparented resources), and allow changing its default providers
• allow overriding the kuberenetes packages default provider directly via kubernetes.setDefaultProvider or similar.

These would both require being careful about timing, as they modify glove state and subtle changes to module loading order could potentially cause this switch to happen at a different time.

[hausdorff]

Just to capture this what else is going on in the discussion, and why I think it's a big deal: this is something that bites pretty much 100% of consequential Kubernetes users at one point or another. And when you do, can be very confusing. For example, the other day, Joe was asking why his Service wasn't getting allocated a public IP, and because I recognized the .status field as being unique to Docker for Mac, I was able to discern immediately that he hadn't set providers correctly. If this issue confuses Joe, it will confuse our users, too. And imagine what would have happened if Joe's kubeconfig active context had been pointed at prod?

[lukehoban]

Another option here might be to allow some way to get an "instance" of the kubernetes module which used a different default provider. This might even be something where you could do:

import * as kubernetes from "@pulumi/kubernetes";
let aksKubernetes = new kubernetes.Provider("myprovider", {...});
let deployment = new aksKubernetes.apps.v1.Deployment("...", { ... });

[clstokes]

In my case, I want to explicitly define and configure a provider based on outputs from a StackReference. If I do this, I still get errors about missing configuration. I can provide dummy config values to the ambient provider to get past the error: Missing required configuration variable 'aws:region' errors, but this is annoying and makes for confusing configuration.

Ideally, I'd be able to set the default/ambient provider with explicitly provided arguments and have all related resources use it - i.e. Luke's second option at #2059 (comment).

[pgavlin]

If I do this, I still get errors about missing configuration. I can provide dummy config values to the ambient provider to get past the error: Missing required configuration variable 'aws:region' errors, but this is annoying and makes for confusing configuration.

This doesn't actually come from the default provider: this comes from the corresponding Node/Python/etc. package performing a config.require that corresponds to a provider configuration key.

[hausdorff]

@pgavlin @lukehoban Just so we have inputs to make sensible prioritization decisions, one of our more advanced users ran into this problem and spend days figuring it out:

I narrowed this issue to a couple of things - i had a CRD where i hadn't specified the provider so it was using the one from my KUBECONFIG env var... despite that not being a cluster i was using with pulumi.

I know this is marked for M21 -- but, we've bumped this since M19, and my personal opinion is that this is a very strong candidate to not get bumped this time. :) I don't want to beat a dead horse, so let me know if this is getting annoying, but: every significant user of a managed Kubernetes platform will get bitten by this issue eventually. And when they do, it will usually really hurt.

[lukehoban]

this is marked for M21

Yep - this is important.

[almson]

I would argue that a best-practice is to declare all resources in a ComponentResource, which already has this feature.

[raphaelauv]

@almson is there any convention for the naming of type t of a ComponentResource ? I did not found any documentation

gcp_provider = gcp.Provider(resource_name="default_5_23_0",
                            opts=pulumi.ResourceOptions(version="5.23.0"),
                            project="XXX-XXXX")

all_resource = pulumi.ComponentResource(t="gcp:component:all_resource" ,name="all_resource", opts=pulumi.ResourceOptions(provider=gcp_provider))

airflow_XXXX = gcp.composer.Environment(
    ...
    opts=pulumi.ResourceOptions(parent=all_resource ),
    ....

[almson]

@raphaelauv ComponentResource is an abstract base class. You have to extend it and then set the parent of child resources. See here https://pulumi.awsworkshop.io/45_componens/10_create_component.html Edit: this example doesn't set default providers, but you can do that in the ComponentResourceOptions that are passed to super.

[iwahbe]

We have merged #3383. This doesn't provider a way to programmatically set default providers, but it does prevent accidentally referencing the default providers.

[pyoio]

We have merged #3383. This doesn't provider a way to programmatically set default providers, but it does prevent accidentally referencing the default providers.

This seems to disable inheritance of providers for resources inside a ComponentResource class with { parent: this }.

Being able to disable the default provider and explicitly specify a "default" provider to be inherited by child resources would be excellent

[iwahbe]

@pyoio If I'm understanding correctly, #3383 should compose with ComponentResources and { parent: this } as you expect it to. What version of pulumi are you using?

[pyoio]

latest, in our other stacks we just set the kubeconfig to trash but I was experimenting with the new disable support in this one.

package.json

{
  "name": "performance",
  "main": "src/index.ts",
  "devDependencies": {
    "@google-cloud/container": "^2.6.0",
    "@google-cloud/resource-manager": "^3.0.0",
    "@kubernetes/client-node": "^0.16.1",
    "@types/chai": "^4.0.0",
    "@types/mocha": "^9.0.0",
    "@types/node": "^16",
    "chai": "^4.0.0",
    "mocha": "^9.0.0",
    "node-fetch": "^2.6.0"
  },
  "dependencies": {
    "@js-joda/core": "^5.0.0",
    "@pulumi/gcp": "^6.0.0",
    "@pulumi/kubernetes": "^3.0.0",
    "@pulumi/pulumi": "^3.0.0",
    "@pulumi/random": "^4.0.0",
    "@types/promise-poller": "^1.7.1",
    "promise-poller": "^1.9.1",
    "typescript": "~3.8.0"
  },
  "scripts": {
    "lint": "eslint --fix **/*.ts",
    "check": "eslint **/*.ts",
    "test": "mocha -r ts-node/register tests/**/*.ts"
  }
}

package lock

  '@pulumi/pulumi': 3.24.1
  '@pulumi/query': 0.3.0
  '@pulumi/gcp': 6.10.0
  '@pulumi/kubernetes': 3.15.1
  '@pulumi/pulumi': 3.24.1
  '@pulumi/random': 4.3.1

stack config YAML

config:
  gcp:impersonateServiceAccount: <OMITTED>
  gcp:region: europe-west1
  pulumi:disable-default-providers:
  - kubernetes

index.ts

const k8sProvider = k8slib.gkeProvider(...); // tested and working in our other stacks

new PerformancePipelines(
  "performance-pipelines",
  {
    executionNamespace: "ci-epiu9r6l",
  },
  {
    providers: {
      kubernetes: k8sProvider,
    },
  }
);

PerformancePipelines.ts

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

export interface PerformancePipelinesArgs {
  executionNamespace: pulumi.Input<string>;
}

export class PerformancePipelines extends pulumi.ComponentResource {
  private readonly name: string;

  constructor(name: string, args: PerformancePipelinesArgs, opts?: pulumi.ComponentResourceOptions) {
    super("core_gcp:performance:PerformancePipelines", name, args, opts);
    this.name = name;

    const ns = k8s.core.v1.Namespace.get("execution-namespace", args.executionNamespace);

    const msg = pulumi.interpolate`ns: ${ns.metadata}`;

    msg.apply((m) => pulumi.log.info(m));
    // TODO: Create pipeline
    // TODO: Create TriggerTemplate
    // TODO: Create TriggerBinding
    // TODO: Create EventListener

    this.registerOutputs({});
  }
}

pulumi preview results in:

 kubernetes:core/v1:Namespace (execution-namespace):
    error: Preview failed: Default provider for 'kubernetes' disabled. 'urn:pulumi:performance.ci::performance::kubernetes:core/v1:Namespace::execution-namespace' must use an explicit provider.

[iwahbe]

I you still need { parent: this } to tell pulumi that "execution-namespace" is a child of "performance-pipelines".

const ns = k8s.core.v1.Namespace.get("execution-namespace", args.executionNamespace, { parent: this });

[pyoio]

I you still need { parent: this } to tell pulumi that "execution-namespace" is a child of "performance-pipelines".

const ns = k8s.core.v1.Namespace.get("execution-namespace", args.executionNamespace, { parent: this });

Jeeeeez, the one thing I forgot to copy 🤦 , alexa, how do you go back in time and take your foot out of your mouth? Cheers!

[mattfysh]

This seems like a very common requirement for people who are migrating to micro-stacks, as recommended by the docs here: https://www.pulumi.com/docs/using-pulumi/organizing-projects-stacks/

Having resources fail because you didn't set the parent or provider resource option is frustrating, when the functionality you really want (default provider config) already exists, but is just out of reach

Having said that, this seems to be limited to kubernetes clusters deployed by an infra-like microstack, and wanting to reference the kubeconfig from app-like microstacks. The best workaround I can think of for now is to use a temporary kubeconfig file, each app-like microstack must ensure the file exists and is populated before it can deploy

echo "$(pulumi stack output kubeconfig --cwd infra)" > .kubeconfig
KUBECONFIG=".kubeconfig" pulumi up --cwd app

If you have multiple stacks for each infra environment, make sure to get the kubeconfig output from the relevant stack

[mattfysh]

Adding another use case in hope of this issue being prioritized 🤞 you might have an infra microstack that deploys a kafka cluster, and then deploy individual kafka topics in app microstacks