Resource timeout for YAML manifests with many resources #1098
Comments
|
The k8s provider has retries built in, so normally you don't have to do anything special to get it working. For large manifests like this, you may want to split the CRDs out into a separate YAML file to make sure they're ready before creating the other resources. Here's an example for Istio that does this: pulumi-kubernetes/tests/integration/istio/step1/istio.ts Lines 43 to 69 in 6c38527 |
|
Thank you, I will try it. |
|
In our case we create a full EKS cluster and multiple objects are installed we recommend using ComponentResource to correctly use depends_on, I solved the dependency problem in python for istio as follows: import pulumi_kubernetes as k8s
istio_init = k8s.helm.v3.Chart(release_name=args.chart_name_init,
config=k8s.helm.v3.ChartOpts(
chart=args.chart_name_init,
version=args.chart_version,
namespace=layer_ns.metadata["name"],
fetch_opts=k8s.helm.v3.FetchOpts(
repo=args.chart_url
),
),
opts=ResourceOptions(
parent=layer_ns,
depends_on=[layer_ns],
)
)
values_istio = data.values_render(
data={
"vars": {
"environment": args.environment,
"domain_name": r53_zone.name,
"istio_record_ttl": "60",
"grafana_enabled": "false",
"kiali_enabled": "false",
"tracing_enabled": "true",
"sds_enabled": "true"
}}
)
job_all_name = f"istio-init-crd-all-{args.chart_version}"
job_mixer_name = f"istio-init-crd-mixer-{args.chart_version}"
job_all_status = istio_init.get_resource(group_version_kind="batch/v1/Job",
name=job_all_name,
namespace=args.istio_namespace,
)
job_mixer_status = istio_init.get_resource(group_version_kind="batch/v1/Job",
name=job_mixer_name,
namespace=args.istio_namespace,
)
def deploy_istio(x):
return k8s.helm.v3.Chart(
release_name=args.chart_name,
config=k8s.helm.v3.ChartOpts(
chart=args.chart_name,
version=args.chart_version,
namespace=layer_ns.metadata["name"],
values=values_istio,
fetch_opts=k8s.helm.v3.FetchOpts(
repo=args.chart_url
)
),
opts=ResourceOptions(
parent=layer_ns,
depends_on=[
x[0],
x[1],
],
)
)
istio = Output.all(job_all_status, job_mixer_status).apply(lambda x: deploy_istio(x))It was detected that when you have several services at the same time, the implementation is not generated, you must deactivate kiali and grafana for the installation to be successful. Apparently there is a problem with concurrency and wait state on resources deployed in kubernetes |
lblackstone
changed the title
lblackstone
added
impact/reliability
Problem description
I am trying to create a Kubernetes Cluster that contains, cert-manager, istio and a sample Nginx Application that uses both cert-manager and istio CRD, and all this 3 are yamls in a total of 149 resources to be created, that I am creating with python pulumi_kubernetes configFile.
I am aware of pulumi await logic here
I am having problem creating the cert-manager CRD ClusterIssuer, it seems to ignore the await logic, even when I use the annotation pulumi.com/timeoutSeconds It fails in 5 min, ignoring the value I set. the resource fails because other resource is not ready yet, it fails with the error:
I have the impression that Pulumi is trying to create the resources in the wrong order.
Because in this example the ClusterIssuer is waiting to make a post to a service that will only be ready when a deployment is created, but pulumi not scheduled the Cert-manager deplyoments yet. I have this impression because when the clusterIssuer fails, then pulumi fails without not event trying to create the deployments. I think that the fault is in the creation order, you can see in the logs bellow that pulumit not even tryed to create the deployment of the webhook.
the webhook fail log:
The deployment that pulumi not tryed to create and is required to the completion:
Is there a documentation about this creation orders?
Errors & Logs
Affected product version(s)
Pulumi
v2.0.0
Pip freeze
Reproducing the issue
If needed I can send the source code.
The text was updated successfully, but these errors were encountered: