Renovate not exiting since 24.40.0

[caphrim007]

What Renovate type, platform and version are you using?

We run the self-hosted renovate full image (renovate/renovate) in a kubernetes cronjob.

Describe the bug
Starting in version 24.40.0, the renovate process no longer exits.

It emits an exiting log message,

DEBUG: http statistics (repository=myorg/myrepo)
       "hostStats": [
         "111111111111.dkr.ecr.us-east-1.amazonaws.com, 7 requests, 52ms request average, 0ms queue average",
         "api.bitbucket.org, 5 requests, 551ms request average, 0ms queue average"
       ],
       "totalRequests": 12
 INFO: Repository finished (repository=myorg/myrepo)
       "durationMs": 8197
DEBUG: Renovate exiting

but the process in the container never dies. This causes the pod to never die, and as a result, the cronjob never finishes. This blocks all future running of the job.

In version 24.39.3 this problem did not exist. However I looked at the source code diff between the two versions of renovate, and there's nothing that jumps out at me as to why the process is suddenly now failing to stop.

Relevant debug logs

Have you created a minimal reproduction repository?

Please read the minimal reproductions documentation to learn how to make a good minimal reproduction repository.

• This is a really small bug, it does not need a reproduction (think small typo)
• I have provided a minimal reproduction repository
• I don't have time for that, but it happens in a public repository I have linked to
• I don't have time for that, and cannot share my private repository
• The nature of this bug means it's impossible to reproduce publicly

Additional context

[viceice]

How do you find the image? Are you overriding the entry point?

[caphrim007]

No, there's no entrypoint specified. Only an args value

          containers:
          - image: renovate/renovate:24.39.3
            args:
            - myorg/myrepo
            env:
            - name: RENOVATE_CONFIG_FILE
              value: /config/renovate-config.js
            - name: RENOVATE_BASE_DIR
              value: /scratch
            - name: TMPDIR
              value: /scratch
            - name: LOG_LEVEL
              value: debug
            - name: RENOVATE_USERNAME
              valueFrom:
                secretKeyRef:
                  key: renovate-username
                  name: renovate-infra-ver-1
            - name: RENOVATE_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: renovate-password
                  name: renovate-infra-ver-1
            - name: GITHUB_COM_TOKEN
              valueFrom:
                secretKeyRef:
                  key: github-token
                  name: renovate-infra-ver-1

[rarkins]

How is ECR authentication achieved? Eg is it run on EC2 with an IAM role?

[caphrim007]

yes, web-identity tokens using IAM Roles for Service Accounts. The pod has IAM permissions to pull from the ECR registry

[viceice]

@caphrim007 Can you check which processes are running in container? When i do some local tests it will simply work for me.
Renovate is exiting normally. (no docker, physical run)

[viceice]

verified with renovate/renovate:24.40.0-slim. used password from aws ecr get-login-password to login

[viceice]

@caphrim007 can you provide steps to pass the IAM stuff to renovate, as i don't know how to do / to test that

[caphrim007]

@viceice some more information

v24.39.3
processes in container. we run the renovate process as user 9999

9999         1     0  0 16:08 ?        00:00:00 dumb-init -- renovate myorg/myrepo
9999         7     1 13 16:08 ?        00:00:01 node /usr/local/bin/renovate myorg/myrepo

v24.40.0

9999         1     0  0 16:10 ?        00:00:00 dumb-init -- renovate myorg/myrepo
9999         7     1 13 16:10 ?        00:00:02 node /usr/local/bin/renovate myorg/myrepo

It feels like something it not being returned to dumb-init, but I don't know what would be not returning. Is it possible that the ecr-client is not closing its connection and just hanging open? There's no authentication error that I can see. Again, renovate runs just fine. Its the "what happens after its done" part that seems to be a problem.

I observed this as well

v24.39.3

[dumb-init] Unable to detach from controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] Child spawned with PID 6.
[dumb-init] Unable to attach to controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] setsid complete.

... lots of renovate debug output here

DEBUG: Renovate exiting
[dumb-init] Received signal 17.
[dumb-init] A child with PID 6 exited with exit status 0.
[dumb-init] Forwarded signal 15 to children.
[dumb-init] Child exited with status 0. Goodbye.

where-as in v24.40.0 I see this

[dumb-init] Unable to detach from controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] Child spawned with PID 6.
[dumb-init] Unable to attach to controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] setsid complete.
...
DEBUG: Renovate exiting
...no dumb-init output <--- not the actual message, its just that there is no further output here

dumb-init is not reporting that it received anything from its child process.

In terms of passing the IAM stuff to renovate, well, I don't know how trivial it is.

The process we're using is called IAM Roles for Service Accounts.

https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

There's a role in IAM with permissions to read cloud resources if needed. That role arn is provided as an annotation on the Kubernetes Service Account. Then AWS magic will cause the Pod to have 2 environment variables set when it starts

AWS_ROLE_ARN=arn:aws:iam::111111111111:role/MyServiceRoleForRenovate
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

The content of the web identity token file is a JWT token. This token can be used to assume a role in IAM; the role in AWS_ROLE_ARN.

In all situations, the way that this is best handled is to just leave it to the AWS SDK. If that env var is present, the SDK libraries will use it and get access to whatever cloud resources they need.

But having said that, again, all of that "stuff" appears to be working just fine. It is able to auth to ECR, it finds new images, it sends PRs, that's all good. Its just when it goes to exit, it behaves like something is...like...still open, or waiting, or the exit code is not trickled up to dumb-init.

I'm suspicious of the aws-sdk/ecr-client. I'm looking in to how I can revert just this patch here

c90c506#diff-922b65de7866726a28ede70241c7f637072005518289aa3755070398bc5b1db7

and re-build a renovate image to see if that is the culprit.

[caphrim007]

adding more notes here for reference.

I added this library to the renovate image https://github.com/myndzi/wtfnode

and added its usage to renovate.js within this block at the end

var wtf = require('wtfnode'); <--- added
(async () => {
    process.exitCode = await globalWorker.start();
    wtf.dump();   <--- added
})();

On the working image (39.3), this outputs the following

DEBUG: Renovate exiting
[WTF Node?] open handles:
- File descriptors: (note: stdio always exists)
  - fd 1 (stdio)
  - fd 2 (stdio)

And then the process exits

On the 40.0 image, the output is the following

DEBUG: Renovate exiting
[WTF Node?] open handles:
- File descriptors: (note: stdio always exists)
  - fd 1 (stdio)
  - fd 2 (stdio)
- Sockets:
  - 10.104.13.233:45560 -> 169.254.169.254:80
  - 10.104.13.233:45590 -> 169.254.169.254:80
  - 10.104.13.233:45620 -> 169.254.169.254:80

Note the mention of the sockets. My hypothesis is that aws-sdk/client-ecr...or the usage of it...is keeping these open. Since they never close, this prevents the anonymous async function from exiting.

I additionally tried changing the anonymous function to be the following to see if it would affect the cronjob run

(async () => {
    process.exitCode = await globalWorker.start();
    process.exit(0);
})();

The above does make the cronjob succeed. This is discussed in several forums and the nodejs notes as not recommended because

"""
Calling process.exit() will force the process to exit as quickly as possible even if there are still asynchronous operations pending that have not yet completed fully, including I/O operations to process.stdout and process.stderr.
"""

This would seem to reasonably explain why it works though when I test its usage; ie, stopping an async operation which hasn't fully completed.

https://nodejs.org/api/process.html#process_process_exit_code
https://stackoverflow.com/a/37592669

I am trying to understand further why the sockets remain open, if that is expected, and whether there is an appropriate way to gracefully shut them down.

I've pasted the output of the following two functions which provide more insight into these sockets

    //console.log("handles:", process._getActiveHandles());
    //console.log("requests:", process._getActiveRequests());

https://gist.github.com/caphrim007/5ec84329d962eff26ad3f2c8d57c9f4a

Will post more as I learn more

[rarkins]

We should check in case the library has a function to close sockets. Otherwise maybe it's a library bug/shortcoming. We can consider a hard process exit too

[viceice]

@caphrim007 Are the ips the originals? Do you have a kind of proxy between ecr and your renovatebot?

I remember we had some trouble with proxies. Maybe they can be the cause why the connection isn't closed. There are a lot issues in node about not closing http sockets.

[rarkins]

Those IPs look like the EC2 metadata endpoint: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/instancedata-data-retrieval.html

Ie used to fetch credentials, not ECR itself

[viceice]

ok, so it looks like the problem only occures aws internally. make it even harder to debug. I'll check the awds sdk issues for this problem later

[viceice]

btw: there was an aws sdk update a few version after 24.40. @caphrim007 did you chaecked latest renovate version too?

[rarkins]

I updated the ECR lib in #8680

Don't see any mention of this problem being fixed but you never know.

[viceice]

Renovate v24.40 comes with aws sdk v3.2. v24.42 brings aws v3.4.1 and your pr is in renovate v24.49.7 and brings aws v3.5.

That's way i ask if he tested latest renovate version too.

Maybe aws sdk v3.3 makes a difference.

[caphrim007]

@viceice I've assembled a minimal (non-renovate) repro that exhibits the behavior I've been describing.

Before I post it, I looked through the issues for client-ecr and found these

• Credentials from EKS service account aws/aws-sdk-js-v3#1808
• Set default H2 session timeout aws/aws-sdk-js-v3#1206

Though I can't say for sure, because again, renovate doesn't have any problem querying my private ECR registry from inside the cluster...afaik...I could be convinced that issue 1808 is related to this.

When I run the following code in a cronjob pod within kubernetes, I get the same behavior I described in this thread.

const { ECR } = require("@aws-sdk/client-ecr");

async function getECRAuthToken() {
  const config = {
     region: "us-east-1", accessKeyId: undefined, secretAccessKey: undefined,
  };
  const ecr = new ECR(config);
  try {
    const data = await ecr.getAuthorizationToken({});
    const authorizationToken = data?.authorizationData?.[0]?.authorizationToken;
    if (authorizationToken) {
      return authorizationToken;
    }
  } catch (err) {
    console.log({ err }, 'err');
    console.log('ECR getAuthorizationToken error');
  }
  return null;
}

(async () => {
  process.exitCode = await getECRAuthToken();
  console.log("Exiting");
})();

It's a semi-faithful repro.

I also made a repro that uses the before-change code

var AWS = require("aws-sdk");
var wtf = require('wtfnode');

async function getECRAuthToken() {
  const config = {
     region: "us-east-1", accessKeyId: undefined, secretAccessKey: undefined,
  };
  const ecr = new AWS.ECR(config);
  return new Promise((resolve) => {
    ecr.getAuthorizationToken({}, (err, data) => {
      if (err) {
        console.log({ err }, 'err');
        console.log('ECR getAuthorizationToken error');
        resolve(null);
      } else {
        const authorizationToken =
          data?.authorizationData?.[0]?.authorizationToken;
        if (authorizationToken) {
          resolve(authorizationToken);
        } else {
          console.log(
            'Could not extract authorizationToken from ECR getAuthorizationToken response'
          );
          resolve(null);
        }
      }
    });
  });
}

(async () => {
  process.exitCode = await getECRAuthToken();
  wtf.dump();
  console.log("Exiting");
})();

It produced this output

[dumb-init] Unable to detach from controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] Child spawned with PID 6.
[dumb-init] Unable to attach to controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] setsid complete.
[WTF Node?] open handles:
- File descriptors: (note: stdio always exists)
  - fd 2 (stdio)
  - fd 1 (stdio)
- Sockets:
  - (?:?) -> api.ecr.us-east-1.amazonaws.com:? (destroyed)
Exiting
[dumb-init] Received signal 17.
[dumb-init] A child with PID 6 exited with exit status 0.
[dumb-init] Forwarded signal 15 to children.
[dumb-init] Child exited with status 0. Goodbye.

So there is clearly a difference between aws v2 sdk and v3 sdk. Even if the difference between await and callbacks is functionally identical, something is not the same buried in the v3 sdk.

According to the first GH issue I linked above, AWS claims an EKS web-id token fix is aimed for release in v3.6.0 scheduled for release at/around 2/18.

@rarkins and @viceice if you don't mind, I'm going to temporarily table this discussion because I don't think this is renovate's problem; you're just the affected party.

I'm going to reach out to my AWS TAM and provide my aws repros and have them tell me if there's a bug in their SDK. When I learn more I'll post here.

In the meantime I can leave myself pinned to the 24.39.3 version or (at worst) I can build a new image with FROM renovate/renovate with an added inclusion of a process.exit(0) in the renovate.js file.

Will update this thread when I know more

[rarkins]

My suspicion now is that it's related to IAM code and not ECR directly. Ie it's the code that fetches your credentials that's left hanging and not the code that fetches from ECR

[caphrim007]

The AWS team has pushed back the release of the web-id token support to work on (in their opinion) a more important missing feature. As such, this issue will be pushed back accordingly. just fyi

[Turbo87]

FWIW we are experiencing a similar issue. our setup is a self-hosted renovatebot installed via npm and run on GitLab CI in the regular node:12 Docker image.

[viceice]

FWIW we are experiencing a similar issue. our setup is a self-hosted renovatebot installed via npm and run on GitLab CI in the regular node:12 Docker image.

Please use node:14, or even better renovate/renovate:slim, which works fine with dind.

https://gitlab.com/renovate-bot/renovate-runner

[Turbo87]

Please use node:14

yeah, we just upgraded to that, but it did not solve the issue unfortunately :-/

[rarkins]

Shall we add a delay followed by process.exit() until AWS fixes it? It would be nice if there was some function we could call to find out what's stopping the exit normally and log it too.

[narwold]

I'm experiencing this issue w/ both the renovate/renovate and renovate/renovate:slim images on Buildkite. Any current workarounds?

[rarkins]

@narwold are you also using an ECR registry?

[viceice]

@rarkins Maybe use an env variable a workaround to do a force exit when renovate is done?

[caphrim007]

the AWS team behind the SDK, as i mentioned in a previous post, seems to have de-prioritzed (what I think) is the solution (or towards a solution?). I haven't commented back on this thread yet because they haven't made any commits related to it. just fyi.

[rarkins]

@viceice does #9152 looko?

[viceice]

We have a new experimental environment variable to force a hard exit, so please have a try until the issue is fixed upstream on aws sdk.

RENOVATE_X_HARD_EXIT=true

https://docs.renovatebot.com/self-hosted-experimental/#renovate_x_hard_exit

[viceice]

Maybe it's time to raise an issue at aws sdk.

[Turbo87]

FWIW I can confirm that using RENOVATE_X_HARD_EXIT fixes the problem for us. Thanks a lot! :)

[msw-kialo]

@rarkins aws/aws-sdk-js-v3#1808 has been implemented and released as 3.10.0. Would it be possible to bump aws-sdk/ecr-client to a newer version? Based on some tests in EKS it appears to resolve both the termination issue and the ECR authentication issue.

[viceice]

Updated, should be released in a few minutes

[rarkins]

Ref: #10527

[viceice]

Release as v25.49.1, can anybody confirm that it works now without the RENOVATE_X_HARD_EXIT workaround?

[caphrim007]

I will chime back in with results on my end once I see the image come through.

[msw-kialo]

v25.49.1 works great for me. Renovate terminates correctly (without RENOVATE_X_HARD_EXIT). Authentication to fetch ECR tags is handled automatically by assuming an AWS role via web identity / OIDC. Both did not work with v25.49.0

[caphrim007]

I can verify what @msw-kialo says. it's working for me also without the env var.

[rarkins]

Thanks everyone. We'll remove the env var now and close this issue in the process

[caphrim007]

i dont know if this is specifically related to the last release, but somewhere in my upgrade path, after i put 25.49.1 into prod, renovate autoclosed all my existing open PRs and hasn't opened any new PRs since....not sure what happened

[caphrim007]

ahh...seems it might be related.

           {
             "packageFile": "k8s/overlays/stg-01/values.yaml",
             "deps": [
               {
                 "depName": "11111.dkr.ecr.us-east-1.amazonaws.com/vssopsd",
                 "currentValue": "2021.6.21-696",
                 "datasource": "docker",
                 "versioning": "loose",
                 "replaceString": "image: \"11111.dkr.ecr.us-east-1.amazonaws.com/vssopsd:2021.6.21-696\"\n\n    ",
                 "depIndex": 0,
                 "updates": [],
                 "warnings": [
                   {
                     "topic": "11111.dkr.ecr.us-east-1.amazonaws.com/vssopsd",
                     "message": "Failed to look up dependency 11111.dkr.ecr.us-east-1.amazonaws.com/vssopsd"
                   }
                 ]
               }
             ],

seems i need to retract the earlier statement about it knowing about webid tokens. the process exits completely with the removal of the env var, but I dont see web-id tokens working on my end. @msw-kialo can you provide any more detail on your env working?

[caphrim007]

for reference i reverted back to 25.47.6, with env var to force quit, and ECR registries were found again without issue. Will need to spend time tomorrow bisecting this unless anyone has ideas from merged PRs between 25.47.6 and latest

[viceice]

Can you verify v25.49.0 is working, so the issue is the SDK update in v25.49.1?

[msw-kialo]

@caphrim007 The usual setup: the serviceAccount for the cronjob/pods has the eks.amazonaws.com/role-arn annotation (and the role has the needed permissions and can be assumed via webidenity). One caveat we stumbled on a couple of times: the token is passed into the pod via a file. Renovate might not have file permissions to actually read its content. The following securityContext works for us

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 1000  # this was crucial for us

In our case the EKS cluster and the ECR repositories are in different AWS accounts (shouldn't matter) but in the same AWS region: We haven't checked whether this case is handled correctly. We did have issues with this in other software (the injected AWS_REGION and AWS_DEFAULT_REGION values were a problem, setting them blank resolved the issue).

[viceice]

@caphrim007 Would this work:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 0

As our default image user has uid=1000 and gid=0.

[caphrim007]

@viceice @rarkins i think I figured it out...after a long day of console.log'ing my way through the code.

The first thing that I got stuck on was this

https://github.com/renovatebot/renovate/blob/main/lib/datasource/docker/index.ts#L97

The first time through, that appears to work fine (it skips), however all future runs of renovate (i was running it many times in a single long-running pod) would return there because the return value of .get is null on a second pass, not undefined.

I looked at some other datasources and they all use if (cachedResult)

• https://github.com/renovatebot/renovate/search?q=cachedResult

docker, however, does not (among like 3 in total). Perhaps this is a bug. It certainly made debugging in run-time challenging.

After I got that sorted, I realized I needed to get to the bottom of whatever was in this err

• https://github.com/renovatebot/renovate/blob/main/lib/datasource/docker/common.ts#L48

I spent some time reading about ecr client, the newer web-identity packages that aws released, and I tried this solution to see if it just plain out wasn't reading the token to get a credential

• Call fromTokenFile credential provider by default in credential-provider-node aws/aws-sdk-js-v3#2148 (comment)

I finally reached a point where the aws-sdk-js sent to stdout that I was "unable to sts:AssumeRoleWithWebIdentity". This was super weird to me because...I swear...this has been working without issue for ages. There was this ticket that prompted the _X_ env var, but adding said env var made it work.

To answer @viceice 's question, yes, it was working with v25.49.0. I specifically tried it because you had asked. I tried the suggested securityContext tweaks as well, without success; same error I posted earlier about Failed to look up dependency.

When I looked at the trust relationship as well as the annotation specifying the role arn in the service account in k8s, I found what @msw-kialo so presciently alluded to; they were different.

My ServiceAccount annotation was Renovate. The role in IAM was RenovateBot. Somewhere...in the history of the universe...we must have made this change in Kubernetes, but not updated it in cloud. So I'm baffled by how it was even working in that clearly incorrect configuration.

After that fix, there was one more error. The existing IAM policy that I've used for ages, seems to have gained a requirement that I missed somewhere in release notes. Specifically, it now additionally needs ecr:GetAuthorizationToken.

After I added that, the service came back and the lookup errors went away.

I'm going to do some more clean-up on my end to make sure this is now correct, as well as try the newer releases of renovate to see if I continue to see success.

Will update the ticket here shortly once I do that.

Appreciate the help that everyone on this thread provided!

[rarkins]

The first time through, that appears to work fine (it skips), however all future runs of renovate (i was running it many times in a single long-running pod) would return there because the return value of .get is null on a second pass, not undefined.

In Renovate datasources we use null to mean "no lookup result". So I do think it's correct to cache a null result if it's non-temporary and consider that a cache hit. @JamieMagee have you noticed this during your refactorings?

[JamieMagee]

I've not seen that issue before. I've copied the same cache checking logic as is used everywhere else. We will cache a null. See

renovate/lib/util/cache/package/decorator.ts

Lines 106 to 118 in 226ec07

	const cachedResult = await packageCache.get<unknown>(
	finalNamespace,
	finalKey
	);
	if (cachedResult !== undefined) {
	return cachedResult;
	}
	const result = await callback();
	await packageCache.set(finalNamespace, finalKey, result, ttlMinutes);
	return result;

[rarkins]

I've decided to leave the env var there in case it's of use to people in future, so will close this now.