Harden automerge on slower Self-Hosted GitLab instances

[malmor]

Tell us more.

Hey there,
we recently stumbled upon an issue where Renovate auto-merged a dependency update, while the pipeline was failing. After digging a bit more into it, and checking the debug logs, it turns out that our GitLab instance was kinda slow with creating the new pipeline - and Renovate merged before a new test was ever started.

For reference:

• GitLab CE v19.2 (Self-Hosted)
• Renovate v44.35.2 (Docker image)
• platformAutomerge feature: enabled

We traced this back to the tryPrAutomerge function here:

renovate/lib/modules/platform/gitlab/index.ts

Lines 579 to 703 in dd72cf0

	async function tryPrAutomerge(
	pr: number,
	platformPrOptions: PlatformPrOptions | undefined,
	): Promise<void> {
	try {
	if (platformPrOptions?.gitLabIgnoreApprovals) {
	await ignoreApprovals(pr);
	}
	if (platformPrOptions?.usePlatformAutomerge) {
	// https://docs.gitlab.com/ee/api/merge_requests.html#merge-status
	const desiredDetailedMergeStatus = [
	'mergeable',
	'ci_still_running',
	'not_approved',
	];
	const desiredPipelineStatus = [
	'failed', // don't lose time if pipeline failed
	'running', // pipeline is running, no need to wait for it
	];
	const desiredStatus = 'can_be_merged';
	const env = getEnv();
	// The default value of 5 attempts results in max. 13.75 seconds timeout if no pipeline created.
	const retryTimes = parseInteger(
	env.RENOVATE_X_GITLAB_AUTO_MERGEABLE_CHECK_ATTEMPS,
	5,
	);
	const mergeDelay = parseInteger(
	env.RENOVATE_X_GITLAB_MERGE_REQUEST_DELAY,
	250,
	);
	// Check for correct merge request status before setting `merge_when_pipeline_succeeds` to `true`.
	for (let attempt = 1; attempt <= retryTimes; attempt += 1) {
	const { body } = await gitlabApi.getJsonUnchecked<{
	merge_status?: string;
	detailed_merge_status?: string;
	merge_when_pipeline_succeeds?: boolean;
	pipeline: {
	status: string;
	};
	}>(`projects/${config.repository}/merge_requests/${pr}`, {
	memCache: false,
	});
	// Exit early if merge_when_pipeline_succeeds is already set
	if (body.merge_when_pipeline_succeeds === true) {
	logger.debug(
	'Skipping automerge retry - merge_when_pipeline_succeeds already enabled',
	);
	return;
	}
	// detailed_merge_status is available with Gitlab >=15.6.0
	const use_detailed_merge_status = !!body.detailed_merge_status;
	const detailed_merge_status_check =
	use_detailed_merge_status &&
	desiredDetailedMergeStatus.includes(body.detailed_merge_status!);
	// merge_status is deprecated with Gitlab >= 15.6
	const deprecated_merge_status_check =
	!use_detailed_merge_status && body.merge_status === desiredStatus;
	// Only continue if the merge request can be merged and has a pipeline.
	if (
	(detailed_merge_status_check || deprecated_merge_status_check) &&
	body.pipeline !== null &&
	desiredPipelineStatus.includes(body.pipeline.status)
	) {
	break;
	}
	logger.debug(`PR not yet in mergeable state. Retrying ${attempt}`);
	await setTimeout(mergeDelay * attempt ** 2); // exponential backoff
	}
	// The merge_trains endpoint's auto_merge parameter requires GitLab
	// 17.11+. On older versions we fall back to the /merge endpoint so the
	// MR still automerges, just not on the train.
	// https://docs.gitlab.com/api/merge_trains/#add-a-merge-request-to-a-merge-train
	const useMergeTrain =
	config.mergeTrainsEnabled && !semver.lt(defaults.version, '17.11.0');
	if (config.mergeTrainsEnabled && !useMergeTrain) {
	logger.once.warn(
	{ version: defaults.version },
	'Merge trains require GitLab 17.11.0 or later, falling back to /merge endpoint',
	);
	}
	// Even if Gitlab returns a "merge-able" merge request status, enabling auto-merge sometimes
	// returns a 405 Method Not Allowed. It seems to be a timing issue within Gitlab.
	for (let attempt = 1; attempt <= retryTimes; attempt += 1) {
	try {
	if (useMergeTrain) {
	await gitlabApi.postJson(
	`projects/${config.repository}/merge_trains/merge_requests/${pr}`,
	{
	body: {
	auto_merge: true,
	},
	},
	);
	} else {
	await gitlabApi.putJson(
	`projects/${config.repository}/merge_requests/${pr}/merge`,
	{
	body: {
	should_remove_source_branch: true,
	merge_when_pipeline_succeeds: true,
	},
	},
	);
	}
	break;
	} catch (err) {
	logger.debug(
	{ err },
	`Automerge on PR creation failed. Retrying ${attempt}`,
	);
	}
	await setTimeout(mergeDelay * attempt ** 2); // exponential backoff
	}
	}
	} catch (err) /* v8 ignore next -- defensive: retry loop already swallows errors, outer catch is a last resort */ {
	logger.debug({ err }, 'Automerge on PR creation failed');
	}
	}

Before Renovate calls the /merge endpoint, it checks the merge request status a few times (with customizable retries and backoff), until this check passes:

renovate/lib/modules/platform/gitlab/index.ts

Lines 641 to 646 in dd72cf0

	// Only continue if the merge request can be merged and has a pipeline.
	if (
	(detailed_merge_status_check || deprecated_merge_status_check) &&
	body.pipeline !== null &&
	desiredPipelineStatus.includes(body.pipeline.status)
	) {

We did some tests by polling a merge request while adding a commit, here is some example logging:

Output from polling merge request via API every 0.25s

[10] {
  "id": 674286,
  "sha": "0755a167a90ebbe5e5fbab94615938af8884aae0",
  "merge_status": "can_be_merged",
  "detailed_merge_status": "mergeable",
  "merge_when_pipeline_succeeds": false,
  "pipeline": {
    "id": 16193596,
    "iid": 1197,
    "project_id": 5042,
    "sha": "0755a167a90ebbe5e5fbab94615938af8884aae0",
    "ref": "test-fail-event-change",
    "status": "failed",
    "source": "push",
    "created_at": "2026-08-19T14:06:41.673+02:00",
    "updated_at": "2026-08-19T14:07:02.585+02:00",
    "web_url": "https://git.company.local/demo/renovate/-/pipelines/16193596"
  }
}
[11] {
  "id": 674286,
  "sha": "0755a167a90ebbe5e5fbab94615938af8884aae0",
  "merge_status": "can_be_merged",
  "detailed_merge_status": "conflict",
  "merge_when_pipeline_succeeds": false,
  "pipeline": {
    "id": 16193596,
    "iid": 1197,
    "project_id": 5042,
    "sha": "0755a167a90ebbe5e5fbab94615938af8884aae0",
    "ref": "test-fail-event-change",
    "status": "failed",
    "source": "push",
    "created_at": "2026-08-19T14:06:41.673+02:00",
    "updated_at": "2026-08-19T14:07:02.585+02:00",
    "web_url": "https://git.company.local/demo/renovate/-/pipelines/16193596"
  }
}
[12] {
  "id": 674286,
  "sha": "8ea7197dac74fd87b221642ea4812db371fa3aac",
  "merge_status": "checking",
  "detailed_merge_status": "checking",
  "merge_when_pipeline_succeeds": false,
  "pipeline": {
    "id": 16193596,
    "iid": 1197,
    "project_id": 5042,
    "sha": "0755a167a90ebbe5e5fbab94615938af8884aae0",
    "ref": "test-fail-event-change",
    "status": "failed",
    "source": "push",
    "created_at": "2026-08-19T14:06:41.673+02:00",
    "updated_at": "2026-08-19T14:07:02.585+02:00",
    "web_url": "https://git.company.local/demo/renovate/-/pipelines/16193596"
  }
}
[13] {
  "id": 674286,
  "sha": "8ea7197dac74fd87b221642ea4812db371fa3aac",
  "merge_status": "can_be_merged",
  "detailed_merge_status": "mergeable",
  "merge_when_pipeline_succeeds": false,
  "pipeline": {
    "id": 16193596,
    "iid": 1197,
    "project_id": 5042,
    "sha": "0755a167a90ebbe5e5fbab94615938af8884aae0",
    "ref": "test-fail-event-change",
    "status": "failed",
    "source": "push",
    "created_at": "2026-08-19T14:06:41.673+02:00",
    "updated_at": "2026-08-19T14:07:02.585+02:00",
    "web_url": "https://git.company.local/demo/renovate/-/pipelines/16193596"
  }
}
[14] {
  "id": 674286,
  "sha": "8ea7197dac74fd87b221642ea4812db371fa3aac",
  "merge_status": "can_be_merged",
  "detailed_merge_status": "mergeable",
  "merge_when_pipeline_succeeds": false,
  "pipeline": {
    "id": 16193596,
    "iid": 1197,
    "project_id": 5042,
    "sha": "0755a167a90ebbe5e5fbab94615938af8884aae0",
    "ref": "test-fail-event-change",
    "status": "failed",
    "source": "push",
    "created_at": "2026-08-19T14:06:41.673+02:00",
    "updated_at": "2026-08-19T14:07:02.585+02:00",
    "web_url": "https://git.company.local/demo/renovate/-/pipelines/16193596"
  }
}
[15] {
  "id": 674286,
  "sha": "8ea7197dac74fd87b221642ea4812db371fa3aac",
  "merge_status": "can_be_merged",
  "detailed_merge_status": "mergeable",
  "merge_when_pipeline_succeeds": false,
  "pipeline": {
    "id": 16193607,
    "iid": 1198,
    "project_id": 5042,
    "sha": "8ea7197dac74fd87b221642ea4812db371fa3aac",
    "ref": "test-fail-event-change",
    "status": "running",
    "source": "push",
    "created_at": "2026-08-19T14:09:43.860+02:00",
    "updated_at": "2026-08-19T14:09:45.279+02:00",
    "web_url": "https://git.company.local/demo/renovate/-/pipelines/16193607"
  }

Here we can see that GitLab considered the merge request mergeable while the pipeline.id still pointed to the old commit - so Renovate might merge before a pipeline got created.

I am aware that the root cause here is a (occasionally) slow GitLab instance, and the fact that our repository did not have the Pipeline must succeed feature enabled - which would have blocked the merge in the first place. But since there are some scenarios where that feature is disabled, the question is if the existing path could maybe be hardened a bit.

Looking at this check again:

renovate/lib/modules/platform/gitlab/index.ts

Lines 641 to 646 in dd72cf0

	// Only continue if the merge request can be merged and has a pipeline.
	if (
	(detailed_merge_status_check || deprecated_merge_status_check) &&
	body.pipeline !== null &&
	desiredPipelineStatus.includes(body.pipeline.status)
	) {

This check only waits for a pipeline to be part of the merge request, which can also be for an older commit. Looking at the logs above, we can see that the pipeline.sha did not match the main sha until the new pipeline started.

What do you think about adding this as an additional hardening to the condition? This would ensure that the pipeline / status is actually based on a pipeline for that last commit, and the retries / backoffs would help for slower instances. In the end this does not break existing flows, since the fallback (after all retires) still just attempts to merge.

I'd be happy to create a pull request for that.

Cheers, Malte

[gaeljw]

Maybe related to #46225

[pdecat]

And #46236