Security updates for indirect dependencies for pip-compile

[not7cd]

This is a continuation of thread from pip-compile support feature request #24725 (comment)

In my specific use-case, I wish to create updates for indirect dependencies that can be only found in lock files requirements.txt. As of version 37.162 Renovate creates PRs only for dependencies that exist in package files requirements.in. For libraries, we will favour lockFileMaintenance but for services we prefer small incremental updates. Hence, this feature request.

In draft PR #26871, commit cffa2d5 I was able to cobble together a solution that is able to create PRs for all indirect dependencies by setting their depType to pip-indirect, and executing for each depType==pip-indirect function updateLockedDependency defined by pip-compile manager. This is a workaround and is quite limited as in configuration you need to disable updates for all packages, else it will create updates for all indirect packages. It will also silently fail if command errors out in updateLockedDependency. That's where updateArtifacts has an advantage, as it always creates PR and will include a comment with error message. This is desirable as it presents a place to start for resolving package conflicts.

My draft PR proves that this is possible for pip-compile to update single indirect dependency with --upgrade-package option, and it can be incorporated into Renovate.

I have created this discussion to explore ways to better incorporate such feature that will not require any additional configuration and work well out-of-the-box. Because what I understand, it will require a bit of refactoring. Hopefully this can begin a support for other package managers.

Here are similar discussions, I'm not aware of any other related to pip-compile:

• related to pep621 (poetry) Python-Poetry indirect dependency are not updated #14995
• related to Go Security updates for indirect dependencies #26703
• related to Go Indirect Go dependencies vulnerability alerts #22505
• related to Go gomod: Update transitive dependencies (with security vulnerabilities) #22487
• related to Java Does Renovate support scanning indirect dependencies of Java projects? #23610

[viceice]

updateLockedDependency should not call a package manager.

that should be done in updateArtifacts

indirect deps should be disabled by default.

[not7cd]

I will list functions, that I think, are crucial for such feature to work.

Here, updateArtifacts is called.

renovate/lib/workers/repository/update/branch/get-updated.ts

Lines 38 to 50 in ea5e20e

	export async function getUpdatedPackageFiles(
	config: BranchConfig,
	): Promise<PackageFilesResult> {
	logger.trace({ config });
	const reuseExistingBranch = config.reuseExistingBranch!;
	logger.debug(
	`manager.getUpdatedPackageFiles() reuseExistingBranch=${reuseExistingBranch}`,
	);
	let updatedFileContents: Record<string, string> = {};
	const nonUpdatedFileContents: Record<string, string> = {};
	const packageFileManagers: Record<string, Set<string>> = {};
	const packageFileUpdatedDeps: Record<string, PackageDependency[]> = {};
	const lockFileMaintenanceFiles = [];

Here in generateUpdate is decided if something is a isLockfileUpdate. This is the only place in code that sets this flag.

renovate/lib/workers/repository/process/lookup/generate.ts

Lines 95 to 97 in ea5e20e

	if (rangeStrategy === 'update-lockfile' && currentValue === update.newValue) {
	update.isLockfileUpdate = true;
	}

Here in lookupUpdates range strategy is adapted for security updates. lookupUpdates calls generateUpdate. This method also ignores certain dependencies.

renovate/lib/workers/repository/process/lookup/index.ts

Lines 228 to 234 in ea5e20e

	if (
	config.isVulnerabilityAlert &&
	rangeStrategy === 'update-lockfile' &&
	!config.lockedVersion
	) {
	rangeStrategy = 'bump';
	}

[viceice]

I think you need to extract and return indirect deps. set them to disabled by default. then the security stuff can reenable when required. or a user can do with a package rule of cause

[viceice]

I think the go manager already does this

[not7cd]

What is the meaning of transitive dependency in the context of npm and Renovate?

https://docs.renovatebot.com/configuration-options/#transitiveremediation
https://fossa.com/blog/direct-dependencies-vs-transitive-dependencies/