CFEP 21 - conda-forge / Bioconda Transfer Policy - DRAFT

[mfansler]

I am proposing we formalize transfers between Conda Forge and Bioconda. Conda Forge to Bioconda transfers create problems for other packages on Bioconda that need the newer versions. I am suggesting Conda Forge creates a dedicated label that will be applied to all builds from a transferred feedstock. This would alleviate solving issues by moving the older builds off the main label.

@conda-forge/bioconda-recipes

Example Impacted Feedstocks

• archival of this recipe r-shazam-feedstock#13
• archival of recipe r-alakazam-feedstock#27
• Add r-ggm bioconda/bioconda-recipes#50793

Checklist

• Used a personal fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

[jaimergp]

I guess we should get sign off from bioconda members in this CFEP too. cc @conda-forge/bioconda-recipes

[mfansler]

Are you thinking a specific number? @bgruening and I are both members.

[jaimergp]

We might need to add code for this.

But I do see one issue: the package will disappear from main for those that do not use bioconda. What's the recommended practice here? Them adding the label after experience breakage? Is there a way to avoid the breakage from happening?

[mfansler]

Yes, users would experience an inability to find the package and would have to add the label to use it again. I don't think there is a way around this. It would be discoverable on Anaconda Cloud web searching, but perhaps something should be added to the archived feedstock itself. Can a message be added to the README? or create a pinned Issue with a link to documentation?

Maybe I can put some numbers together on what packages actually have moved before, but in my experience they are already bioinformatics oriented, so my expectation is that most users of such packages will already have Bioconda configured.

[mfansler]

I mention in the Implementation section that we would add a section under "User Documentation > FAQ" to cover how to use the label.

[jaimergp]

I wasn't aware that conda-forge recipes were being migrated to bioconda. I would have expected that a better solution was to onboard maintainers to the (I presume) outdated feedstocks, and this way we don't have to deal with these priority issues.

Is this extended practice or just a new case found in this r-alakazam package?

[mfansler]

I wasn't aware that conda-forge recipes were being migrated to bioconda. I would have expected that a better solution was to onboard maintainers to the (I presume) outdated feedstocks, and this way we don't have to deal with these priority issues.

Is this extended practice or just a new case found in this r-alakazam package?

I can try to add more examples at some point. The ones I see are driven by R packages adding a Bioconductor dependency, none of which we host. It is acutely problematic for r-alakazam because another Bioconda package has a lower bounds dependency above any of the CF builds.

An alternative solution could be that Bioconda creates an ability for recipes to optionally build with flexible channel priority. However, I still don't think it would hurt to formalize a transfer policy.

[jaimergp]

Ah, I see, not a matter of lack of maintenance, but new requirements not found in conda-forge and available in bioconda. I see.

Even if you manage to build with flexible, downstream users would have to also realize that, and they might be using strict, so yes, I don't think there's a way around 😬

If only conda could tell you why your package disappeared... I envision a removed section in repodata.json where not just the package is listed (like it is now), but there's a reason attached to it too, and we can display it to the user.

[mfansler]

@bgruening sorry for the delay in this. I'll try to touch it up to move it from Draft to Proposed.

I have one question for the strict-channel setting. If I have package A-0.1 in CF and A-0.5 in BC. Is a pin - A >=0.5 not working?

No, it does not work. In strict mode, having any version of package A in CF has the solver ignore all repodata in BC about A.

[mfansler]

@bgruening @conda-forge/bioconda-recipes I've updated this. Please review.

Note that the bioconda-recipes linter seems to strictly reject CF-to-BC transfers currently, such that these transfers cannot proceed further in PR CI without some solution along the proposed lines. (e.g., bioconda/bioconda-recipes#50793)

@beckermr is it necessary to prototype an implementation (e.g., a PR to admin-requests) to move this from "Draft" to "Proposal"?

[jakirkham]

cc @mbargull

[ggabernet]

Hi @mfansler, thanks for looking into that. Is there any progress here, as the issue with our two recipes still persists.

[ggabernet]

Hi @mfansler @beckermr @conda-forge/core is there a way to move this forward? It would be great to be able to solve this with our packages :)

[mfansler]

Last I checked in on this (January), there was openness in Bioconda to enable recipes to override strict channel priority to use flexible solve instead. In the end, that would likely be a more simple implementation, but I haven't yet looked into what work would need to be done to make this possible.

[beckermr]

We need to double check support for "flexible" solver priority across mamba and rattler. We may not have it in all cases.

[ggabernet]

@mfansler @beckermr I see that it would be advantageous to rather archive the old conda-forge recipes than allow flexible solve within bioconda, as also @jaimergp pointed out "even if you manage to build with flexible, downstream users would have to also realize that, and they might be using strict". They would then be using a very outdated version.

[ggabernet]

@beckermr @mfansler just bumping this as we'd still very much like to get a conda-installable up to date version of our packages

[jaimergp]

Sorry for dropping this, adding this to my list!

I think we need a clearer discussion on how this will impact end-users that only want to install packages and don't care where they come from. They should not need to change the configured channel priority and, at most, add a new channel to the list in a way that doesn't impact the environment resolution. This should be added to Rationale. Example cases I'd like to see discussed:

• How a conda-forge user may be able to install the package that moved to conda-forge. e.g. add bioconda to the channel list in which position and with which priority.
• How a bioconda user may use the package that moved to conda-forge. Probably no changes, because it's already part of their config.

[jaimergp]

Suggested change

	## Other sections
	Other relevant sections of the proposal. Common sections include:
	* Specification -- The technical details of the proposed change.
	* Motivation -- Why the proposed change is needed.
	* Rationale -- Why particular decisions were made in the proposal.
	* Backwards Compatibility -- Will the proposed change break existing
	packages or workflows.
	* Alternatives -- Any alternatives considered during the design.
	* Sample Implementation -- Links to prototype or a sample implementation of
	the proposed change.
	* FAQ -- Frequently asked questions (and answers to them).
	* Resolution -- A short summary of the decision made by the community.
	* Reference -- Any references used in the design of the CFEP.

[jaimergp]

I'd suggest not using too specific terms or guidelines in the Specification, in case those specifics change over time. For example, I'd remove the label details here, and then add an "Example workflow" section where those details are mentioned. The Specification should only say "Create an issue in bioconda/bioconda-recipes to notify the intent of transferring a package to conda-forge".

[jaimergp]

This should only happen once the bioconda team has agreed to allow the transfer to minimize noise. The reference to the comment in the bioconda org allowing the transfer should be mandatory.

[jaimergp]

This is an implementation detail we don't need in the specification. Perhaps in the Example workflow section.

[jaimergp]

This is implicit if the suggestions to point 2 are applied.

[jaimergp]

This should only happen after explicit approval by the feedstock maintainers and the bioconda team. I think I'm inclined to mark them as broken instead of relabeling. Not sure what happens with the original URL once they are relabeled, but we want to avoid 404s in lockfiles.

[jaimergp]

This should happen before the admin-requests PR. Once approved (but before the merge), then the admin-requests PR should be submitted. Both PRs should be merged as simultaneously as possible.

[jaimergp]

We need an extra step here where all the conda-forge packages for that named are copied to bioconda. I think this can be done programmatically with little effort.

[jaimergp]

Suggested change

	### Expected Implementation
	### Recommended Implementation

[jaimergp]

I don't think we need to discuss the documentation sections that explicitly.

[jaimergp]

I like this, tbh.

[dlaehnemann]

Yes, this sounds like a viable alternative.

[jaimergp]

Why is conda-forge placed before bioconda?

[dlaehnemann]

Because previously, packages have only migrated from bioconda to conda-forge, especially CRAN packages that don't have any bioconda dependencies. To properly resolve them, the channel order is recommended in that way. At least I think that was the motivation, but I gladly stand corrected by anybody with more in-depth knowledge.

[dlaehnemann]

I ran into the same issue, a CRAN package adding a bioconductor dependency for a bioconductor package that lives on bioconda. By the established rules, this suggests to migrate this package over to bioconda, but this is a headache with the current setup.

Just to cross-reference in my attempts, here is the attempt to create an r-nmf package on bioconda, which fails because of the channel_priority: strict, with the existing old package versions on conda-forge. And here is the first failing version of r-nmf on conda-forge, version 0.22.0.

Looking into this repeatedly, I think the two viable options I see would be the following. I would prefer the first one over the second, but wanted to put this up for reviving the discussion here:

1. Simply leave the package on conda-forge, adding the bioconda channel to the recipe/conda_build_config.yaml and dropping windows builds with skip: true # [win], as bioconda doesn't build for windows. This at least builds r-nmf again for linux and mac, for all of the numerous packages that depend on it, especially the bioconductor ones. And it doesn't cause any confusion with the discoverability and priority across channels. And moving the package to bioconda would also drop windows builds, anyways...
2. Go ahead with this migration proposal here, and transfer the old conda-forge build artefacts over to bioconda manually, as suggested as an alternative here. I think this would be the cleanest solution from the user perspective. The only thing that it breaks. is when users have environment definitions that only include conda-forge and not bioconda as a channel, but still want to install r-nmf.

[beckermr]

I ran into the same issue, a CRAN package adding a bioconductor dependency for a bioconductor package that lives on bioconda. By the established rules, this suggests to migrate this package over to bioconda, but this is a headache with the current setup.

Just to cross-reference in my attempts, here is the attempt to create an r-nmf package on bioconda, which fails because of the channel_priority: strict, with the existing old package versions on conda-forge. And here is the first failing version of r-nmf on conda-forge, version 0.22.0.

Looking into this repeatedly, I think the two viable options I see would be the following. I would prefer the first one over the second, but wanted to put this up for reviving the discussion here:

1. Simply leave the package on conda-forge, adding the bioconda channel to the recipe/conda_build_config.yaml and dropping windows builds with skip: true # [win], as bioconda doesn't build for windows. This at least builds r-nmf again for linux and mac, for all of the numerous packages that depend on it, especially the bioconductor ones. And it doesn't cause any confusion with the discoverability and priority across channels. And moving the package to bioconda would also drop windows builds, anyways...
2. Go ahead with this migration proposal here, and transfer the old conda-forge build artefacts over to bioconda manually, as suggested as an alternative here. I think this would be the cleanest solution from the user perspective. The only thing that it breaks. is when users have environment definitions that only include conda-forge and not bioconda as a channel, but still want to install r-nmf.

So a third, somewhat simpler proposal that would not require a CFEP would be to:

• copy the artifacts from conda-forge to bioconda
• mark them broken on conda-forge (a special procedure for conda-forge that removes them from the main label repodata, but keeps them available for lockfiles, explicit env exports, etc.)
• add the package to bioconda
• change the conda-forge recipe by having it be a simple metapackage that installs from bioconda via bioconda::<package>
• possibly backfill old package versions with metapackages as well

I think this would:

• fix the solver priority issue above (solver priority is enforced for the items found in the repodata at run time IIUIC)
• ensure that folks using only conda-forge can get at least get some sort of solver problem or issue or even an install of the package for updated versions
• ensure that folks using both bioconda and conda-forge can get updated versions
• avoid a circular dependence in the conda-forge feedstock on using bioconda to build a package (instead it is only used to point to bioconda via the metapackage, which is less fragile)

Thoughts?

[dlaehnemann]

Yes, this actually does sound like it seamlessly solves the all issues for users. Thanks for thinking this through with the extra insights you have into conda-forge workings and capabilities. I wouldn't have known about the broken label, or the ability to backfill old package versions.

Only the migration process is still a bit lengthy, with multiple steps. So maybe it does make sense to update this CFEP with all of these steps, fleshing out how exactly they are done and who will do what? Is there anything I can do to help with this?

[beckermr]

Let's try it once and see how it goes. If it actually works, then I think we add the steps to the docs and call it good. CFEPs are about policy, and this stuff is all about details.

[dlaehnemann]

Ah, OK. The docs probably are more discoverable, as well. Should we try it out with r-nmf? I would be up for getting started on this.

The only caveat is, that I will be starting parental leave any day now, and would then be away for 6 weeks. But until then I would make this a priority. And the bioconda pull request for r-nmf already exists, so as I don't have any permissions for conda-forge, most stuff would have to be handled by others, anyway.

Also, to give a little context: The package version I propose for bioconda is an old one, but the first one that didn't get built on conda-forge. So the idea is to add this and then go through all the incremental updates version by version, so we have all the newer versions up on bioconda through the regular process.

[beckermr]

Yes we can start with this package for sure. Let's make a tracking issue and try to get as far as we can when folks have time. You should definitely not worry about this though. Lots of changes coming your way!

[beckermr]

See conda-forge/conda-forge.github.io#2809. As we finish steps, we can mark them as done there.