Uploading to a custom channel

[astrofrog]

Is there a way to set up a feedstock to upload to a custom channel in addition to uploading to the conda-forge channel?

[Cadair]

+1 on this, it would be nice to be able to provide a curated subset at times.

[ocefpaf]

That is a good question!

I was maintaining the IOOS channel and conda-forge packages for a while, but we (IOOS) decided to just eliminate the IOOS channel in favor of conda-forge. However, I still see a lot of value in doing that.

There are a few options:

1. Fetch the recipe from the feedstock and re-build the it in another project;
2. Create a script that copies the binary from http://anaconda.org/conda-forge to another account;
3. Create a mechanism within conda-forge to perform an extra upload.

I do (1) here and right now that is the best option IMO b/c it gives me control over the build, and simplify the recipe maintenance. One canonical recipe and two customized builds.

I can see value in (2) and (3), but I would not use them as I (my employers actually) do want control over the build in other projects.

[astrofrog]

For us, I think 2 or 3 would be acceptable (we would trust the conda-forge builds and don't want to have to duplicate the build infrastructure). I think it would be really valuable to have option 3, i.e. have something built in to conda-forge to allow this.

[ericdill]

Regarding (2), that sounded like a great idea and something that I wanted for my day job. So here's my first pass at such a script. I'll start using it at work over the next week or so and probably find some bugs or corner cases, but feel free to do with it what you will with no strings attached. @ocefpaf any interest in putting this script in one of the conda-forge repos?

Usage is basically this:

Mirror the packages I'm working on adding:

BINSTAR_TOKEN=`cat token` conda-execute mirror.py --to-owner lightsource2 --from-owner conda-forge scikit-beam lmfit xraylib

Or mirror all of conda forge (not that this is necessarily a good idea...)

BINSTAR_TOKEN=`cat token` conda-execute mirror.py --to-owner lightsource2 --from-owner conda-forge .tar.bz2

[ocefpaf]

@ericdill Nice!

@ocefpaf any interest in putting this script in one of the conda-forge repos?

I believe that could live in https://github.com/conda-forge/conda-forge.github.io/tree/master/scripts, but I will pass that question to @pelson 😉

[astrofrog]

Ohh, this is great, thanks @ericdill! (and ping @Cadair and @mwcraig)

[mwcraig]

Hoping to open a PR on conda-build-all in the next day or two that will allow channel-to-channel copying...

[ericdill]

Could you elaborate a little on what you mean by channel-to-channel copying? conda-build-all currently allows you to upload to multiple channels via the --upload-channels argument which takes a list of channels.

[mwcraig]

What (I thought) was not working yet was moving a package from an --inspect-channels to one (or more) of the channels in --upload-channels. It is the code branch at artefact_destination.py#L116 that I'm thinking of.

[ericdill]

Ah I see! That seems like a useful thing to be able to do.

[jakirkham]

Adding this to the hackpad. Partially as I haven't been able to stay on top of the progress here, but would like to get a sense of how this is going and what the best way forward is. In particular, it seems like there are a few strategies in various stages to do 2 and it would be good if we could agree on a single strategy and push that forward. I can't promise how much will be talked about on this topic in the next meeting just because of how full it is and we have quite a few guests. Though we should certainly get to it in the following meeting. It would be great also if the future stakeholders can come and share a few thoughts about what would be a good way to do this.

cc @pelson @mcg1969 @ericdill @mwcraig

[mwcraig]

@jakirkham -- I'll try to make it to the next dev meeting but have family coming that day so might miss it.

A couple thoughts:

• I don't think it works to allows conda-forge builds to push themselves to a different channel...or, at the least, doing so will require adding a token to the feedstock that allows uploads to the non-conda-forge channel, which hands all other maintainers of that feedstock the same power.
• I have opened a PR to conda-build-all (Add cross-owner package copying conda-tools/conda-build-all#38) that would make this (fairly) easy, as it allows built packages found on inspect-channels to be pushed to upload-channels that have a different owner. I suppose this is still a little inconvenient because it requires that one have a recipe, but I'm mostly auto-generating those.
• Continuum willing, something like conda copy --from-channel conda-forge --to-channel astropy pkg1 pkg2... would be pretty sweet.

[ericdill]

@mwcraig IIUC the difference between the proposed syntax above and what already exists with anaconda-client is that you can

• specify multiple packages
• not have to specify the exact version you want to copy

Is that right?

[astrofrog]

Just for info, I've got a script that does the channel-to-channel syncing and copies all available builds for a given package (without the need to specify the version): https://github.com/glue-viz/conda-sync

[mwcraig]

Just for info, I've got a script that does the channel-to-channel syncing and copies all available builds for a given package (without the need to specify the version): https://github.com/glue-viz/conda-sync

Funny, me too: https://github.com/astropy/conda-build-tools/blob/master/extruder/copy_packages.py

[jakirkham]

A good tool along these lines in conda-mirror (if you are not already aware).

[jakirkham]

Given the general sentiment is to sync between channels and there has been a fair amount of work done in that direction, would suggest that we close this issue.