Windows VC matrix

[jakirkham]

While we have discussed this several places before, wasn't seeing this tracked anywhere. So thought it best to open an issue here. Guessing we will need to handle VC correctly here before we can address it in conda-smithy.

Currently in order to get a package to build with multiple VC runtimes, we need to add python as a build dependency and track a feature. However, this has proved impractical and in some cases limiting. Instead discussion over the past several months has concluded the best way forward is to have a separate vc package for tracking the compiler that python and other packages track. Though ultimately this must be addressed with some kind of matrix. While we can find somewhat effective ways to do this manually, conda-build-all has reigned supreme at helping convert an implicit matrix to an explicit matrix (via leveraging by conda-smithy). Hence I'm raising this here to ask that vc be handled in the matrix configuration used by staged-recipes and can picked up with conda-smithy for use in the feedstocks. There are already some considerations about how this might play out.

xref: conda-forge/glpk-feedstock#10
xref: conda-forge/tk-feedstock#2
xref: conda-forge/curl-feedstock#11
xref: conda-forge/conda-smithy#245

cc @jjhelmus @msarahan

[jakirkham]

Friendly nudge @pelson. 😄 Continuing to suffer without this functionality. See this re-rendering PR ( conda-forge/tk-feedstock#8 ) for an example. Any chance we could add vc detection to the version matrix?

[pelson]

I guess the most important one to xref is: #36
That PR needs resurrecting, and we should be good to go, right?

[jakirkham]

SGTM. Would it help to have conda-build learn more about VC long term? ( conda/conda-build#1445 )

[jakirkham]

Could you please also weigh in on this discussion @msarahan ?

[jakirkham]

We might be able to get around a matrix by doing clever hacks like PR ( conda-forge/tk-feedstock#20 ), but I don't know if we should be rolling that out on a large scale. Maybe conda-build 3 has better solutions available?

[pelson]

@jakirkham - what is the status on VC nowadays? Would you like me to resurrect #36, or shall we keep holding fire on that until we have a conda-build 3 usage plan (if we don't already)?

[jakirkham]

I'm pretty happy with the solution devised in PR ( conda-forge/tk-feedstock#20 ) even if it may be a hack. FWIU there is a solution in conda-build 3 that is different, but have had no time to familiarize myself with it. Maybe @msarahan can comment. My guess is we can close this though.

[msarahan]

cb3 uses variants, and one of the keys is vc:

https://github.com/AnacondaRecipes/qt-feedstock/blob/master/recipe/meta.yaml#L49

As for CB3 usage plan, I think that there are two steps, as I outlined at conda/conda-build@a617f57#commitcomment-24461476

1. Using cb3 with conda-build-all. I hope this is a drop-in thing. I tried to maintain the backward compatibility.
2. Taking advantage of cb3's build matrices while also maintaining the job splitting capability of conda-build-all. This is a little bit bigger project. It will probably largely duplicate concepts in https://github.com/conda/conda-concourse-ci (but c3i can probably be said to have duplicated conda-build-all). Specifically, conda-build-all or something like it render output metadata, then write that output data to a temporary location to be used by the build servers, or alternately, store environment variables as necessary.

Where conda-build 3 and conda-build-all differ most is that cb3 supports an arbitrary variable space, whereas c-b-a is limited to what conda-build 2 supports, which is numpy, python, perl, lua, and R. CB3 supports the same env vars as cb2, but the arbitrary extensibility doesn't translate into env vars. If you can think up a good way to have it available in env vars, I'd happily accept it.

Perhaps it could be something like the way that conda does its configuration vars, CONDA_*

[jakirkham]

TBH I think moving away from environment variables might be healthier. The trick that I imagine with doing this is creating sensible CI matrices from it. Though that is a problem at the conda-smithy level. Depending on what the answer is to that question, it may make more sense to go to conda-build directly from conda-smithy or preserve the status quo (i.e. conda-smithy -> conda-build-all -> conda-build).

[msarahan]

yeah, environment variables are not as nice, but they do make moving across multiple CI worker instances easier. Having temporary files makes the build infrastructure quite a bit more complicated.

[pelson]

Thanks the the insights. I'm going to close this out - the proof of concept in #36 is the way I'd solve the problem for CB2, but CB3's variants is the right way to go IMO. I'm definitely up for having conda-build-all/conda-smithy/conda-whatever generate the right conda_build_config.yaml if necessary.

If anything, just having conda-build-all and conda-build 3 playing together would be a nice first step.