first draft of conda-build variants docs #414

msarahan · 2017-01-29T01:49:33Z

Proposed docs for conda/conda-build#1585

@cio-docs please ignore for now, but would appreciate copy editing soon, after this functionality has undergone more review and testing.

electronwill · 2017-01-30T23:25:40Z

Sounds good! Just let us know when it's ready and we should be able to help with copy editing.

groutr · 2017-02-07T18:51:05Z

docs/source/building/variants.rst

+2. The dependency is listed by name (no pinning) in the requirements/run section
+3. The ``pin_run_as_build`` key in the variant has a value that is a dictionary,
+   containing a key that matches the dependency name listed in the run
+   requirements. The value should be a pinning expression, or a tuple of two pinning expressions to


to ... ? Is there more to this item?

groutr · 2017-02-07T18:56:36Z

docs/source/building/variants.rst

+The result here is that the runtime numpy dependency will be pinned to
+``>=(current numpy 1.11.x version),<1.12``
+
+Numpy is an interesting example here. It actually would not make a good case for


Maybe have a more straightforward example for pinning at the variant level and leave numpy as a note. Would boost be a better example?

groutr · 2017-02-07T18:59:50Z

docs/source/building/variants.rst

+                                                  u'source': {u'path': u'/Users/msarahan/code/conda-build/tests/test-recipes/split-packages/_rm_rf_stays_within_prefix'}}}
+
+
+Extra Jinja2 functions


Would be good if arguments and their meaning were explained here.

msarahan · 2017-02-13T16:07:26Z

Thanks @kerrywatson1 - I'm not sure, but I think your comments apply to the docs as a whole, not specifically to this PR. Can we separate your comments that apply to content outside of this PR to another PR?

kerrywatson1 · 2017-02-13T16:10:20Z

Sure, @msarahan. Moved to https://github.com/ContinuumIO/docs/issues/879

groutr

Some general comments and a few minor nitpicks (which I won't feel bad if you choose to ignore).

groutr · 2017-02-16T00:47:42Z

docs/source/building/variants.rst

+characters allowed). This example builds python 2.7 and 3.5 packages in one
+build command:
+
+variant config file like:


A nitpick here for consistency. I would suggest referring to the variant config by a filename (conda_build_config.yaml) here to be consistent with how you refer to meta.yaml below.

groutr · 2017-02-16T00:52:39Z

docs/source/building/variants.rst

+   lists of versions to iterate over. These are aggregated as detailed in the
+   Aggregation of multiple variants section below.
+
+2. Set the ``variant`` member of a Config object. This is just a simple


What is the difference between "just a dictionary" and "just a simple dictionary"?

groutr · 2017-02-16T00:57:33Z

docs/source/building/variants.rst

+   Aggregation of multiple variants section below.
+
+2. Set the ``variant`` member of a Config object. This is just a simple
+   dictionary. The values for fields should be strings, except "extended keys",


I don't think this is an accurate statement. The example variant in line 119 shows a field with a list for a value.

groutr · 2017-02-16T01:05:49Z

docs/source/building/variants.rst

+variant. The way this works is that all variants are evaluated, but if any
+hashes are the same, then they are considered duplicates, and are deduplicated.
+By omitting some packages from the build dependencies, we can avoid creating
+unnecessarily unique hashes, and allow this deduplication.


I would probably word this "avoid creating unnecessarily specific hashes". Since hashes are meant to be unique, it was a little confusing to understand at first.

groutr · 2017-02-16T01:10:00Z

docs/source/building/variants.rst

+
+
+Extended keys
+-------------


And example of extended keys would be great. The concept is still a little nebulous for me. What are the use cases?

groutr · 2017-02-16T01:12:15Z

docs/source/building/variants.rst

+expressing the compiler to be used. Three new Jinja2 functions are available when
+evaluating ``meta.yaml`` templates:
+
+* ``pin_compatible``: To be used as pin in run and/or test requirements. Takes


Another nitpick: An easier to read format for arguments might be:
pin_compatible(package)

msarahan · 2017-02-16T03:42:14Z

Thanks @groutr. I incorporated your comments in the latest commit and tried to add more examples (an intro section that outlines broad use cases as a guide to deeper features). I also documented pin_downstream and pin_subpackage a little better.

cekees

Added a few questions and one typo correction.

cekees · 2017-02-18T17:12:53Z

docs/source/building/variants.rst

+we need to build binary packages (and any package containing binaries) with
+potentially several variants to support different usage environments. For
+example, using Numpy's C API means that a package must be used with the same
+version of Numpy as runtime that was used at build time.


as runtime -> at runtime

cekees · 2017-02-18T18:35:56Z

docs/source/building/meta-yaml.rst

@@ -439,6 +440,42 @@ emitted during the build process and the variable will remain
 undefined.


+.. _pin_downstream:
+
+Pin downstream


I think this may be a need that doesn't arise in the hashdist design. I guess this is comparable to how hashdist allows upstream packages to specify information to downstream packages if the upstream package is a "build" dependency:

https://github.com/hashdist/hashstack/blob/master/pkgs/boost/boost.yaml#L108

For "run" dependencies the upstream packages resources would be linked into profile by default and any change to the upstream source or configuration would result in a new downstream build artifact hash. I guess the pinning mechanism allows relaxation of the dependency so changes to the upstream package within the constraints specified in the pinning expression would not generate a change to the downstream package.

Yeah, with conda's need to be more flexible, it's important to allow different packages to specify their minimum and maximum requirements. Otherwise we end up with unsatisfiable dependencies very easily. For you, you are building a single unique system (and moreover, building is less of a concern for you than for consumers of conda), so it's less of an issue.

cekees · 2017-02-18T19:04:31Z

docs/source/building/variants.rst

+       run:
+           - python {{ python }}
+
+


How would you use variants to express the different combinations of compilers, processor-specific optimizations, and critical host/vendor packages (e.g. the mpi and blas/lapack libraries that are paired with the compiler toolchain?

I think you already figured this out below. The central idea is "everything is a package" - whether it is files or just simply environment activation scripts that configure compilers. I'll write more in later replies about exactly what goes into the hash.

cekees · 2017-02-18T19:09:11Z

docs/source/building/variants.rst

+  numpy:
+    - 1.11
+    - 1.12
+


Can the variant descriptor be git commits or would we have to enforce that every package is a release?

It's anything that conda recognizes as a version. Git hashes are perfectly fine.

cekees · 2017-02-18T19:17:34Z

docs/source/building/variants.rst

+The information that goes into this hash is currently defined in conda-build's
+metadata.py module; the _get_hash_contents member function. This function
+captures the following information:
+


Does the package (builid) hash come from hashing these entries in the metata file or what they point to? e.g. does it change when I modify the source section or when I modify the source of a package referenced in the section?

It's a few things. Relevant code is at https://github.com/conda/conda-build/pull/1585/files#diff-c86934c27c2f25579540ddb447fa45bdR767

source, requirements, and build sections of the recipe. This captures inputs, as well as exact versions of packages present at build time. The build section has a few options that affect the output, so it's there too. A few fields in the build section are omitted, so that the hashes are more reproducible.

Any other files in the recipe folder (alongside meta.yaml). Everything but meta.yaml is part of the hash.

cekees · 2017-02-18T19:21:18Z

docs/source/building/variants.rst

+compilers that are installed on the system. The analogous compiler packages on
+Windows run any compiler activation scripts and set compiler flags instead of
+actually installing anything.
+


I like this. It seems like the compiler package variants and more generally host/vendor package variants are critical in the HPC environment. Like the windows situation many of these packages would either run 'module load' commands or freeze and enforce the environment changes executed by the 'module' command.

Yeah, once I realized that everything really could be a package, then the problem became a lot more tractable. =)

asmeurer · 2017-03-03T18:46:51Z

docs/source/building/variants.rst

+* CONDA_PERL
+* CONDA_LUA
+
+Legacy CLI flags are also still available. These are sticking around for their


I don't think these should be "legacy", especially if they aren't going to be removed. The environment variables and command line flags are much more useful for an audience that is just building a handful of packages. The above is overkill in those cases.

I agree that legacy seems like the wrong word. It might be worth, somewhere in the larger set of conda build documentation, pointing out this simple way of pinning.

stuarteberg · 2017-03-03T18:50:27Z

docs/source/building/meta-yaml.rst

+  build:
+    pin_downstream:
+      - libstdc++  {{ pin_compatible('g++', 'x') }}
+


Formatting issue: missing .. code-block:: yaml

mwcraig

A short transition guide, perhaps consisting of a few complete before/after meta.yaml files, would be useful (or at least I would find it useful).

Overall I like the separation of the build variants from the recipe.

One question -- will all current recipes that use numpy x.x need to be updated for 3.0, or will it still recognize those pinnings?

mwcraig · 2017-03-14T03:24:04Z

docs/source/building/variants.rst

+
+   requirements:
+       build:
+           - numpy


This may be addressed later, but should this be numpy {{ numpy }} also?

mwcraig · 2017-03-14T03:32:27Z

docs/source/building/variants.rst

+When our two proper yaml config files are combined, ordinarily the recipe-local
+variant would clobber the site-wide variant, yielding ``{'some_trait':
+'pony'}``. However, with the extend_keys entry, we end up with what we've always
+wanted: a dog *and* pony show: ``{'some_trait': ['dog', 'pony'])}``


mwcraig · 2017-03-14T03:37:16Z

docs/source/building/variants.rst

+API, it would actually be better to not pin numpy with ``pin_run_as_build``.
+Pinning it is over-constraining your requirements unnecessarily when you are not
+using Numpy's C API. Instead, we should customize it for each recipe that uses
+numpy.  See also the `Avoiding unnecessary builds`_ section above.


It would be helpful to have a clear statement at the beginning of this section that pin_run_as_build is intended only for those cases when all packages should be pinned (it seems like the most familiar case would be python if you are building python packages).

mwcraig · 2017-03-14T03:38:29Z

docs/source/building/variants.rst

+       build:
+           - numpy {{ numpy }}
+       run:
+           - numpy {{ pin_compatible('numpy', lower_bound='1.10', upper_bound='3.0') }}


This is very nice!

mwcraig · 2017-03-14T03:40:01Z

docs/source/building/variants.rst

+
+Any contents in ``recipe_append.yaml`` will add to the contents of meta.yaml.
+List values will be extended, and string values will be concatenated.
+


What is the use case for appending to recipes?

mwcraig · 2017-03-14T03:41:03Z

docs/source/building/variants.rst

+
+Since conflicts only need to be prevented within one version of a package, we
+think this will be adequate. If you run into hash collisions with this limited
+subspace, please file an issue on the conda-build issue tracker.


Can us add a link to it?

mwcraig · 2017-03-14T03:41:53Z

docs/source/building/variants.rst

+  * ``number``
+  * ``string``
+* any other recipe files, such as bld.bat, build.sh, etc. Every file other than
+  meta.yaml is part of the hash.


Does this include recipe_append and recipe_clobber?

mwcraig · 2017-03-14T03:52:14Z

docs/source/building/variants.rst

+* CONDA_PERL
+* CONDA_LUA
+
+Legacy CLI flags are also still available. These are sticking around for their


I agree that legacy seems like the wrong word. It might be worth, somewhere in the larger set of conda build documentation, pointing out this simple way of pinning.

mwcraig · 2017-03-14T03:53:49Z

docs/source/building/variants.rst

+
+* ``pin_run_as_build``: should be a dictionary. Keys are package names. Values
+  are "pinning expressions" - explained in more detail in `Customizing
+  compatibility`_. This is a generalization of the ``numpy x.x`` spec.


Maybe don't mention numpy x.x here because later on you point out that it is a bad candidate for using with pin_run_as_build.

msarahan · 2017-03-24T03:03:42Z

@mwcraig thanks for the comments. Sorry it has taken me so long to review them. My latest PR should address all of them. Regarding your numpy x.x question - it should work in conda-build 3.0, but you should see a deprecation warning. It will be removed in some future version (4.0 or above).

timsnyder · 2017-04-16T15:28:06Z

docs/source/building/variants.rst

+   requirements:
+       build:
+           - python
+           - numpy


Does this example need to use the numpy variant value in the build requirements? I don't see how the variants do anything if you're missing it in the build requirements. My guess is that this example would build the package twice with the same version of numpy being used.

On this note, does conda-build at least warn you when you've put something in conda_build_config.yaml that isn't used in the rendering of the recipe?

Nevermind, after reading Avoiding unncessary builds again, I think I understand what you're trying to convey. I'm going to submit a PR to your branch to help clarify this example.

@msarahan I created the PR on your fork's branch msarahan#1

Thanks! Merged.

Make it clear that you would be building two different numpy dependent packages by giving them different names and add a little bit more prose to explain the first example that doesn't pin numpy.

timsnyder · 2017-04-18T14:13:29Z

docs/source/building/variants.rst

+  outputs a single compiler package name. For example, this could be used to
+  compile outputs targeting x86_64 and arm in one recipe, with a variant.
+
+There are default "native" compilers that and runtimes that are used when no


You can drop "that" after compilers.

This is the only occurrence of "runtimes" and I assume it is what you meant by "detailed further" in .. _special_variant_keys.

It would be helpful to put runtimes in bactics so that it stands out like a key from conda_build_config.yaml when following the link from .. _special_variant_keys.

It would be nice to have a definition of what you consider runtimes and how they are different than compilers.

Thanks. This is a remnant of the time before pin_depends or run_exports. I'll certainly add a better explanation, too.

clarify numpy variant example

183amir

Wow! Such amazing new features. We have been longing to write our CI with conda recipes but this was missing to make the migration feasible.
I added a few minor comments.

183amir · 2017-06-08T20:43:12Z

docs/source/building/variants.rst

+             - {{ pin_subpackage('libxgboost', exact=True) }}
+             - python  {{ python }}
+
+       - name: py-xgboost


I think you meant to say r-xgboost here.

183amir · 2017-06-08T20:50:21Z

docs/source/building/variants.rst

+
+.. code-block:: python
+
+    variants = [{'numpy': ['1.10', '1.11'], 'ignore_version': ['numpy]}]


The last numpy string here does not seem to close properly.

msarahan · 2017-06-08T22:50:36Z

Thanks for the review @183amir. Looking forward to seeing the fun things people do with these new capabilities.

msarahan · 2017-06-09T19:26:36Z

@cio-docs please review this when you can. I will be tagging the conda-build 3 release by Monday, and these docs are very important for people who start using the new version.

kerrywatson1 · 2017-06-09T19:46:09Z

I chatted with @msarahan re outside editors have just finished editing conda docs and are in final review. @cio-docs will ping @msarahan as soon as we receive the edited docs back from outside team, and we'll incorporate these changes into the edited docs.

small style edits to meta-yaml

electronwill · 2017-06-22T20:19:38Z

docs/source/building/variants.rst

+
+.. _pinning_expressions:
+
+Pinning expressions


@msarahan The code in the first example only seems to mention 1.11 but the comment mentions 1.11.2. Is this correct?

Yes. Added a small blurb explaining why.

I understand much better now. Thanks!

electronwill · 2017-06-22T20:22:11Z

docs/source/building/variants.rst

+   requirements. The value should be a dictionary with up to 4 keys:
+   ``min_pin``, ``max_pin``, ``lower_bound``, ``upper_bound``. The first two are
+   pinning expressions. The latter two are version numbers, overriding detection
+   of current version. (defaulting to None).


@msarahan what defaults to None? All four keys, the latter two, or the dictionary that contains the four keys?

I removed this comment and instead added a reference to a lower section where that function's default arguments are shown.

Perfect. Thanks!

…onda-docs into document_conda_build_variants

review & edit variants.rst

electronwill · 2017-06-22T21:37:35Z

The xgboost example and the clarity of its explanation are excellent! I also liked the dog and pony show example to demonstrate extend_keys. If none of my edits look like a problem to you, I'll merge this when the CI passes.

github-actions · 2022-04-15T06:02:57Z

Hi there, thank you for your contribution!

This pull request has been automatically locked because it has not had recent activity after being closed.

Please open a new issue or pull request if needed.

Thanks!

msarahan mentioned this pull request Jan 29, 2017

WIP Build customization conda/conda-build#1585

Merged

12 tasks

groutr reviewed Feb 7, 2017

View reviewed changes

msarahan force-pushed the document_conda_build_variants branch from 7e98d45 to 5579dee Compare February 13, 2017 15:42

groutr reviewed Feb 16, 2017

View reviewed changes

msarahan added 9 commits February 15, 2017 21:17

first draft of conda-build variants docs

390c7c5

document hash naming for variant output

cfa9004

try to fix links

3148277

document pinning expressions

fb6fabb

document conda inspect hash-inputs

b72d762

document pinning expressions and avoiding unnecessary builds

941df00

more docs on variant pinning

eb5cf0c

document pin_downstream

4856526

fix missing language arg to code block

58804e2

msarahan force-pushed the document_conda_build_variants branch from 5579dee to 1f582c5 Compare February 16, 2017 03:41

@groutr revisions. add pin_subpackages example.

4903252

msarahan force-pushed the document_conda_build_variants branch from 1f582c5 to 4903252 Compare February 16, 2017 04:11

cekees reviewed Feb 18, 2017

View reviewed changes

change pins to separate min_pin and max_pin

6d44b26

msarahan mentioned this pull request Feb 24, 2017

--numpy cannot take 'all' or be passed multiple times conda/conda-build#1608

Closed

asmeurer reviewed Mar 3, 2017

View reviewed changes

stuarteberg reviewed Mar 3, 2017

View reviewed changes

msarahan mentioned this pull request Mar 13, 2017

Problem: Test stage failed in 2.1.6 under windows conda/conda-build#1835

Closed

mwcraig reviewed Mar 14, 2017

View reviewed changes

msarahan mentioned this pull request Mar 15, 2017

Questions about build customization and compiler specification conda/conda-build#1841

Closed

more detail to conda-build variants in response to @mwcraig

449aced

timsnyder reviewed Apr 16, 2017

View reviewed changes

clarify numpy variant example

3acab8d

Make it clear that you would be building two different numpy dependent packages by giving them different names and add a little bit more prose to explain the first example that doesn't pin numpy.

timsnyder reviewed Apr 18, 2017

View reviewed changes

msarahan added 2 commits April 18, 2017 10:47

Merge pull request #1 from timsnyder/document_conda_build_variants

1ac1477

clarify numpy variant example

add reproducibility note

0e6ed1c

jakirkham mentioned this pull request May 4, 2017

Porting R channel to conda-forge conda-forge/staged-recipes#2009

Closed

msarahan mentioned this pull request May 8, 2017

How to specialize a package's runtime requirements during a build? conda/conda-build#1993

Closed

pin_compatible now returns package name with pin/constraint

2d0ede2

msarahan mentioned this pull request May 22, 2017

Specifying multiple Python versions fails to build conda/conda-build#2049

Closed

183amir reviewed Jun 8, 2017

View reviewed changes

msarahan force-pushed the document_conda_build_variants branch 2 times, most recently from b5f5df4 to 7ae4481 Compare June 9, 2017 18:58

document ignore_run_exports, fix some errors

610c23f

msarahan force-pushed the document_conda_build_variants branch from 7ae4481 to 610c23f Compare June 9, 2017 19:22

small style edits to meta-yaml

2fea343

small style edits to meta-yaml

electronwill reviewed Jun 22, 2017

View reviewed changes

msarahan and others added 3 commits June 22, 2017 16:10

clarify pinning expressions

920e3a6

Merge branch 'document_conda_build_variants' of github.com:msarahan/c…

3b65082

…onda-docs into document_conda_build_variants

review & edit variants.rst

fc646a5

review & edit variants.rst

electronwill mentioned this pull request Jun 22, 2017

document new noarch #416

Merged

electronwill merged commit 8a0e0a1 into conda:master Jun 22, 2017

timsnyder mentioned this pull request Sep 4, 2019

Support rendered docs for conda-canary conda/conda#9200

Closed

github-actions bot added the locked [bot] locked due to inactivity label Apr 15, 2022

github-actions bot locked as resolved and limited conversation to collaborators Apr 15, 2022

		u'source': {u'path': u'/Users/msarahan/code/conda-build/tests/test-recipes/split-packages/_rm_rf_stays_within_prefix'}}}


		Extra Jinja2 functions


		Any contents in ``recipe_append.yaml`` will add to the contents of meta.yaml.
		List values will be extended, and string values will be concatenated.


		.. code-block:: python

		variants = [{'numpy': ['1.10', '1.11'], 'ignore_version': ['numpy]}]



		Extended keys
		-------------

first draft of conda-build variants docs #414

first draft of conda-build variants docs #414

Conversation

msarahan commented Jan 29, 2017

electronwill commented Jan 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msarahan commented Feb 13, 2017

kerrywatson1 commented Feb 13, 2017 • edited Loading

groutr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msarahan commented Feb 16, 2017

cekees left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwcraig left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msarahan commented Mar 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

183amir left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msarahan commented Jun 8, 2017

msarahan commented Jun 9, 2017

kerrywatson1 commented Jun 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

electronwill commented Jun 22, 2017

github-actions bot commented Apr 15, 2022

kerrywatson1 commented Feb 13, 2017 •

edited

Loading