split off recipe render from build; recurse to find meta.yaml #908

msarahan · 2016-04-29T18:11:40Z

Closes #887

This is the start of splitting off recipe rendering from building. I am trying to avoid duplicating code, so I'm splitting the existing code off into other modules.

Right now, it only prints the processed metadata, not a yaml file. That's easy enough to write as an addon option later - I'm expecting that the raw dictionary will be the most useful thing to people, rather than going back to some file on disk. My idea is that the build module will just call functions from this file to render each recipe's meta.yaml.

This works right now, with a trivial test of a recipe from conda-forge (zlib). Original recipe at https://github.com/conda-forge/zlib-feedstock, which I have cloned as a submodule into the packages/zlib folder:

$ conda render packages/zlib
{u'about': {u'home': u'http://zlib.net/',
            u'license': u'zlib (http://zlib.net/zlib_license.html)',
            u'license_family': u'Other',
            u'license_file': u'license.txt',
            u'summary': u'Massively spiffy yet delicately unobtrusive compression library'},
 u'build': {u'features': u''},
 u'extra': {u'recipe-maintainers': [u'groutr', u'msarahan', u'ocefpaf']},
 u'package': {u'name': u'zlib', u'version': u'1.2.8'},
 u'requirements': {u'build': [u'cmake']},
 u'source': {u'fn': u'zlib-1.2.8.tar.gz',
             u'sha256': u'36658cb768a54c1d4dec43c3116c27ed893e88b02ecfcb44f2166f9c0b7f2a0d',
             u'url': u'http://zlib.net/zlib-1.2.8.tar.gz'},
 u'test': {u'commands': [u'test -f ${PREFIX}/lib/libz.a']}}

This change also makes conda-build recursively search for meta.yaml in any folder you provide, so it should make conda build more directly compatible with the conda-forge layout of having the recipe one level deeper.

I'll be cleaning this up and coming up with test cases - please kick the tires and point out any details I should pay special attention to.

CC @jakirkham @stuarteberg @kalefranz @pelson @groutr

msarahan · 2016-05-01T23:11:02Z

This now also renders YAML, keeping the keys in the order we expect:

$ conda render ~/Documents/code/anaconda/packages/zlib --yaml
package:
    name: zlib
    version: 1.2.8
source:
    fn: zlib-1.2.8.tar.gz
    sha256: 36658cb768a54c1d4dec43c3116c27ed893e88b02ecfcb44f2166f9c0b7f2a0d
    url: http://zlib.net/zlib-1.2.8.tar.gz
build:
    features: ''
requirements:
    build:
        - cmake
test:
    commands:
        - test -f ${PREFIX}/lib/libz.a
extra:
    recipe-maintainers:
        - groutr
        - msarahan
        - ocefpaf

jakirkham · 2016-05-01T23:19:48Z

Could we tweak it to space out the main sections?

jakirkham · 2016-05-01T23:22:13Z

Also, just to clarify, will this proceed through the download (but not build phase)? What is the option to enable/disable this in case people don't want to proceed that far?

msarahan · 2016-05-01T23:32:51Z

By default, it will not download the source. You can pass the -s/--source option to do so. If it can't fill in fields without the source, it will throw errors (Jinja strict undefined stuff that @stuarteberg came up with).

You can also pass the -o/--output flag to print the output filename (as conda build did/does), rather than the meta.yaml data.

Sorry, but adding spaces is not something I want to fight PyYAML to do. It was enough fun getting it to where it is now. If you want to tackle that, I think the right place would be https://github.com/conda/conda-build/pull/908/files#diff-154513cee3e5765ecebd315978958f40R212

stuarteberg · 2016-05-02T18:31:12Z

Looks good to me, but I have a couple of questions:

conda build --output first attempts parse meta.yaml without downloading the source, but then it automatically downloads the source if necessary. But as you noted above, the new render command doesn't do that:

By default, it will not download the source. You can pass the -s/--source option to do so. If it can't fill in fields without the source, it will throw errors

So in other words, the user is expected to interpret the error and add --source when necessary? I can see why we wouldn't want to preemptively download the source (it won't be necessary for all recipes), but in the case of an error, isn't it nicer to just go ahead and give it a try on the user's behalf, as we do for conda build --output?

Also, I have an unrelated question about the output format. (If this PR is still WIP, then this question may be premature...)

The current version of your PR writes the fully-rendered YAML to stdout. Are you planning to change that before this PR is merged? It seems like a file output would be more desirable because misc. functions throughout the conda code base write various status messages to stdout, which will pollute the output with non-YAML junk.

msarahan · 2016-05-02T18:39:28Z

@stuarteberg all good ideas. This is why I ping you. I also had the thought that there need not be a --yaml option - who wants output in any other format? If they want the metadata itself as a python object, they can call the render_recipe function themselves. I will replace it with a -f/--file option that writes to a file (the argument to -f/--file), but keep the default as writing to stdout. Sound good?

As for downloading the source: you're right. I like the transparent failover better than assuming interpretation of error messages.

stuarteberg · 2016-05-02T19:00:31Z

Sounds good!

codecov-io · 2016-05-02T20:53:48Z

Current coverage is 100%

Merging #908 into master will decrease coverage by +90.1%

@@            master   #908   diff @@
=====================================
  Files           39      0     -39   
  Lines         5136      0   -5136   
  Methods          0      0           
  Messages         0      0           
  Branches         0      0           
=====================================
- Hits           509      0    -509   
+ Misses        4627      0   -4627   
  Partials         0      0

36 files (not in diff) in conda_build were deleted. more

Powered by Codecov. Last updated by 2db31bb...a67b79d

jakirkham · 2016-05-03T13:51:26Z

I think I missed the recurse portion when skimming this quickly before. Would this be useful for feedstocks?

msarahan · 2016-05-03T13:57:43Z

@jakirkham Yes, that was my motivation. I can now replace things in the main repo with submodules to feedstocks. Closing the loop, little by little.

I still need to tie in rendering the recipe for tools that are expecting static content, but it's progress.

stuarteberg · 2016-05-03T14:01:06Z

This change also makes conda-build recursively search for meta.yaml in any folder you provide

I've been contemplating a kinda similar feature, but haven't taken the time for a PR yet: What do you guys think about a RECIPE_PATH environment variable that conda-build uses to search for recipes it can't find?

RECIPE_PATH=/path/to/some-dir:/path/to/some-other-dir conda build mypackage

... if that's off-topic I can raise it in a different PR.

jakirkham · 2016-05-03T14:05:04Z

I love it. Sorry, I got distracted by the static rendering. Lots of great stuff in this PR. 👍

I can now replace things in the main repo with submodules to feedstocks. Closing the loop, little by little.

👍 Do you want help doing this after the release? It sounds like we are entering phase 3.

cc @pelson

I still need to tie in rendering the recipe for tools that are expecting static content, but it's progress.

Are you meaning like a public API or are there other tools that you guys have that will need this?

msarahan · 2016-05-03T14:05:24Z

@stuarteberg That's not a bad idea. I don't think I would use it, but I'm sure others would. I have many folders of recipes, with very high degrees of overlap. Having this feature would confuse me a lot on which collection I was getting. I do think it deserves its own PR.

msarahan · 2016-05-03T19:05:23Z

@jakirkham

Do you want help doing this after the release? It sounds like we are entering phase 3.

Help removing recipes from conda-recipes would be great. I will ask again about the CLA - if it would be necessary for this. Internal people have to do the internal repo, obviously.

Are you meaning like a public API or are there other tools that you guys have that will need this?

Most just the legacy internal build system. It does a lot of moving packages around and examining metadata, so I need to make sure that wherever it looks, it finds fully rendered recipes.

jakirkham · 2016-05-03T20:18:04Z

Help removing recipes from conda-recipes would be great. I will ask again about the CLA - if it would be necessary for this. Internal people have to do the internal repo, obviously.

I mean if you guys would rather merge PRs that is fine to. I'm not picky here. Whatever works really. I'm just asking so as to know how we and others at conda-forge should start to act in this regard.

Most just the legacy internal build system. It does a lot of moving packages around and examining metadata, so I need to make sure that wherever it looks, it finds fully rendered recipes.

I see. Thanks for clarifying.

jakirkham · 2016-05-03T20:19:36Z

So, I'm thinking this might be part of our solution to automating recipe generation to cut down time spent on minutiae. See this comment for some thoughts.

jakirkham · 2016-05-03T20:37:59Z

So, I tried to build this example package, but it doesn't quite work as it doesn't like conda.yaml. So, I renamed the file to meta.yaml and was able to build it with conda build. However, I can't get conda render to work. Basically, this is complaining there is no source defined though I'm sure it will complain about others.

msarahan · 2016-05-03T20:48:21Z

so the issue with conda render is that it expects the few required fields too early - before the first template pass. I'm going to see about changing that.

… from yaml output

msarahan · 2016-05-03T20:58:43Z

@jakirkham should work better now.

jakirkham · 2016-05-03T20:59:59Z

Thanks. I'll give it another try.

jakirkham · 2016-05-03T21:01:39Z

Works. Thanks @msarahan.

package:
    name: test_package
    version: 1.1.0
build:
    entry_points: ''
    number: '1'
    script:
        - cd $RECIPE_DIR
        - $PYTHON setup.py install --single-version-externally-managed --record=record.txt
requirements:
    build:
        - python
        - setuptools
    run:
        - python
        - Flask
        - werkzeug
test:
    imports: ''
    requires: ''

jakirkham · 2016-05-03T21:27:56Z

So, I tried basically using this on the setup.py from pandas, but it got a little picky about having things like pytz >=2011k instead of pytz >= 2011k. Now we can and probably should clean these sorts of whitespace things in our rendering process (I added my own hack for now), but maybe it could be a little more relaxed about this sort of stuff for rendering as it can be tricky to catch all of that stuff on the first pass and it can be cleaned up right after.

jakirkham · 2016-05-03T21:35:08Z

Kind of weird that the build number keeps getting turned into a string. Not sure where that is coming from.

jakirkham · 2016-05-03T21:52:05Z

When you are doing this rendering @msarahan, are you including the about section? It seems to be missing in many cases.

jakirkham · 2016-05-03T21:52:41Z

conda_build/main_render.py

+# Next bit of stuff is to support YAML output in the order we expect.
+# http://stackoverflow.com/a/17310199/1170370
+class MetaYaml(dict):
+    fields = ["package", "source", "build", "requirements", "test", "extra"]


Could we please add about here?

Should come after test, but before extra.

Good catch. Oversight. Will add.

No worries. With this change, things appear to work for me.

jakirkham · 2016-05-03T21:53:23Z

When you are doing this rendering @msarahan, are you including the about section? It seems to be missing in many cases.

Nevermind. Answered my own question. It would be great if we could add about here.

jakirkham · 2016-05-03T22:08:09Z

So, I made this variant based on the included template, which looks like this still kind of WIP, but it is coming along nicely. From that, I can generate the pandas recipe as long as it is included in the source directory (next to setup.py).

Would be good to figure out how we can work this into more of a conda skeleton pypi operation. That way, we can get checksums, fill out the download url, and such.

Also, we could probably get the recipe-maintainers filled out with a user's handle given a little more detective work.

The license unfortunately is always a mess as no one seems to think it worthwhile to specify the license version in the license field. We might be able to get around this by using classifiers if they added one specifying the license version. Though, in the case of pandas, they don't. Maybe this is the tedious step we do manually. I guess it is not so bad as we really don't want to get this wrong and we have automated most everything else.

jakirkham · 2016-05-04T14:47:48Z

Added this issue ( #914 ) to try and refactor conda skeleton and expose that functionality to meta.yaml in a straightforward manner. We could push the existing approach the rest of the way with this change I think.

msarahan · 2016-05-05T17:51:56Z

Discussed with @kalefranz. Long-term, need to move logic out of CLI files (main_*). OK for now, though. Merging.

github-actions · 2022-05-11T06:06:54Z

Hi there, thank you for your contribution!

This pull request has been automatically locked because it has not had recent activity after being closed.

Please open a new issue or pull request if needed.

Thanks!

start splitting off recipe render from build

cba710d

msarahan added the in_progress [deprecated] use milestones/project boards label Apr 29, 2016

msarahan added 5 commits April 29, 2016 18:57

add option to output rendered recipe as yaml

5600f51

fix build tests; add nested recipe build test

72ac7ff

separate more of conda-build into conda-render (filename output)

2425936

fix pyflakes bugs, remove duplicated recursive glob function

7233673

squelch pyflakes missing unicode define

ca69c7d

msarahan changed the title ~~WIP: start splitting off recipe render from build; recurse to find meta.yaml~~ split off recipe render from build; recurse to find meta.yaml Apr 30, 2016

msarahan added the 4_Needs_Review [deprecated] use milestones/project boards label Apr 30, 2016

output yaml by default. add -f (file) option. Download by default.

0448b2e

pyflakes, no pprint used

ddb852a

msarahan mentioned this pull request May 3, 2016

add pandas conda-forge/staged-recipes#522

Merged

jakirkham mentioned this pull request May 3, 2016

Automating boiler plate conda-forge/staged-recipes#535

Closed

msarahan added 2 commits May 3, 2016 15:49

look for conda.yaml files and meta.yaml files

1c16171

revert strict jinja in MetaData constructor; omit non-existent fields…

9f5185a

… from yaml output

jakirkham reviewed May 3, 2016
View reviewed changes

add about section in rendering

1f3ef2b

jakirkham mentioned this pull request May 4, 2016

More pluggable/flexible conda skeleton (aka Refactoring to use #908) #914

Closed

msarahan merged commit c915b6e into conda:master May 5, 2016

msarahan removed the in_progress [deprecated] use milestones/project boards label May 5, 2016

This was referenced May 5, 2016

use conda_build.metadata for checking metadata #730

Closed

remove info/recipe.json from conda package #781

Merged

Default to conda.recipe when calling conda build #327

Closed

msarahan removed the 4_Needs_Review [deprecated] use milestones/project boards label May 5, 2016

This was referenced May 8, 2016

WIP: Build customization #848

Closed

add test for environ.get access to jinja vars #926

Closed

msarahan mentioned this pull request Jul 23, 2016

Recipe customization / variants / better pinning #1142

Closed

msarahan deleted the add_conda_render branch October 12, 2016 15:31

jakirkham mentioned this pull request Apr 16, 2018

"conda skeleton pypi" Meta-Mega-Issue conda-forge/conda-forge.github.io#565

Open

github-actions bot added the locked [bot] locked due to inactivity label May 11, 2022

github-actions bot locked as resolved and limited conversation to collaborators May 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

split off recipe render from build; recurse to find meta.yaml #908

split off recipe render from build; recurse to find meta.yaml #908

msarahan commented Apr 29, 2016 •

edited

msarahan commented May 1, 2016

jakirkham commented May 1, 2016

jakirkham commented May 1, 2016

msarahan commented May 1, 2016

stuarteberg commented May 2, 2016

msarahan commented May 2, 2016

stuarteberg commented May 2, 2016

codecov-io commented May 2, 2016

jakirkham commented May 3, 2016

msarahan commented May 3, 2016

stuarteberg commented May 3, 2016

jakirkham commented May 3, 2016

msarahan commented May 3, 2016

msarahan commented May 3, 2016

jakirkham commented May 3, 2016 •

edited

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016 •

edited

msarahan commented May 3, 2016

msarahan commented May 3, 2016

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016

jakirkham May 3, 2016

jakirkham May 3, 2016

msarahan May 3, 2016

jakirkham May 3, 2016 •

edited

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016 •

edited

jakirkham commented May 4, 2016

msarahan commented May 5, 2016

github-actions bot commented May 11, 2022

split off recipe render from build; recurse to find meta.yaml #908

split off recipe render from build; recurse to find meta.yaml #908

Conversation

msarahan commented Apr 29, 2016 • edited

msarahan commented May 1, 2016

jakirkham commented May 1, 2016

jakirkham commented May 1, 2016

msarahan commented May 1, 2016

stuarteberg commented May 2, 2016

msarahan commented May 2, 2016

stuarteberg commented May 2, 2016

codecov-io commented May 2, 2016

Current coverage is 100%

jakirkham commented May 3, 2016

msarahan commented May 3, 2016

stuarteberg commented May 3, 2016

jakirkham commented May 3, 2016

msarahan commented May 3, 2016

msarahan commented May 3, 2016

jakirkham commented May 3, 2016 • edited

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016 • edited

msarahan commented May 3, 2016

msarahan commented May 3, 2016

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016

jakirkham May 3, 2016

Choose a reason for hiding this comment

jakirkham May 3, 2016

Choose a reason for hiding this comment

msarahan May 3, 2016

Choose a reason for hiding this comment

jakirkham May 3, 2016 • edited

Choose a reason for hiding this comment

jakirkham commented May 3, 2016

jakirkham commented May 3, 2016 • edited

jakirkham commented May 4, 2016

msarahan commented May 5, 2016

github-actions bot commented May 11, 2022

msarahan commented Apr 29, 2016 •

edited

jakirkham commented May 3, 2016 •

edited

jakirkham commented May 3, 2016 •

edited

jakirkham May 3, 2016 •

edited

jakirkham commented May 3, 2016 •

edited