Automating boiler plate

[jakirkham]

There is quite a bit of boiler plate that goes into recipes. It is a pain making sure it is right for bother reviewers and reviewees (?). We should move towards automating this by using all means at our disposal (e.g. jinja, scripts, metapackages, etc.). The hope is that most of this (if not all) of this could be removed. What I would like us to do here is figure out what and how. One step in this direction would be taking a hard look at this script by @183amir, which converts PyPI packages into conformant recipes.

cc @msarahan @ocefpaf @pelson

[183amir]

@jakirkham I don't have an automation script that converts pypi packages into conformant recipes.
I have some codes lying around that would ease these process though.

[jakirkham]

Ah, automation was a poor choice of word there. I will correct it. I just meant you can point it to a PyPI package and generate a conformant recipe.

[183amir]

I think conda skeleton is already doing a good job on that and we can apply several checks and modifications to its output so that it meets conda-forge standards.

[183amir]

Also, a while ago I opened an issue here: conda-forge/conda-smithy#149 to discuss common functionality to manage feedstocks.

[jakirkham]

Also, a while ago I opened an issue here: conda-forge/conda-smithy#149 to discuss common functionality to manage feedstocks.

Yeah, @pelson has been out, but I think he is back soon (today or tomorrow).

[jakirkham]

I think conda skeleton is already doing a good job on that and we can apply several checks and modifications to its output so that it meets conda-forge standards.

I also agree with that as an option.

[jakirkham]

Another option might be the little known jinja templates like this one for setuptools. We could extend this like they do with this package. If we can leverage new functionality that @msarahan is adding in this PR ( conda/conda-build#908 ), we can generate a good to go recipe quite quickly. This could be extended for different language cases too.

[jakirkham]

So was able to cobble something together yesterday using the aforementioned template with some tweaks @msarahan's PR that will generate a recipe from a source directory. See this comment for details. It would be better still if we had a use_pypi function in conda-build that acts like use_setuptools this combined with @msarahan's work could create a more general conda skeleton pypi that we could customize the end result for.

[msarahan]

Thanks for working on this. It will definitely help. I think we also
really need to address automated cleanup of recipes, though, because a lot
of what we do is just pulling in existing recipes.

On Wed, May 4, 2016 at 9:22 AM, jakirkham notifications@github.com wrote:

So was able to cobble something together yesterday using the
aforementioned template with some tweaks @msarahan
https://github.com/msarahan's PR that will generate a recipe from a
source directory. See this comment
conda/conda-build#908 (comment)
for details. It would be better still if we had a use_pypi function in
conda-build that acts like use_setuptools this combined with @msarahan
https://github.com/msarahan's work could create a more general conda
skeleton pypi that we could customize the end result for.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#535 (comment)

[jakirkham]

Yeah, that is true. We need conda smithy rerender-lint or similar.

[jakirkham]

Also take a look at this issue ( conda/conda-build#914 ). I think we could clean up conda-build a bit to use what you wrote and get recipe generation into a custom template as a benefit.

[jakirkham]

More boiler plate that needs to live somewhere is CI parallelism.

[183amir]

off topic question. Do we have a place that explains our recipe guidelines? Things that our linter doesn't catch.

[jakirkham]

We have this issue ( #139 ), which is sort of a grab bag of everything. We are working on making it official guidelines in this PR ( conda-forge/conda-forge.github.io#95 ). Might be somewhere to pull things from or adding thing too. The document was more a rally of the troops type document to encourage the large scale addition of things from common Continuum repos to conda-forge. Though there is some stuff about style and workflows too. If you would like to take a look and provide feedback, please do. A fresh set of eyes would be nice.

[jakirkham]

Is there a good way to have some environment variables set when a package is installed or the environment is activated? I'm thinking of writing a meta compiler package so we can avoid some cruft in our recipes.

[scopatz]

I was also looking into this today and it doesn't seem like there is :(

[jakirkham]

Yeah, I didn't think so either, but maybe there is something that is undocumented.

[msarahan]

package installed: post-link scripts
http://conda.pydata.org/docs/building/build-scripts.html
env activated: "saved env vars"
http://conda.pydata.org/docs/using/envs.html#saved-environment-variables

On Mon, May 9, 2016 at 9:09 PM jakirkham notifications@github.com wrote:

Yeah, I didn't think so either, but maybe there is something that is
undocumented.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#535 (comment)

[jakirkham]

Are those actually sourced though?

[jakirkham]

Ah, sorry, I see now. Thanks. I didn't know about that.

[msarahan]

be very careful with both of them. Conda has very little (read: None) error handling for these. We have had major problems when they don't work, for example, because packages aren't installed in the order you expect (topological/dependency sort vs. alphabetical). People talk about them practically as expletives. Still, if you need to do things at install time or at activation time, they are good tools.

[jakirkham]

The main point here is to configure clang so we aren't doing the copy-pasta dance everytime someone wants C++11 support.

[jakirkham]

post-link scripts don't get sourced though, right?

[msarahan]

No, they are run, but not sourced. This kind of stinks for the build application, because the build environment isn't truly activated.

We could change, that, though. I think doing so would be a nice consistency improvement.

[jakirkham]

Well, I know this still won't work right without some changes, but maybe you can take a look at what I got and we can discuss the appropriate changes upstream.

[jakirkham]

Here is a PR ( #578 ) with the recipe. FWIW, this will work if the environment is re-activated after install. I have tried this with a real build yet. Though some tests are in order.

[jakirkham]

Much of the discussion here is stale. conda skeleton and conda-forge have converged a bit on best practices as well. Automated updated PRs have lightened maintenance burden a bit. There is still more room for work, but this issue IMHO can be closed. So will close it. Feel free to reopen if you see something that still needs to be done.