conda build --output doesn't expand GIT_xxx environment variables anymore #754

pitrou · 2016-02-02T19:02:03Z

This is a regression in 1.19.0. When running conda build --no-output, the GIT_DESCRIBE_TAG and GIT_DESCRIBE_NUMBER variables are not expanded anymore in the meta.yaml, therefore the generated filename is wrong.

The text was updated successfully, but these errors were encountered:

pitrou · 2016-02-02T19:02:16Z

Ping @jakirkham @meawoppl

jakirkham · 2016-02-02T19:03:02Z

Thanks for doing this. The flag is --output, correct?

jakirkham · 2016-02-02T19:06:36Z

So, the package should have a path like so <CONDA_ROOT>/conda-bld/<OS>-<ARCH>/<PACKAGE>-<VERSION>-<BUILDSTR>_<FEATURE>_<BUILDNUM>.tar.bz2. Using this style of path directly should at least provide you a workaround for finding the package.

jakirkham · 2016-02-02T19:08:41Z

@meawoppl, so I don't know that there is a way to specify the path, but all the paths look something like the one above.

ericdill · 2016-02-02T19:10:14Z

@pitrou what version of conda-build did this work in? conda-build /path/to/recipe --output does not fill in the GIT_* vars for 1.17, 1.18, 1.18.1 either

pitrou · 2016-02-02T19:11:20Z

On our CI systems this worked with 1.18.x, it seems. Perhaps there are some other factors to the variability? (perhaps you need jinja2 installed?)

jakirkham · 2016-02-02T19:15:14Z

Was it 1.18.2, @pitrou? Did you test that as well, @ericdill?

pitrou · 2016-02-02T19:19:12Z

1.18.2 indeed, apparently.

pitrou · 2016-02-02T19:20:03Z

By the way, I think jinja2 should be hard dependency of conda-build. It's too easy to forget installing it and get weird variations.

ericdill · 2016-02-02T19:21:43Z

1.18.2 totally fails with GIT_* vars.

$ conda build --version
conda-build 1.18.2

$ conda-build recipes/databroker/ --output
Error: Failed to parse jinja template in /home/edill/dev/conda/staged-recipes-dev/recipes/databroker/meta.yaml:
'dict object' has no attribute 'GIT_DESCRIBE_TAG'

$ cat recipes/databroker/meta.yaml 
package:
  name: databroker
  version: {{ environ['GIT_DESCRIBE_TAG'] }}.post{{ environ['GIT_DESCRIBE_NUMBER'] }}
<snip>

1.17-1.18.1 and 1.19 just use whatever the GIT_* vars were for the last build in. So if you remove the conda-build/ directory, the GIT_* vars will just fail to be present. e.g.,

$ conda build --version
conda-build 1.19.0

$ conda build recipes/databroker/ --output
/home/edill/miniconda/conda-bld/linux-64/databroker-.post-_py35.tar.bz2

----

$ conda build --version
conda-build 1.17.0

$ conda build recipes/databroker/ --output
/home/edill/miniconda/conda-bld/linux-64/databroker-.post-_py35.tar.bz2

-----

$ conda build --version
conda-build 1.18.0

$ conda build recipes/databroker/ --output
/home/edill/miniconda/conda-bld/linux-64/databroker-.post-_py35.tar.bz2

-----

$ conda build --version
conda-build 1.18.1

$ conda build recipes/databroker/ --output
/home/edill/miniconda/conda-bld/linux-64/databroker-.post-_py35.tar.bz2

@pitrou jinja2 is now a runtime dep: https://github.com/conda/conda-build/blob/master/conda_build.recipe/meta.yaml#L22

ericdill · 2016-02-02T19:31:35Z

No, I'm sorry, I was mistaken. The behavior from 1.17-1.18.1 was to use the GIT_* variables from the last built package. The GIT_* variables were just broken in 1.18.2. Now GIT_* variables do not exist in 1.19.0 for the --output flag

jakirkham · 2016-02-02T19:32:57Z

By the way, I think jinja2 should be hard dependency of conda-build.

Yeah, that happened. I can't remember where, but I'll find the link. Here is the closest thing to it. ( #613 )

jakirkham · 2016-02-02T19:33:53Z

@ericdill, I think the failure you are seeing is because jinja2 is not installed. Could you please try that and see if the error persists?

ericdill · 2016-02-02T19:34:14Z

@jakirkham I do have jinja2 installed.

jakirkham · 2016-02-02T19:38:14Z

Ah, sorry, my mistake.

jakirkham · 2016-02-02T19:38:26Z

I think the problem has something to do with some stronger constraints on how jinja templates work or at least that is my new guess. It may also be related to how many passes of the recipe have occurred before this is outputted. @stuarteberg, any thoughts on the problem seen here?

ericdill · 2016-02-02T19:51:03Z

@jakirkham @pitrou Sorry to derail this thread. I was only trying to point out that conda-build --output has not used the git version info for any of the last few point releases.

@jakirkham If you just pinged @stuarteberg to try to debug why this is failing on 1.18.2 for my sake, I am not particularly concerned with that issue, I can use a different version than 1.18.2.

I did suggest a fix to get access to the GIT_* vars for the conda build --output flag in #750, but I'm sure there is a better approach.

pitrou · 2016-02-02T19:56:36Z

Well, we seem to have worked around the issue by downgrading to 1.18.1 on our CI servers.

jakirkham · 2016-02-02T20:12:20Z

If you just pinged @stuarteberg to try to debug why this is failing on 1.18.2 for my sake, I am not particularly concerned with that issue, I can use a different version than 1.18.2.

I think I know the reason and I think he does too. :) However, that isn't really why I wanted his thoughts. I think he might have a better idea of where this needs to fit in the now 2 pass jinja framework so that was what I was hoping he might be able to point out.

stuarteberg · 2016-02-02T20:21:54Z

Well, we seem to have worked around the issue by downgrading to 1.18.1 on our CI servers.

I strongly suspect this "fix" is not what you want. Can you verify by hand that the output of conda build --output is correctly resolving GIT_ variables? Unless you're accidentally very lucky, I bet they're wrong. As @ericdill correctly points out:

The behavior from 1.17-1.18.1 was to use the GIT_* variables from the last built package.

That is, if you build a package, and then you run conda build --output for a different package, the git information is left over from the first package. The conda build --output never worked correctly for packages that used GIT_ variables in their build strings.

Consider these two recipes:

# requests-old/meta.yaml
package:
  name: requests-old
  version: "1.0.0"

source:
  git_url: https://github.com/kennethreitz/requests
  git_tag: v1.0.0

package:
  name: requests-new
  version: "2.9.1"

source:
  git_url: https://github.com/kennethreitz/requests
  git_tag: v2.9.1

build:
  string: {{environ['GIT_DESCRIBE_TAG']}}

Look at what happens when I run these commands:

$ conda build --version
conda-build 1.18.1

$ conda build requests-old
...snip...

$ conda build --output requests-new
/miniconda/conda-bld/osx-64/requests-new-2.9.1-v1.0.0.tar.bz2

Uh-oh. Look at that build string...

At least the current version of conda doesn't fool you into thinking that conda build --output is actually working in such cases.

pitrou · 2016-02-02T20:26:44Z

I see. Well, each of our CI builders has a separate conda install, that must explain why it worked.

pitrou · 2016-02-02T20:27:20Z

(this probably would all be moot if conda-build allowed passing a user and token for uploading ;-))

ericdill · 2016-02-02T20:32:06Z

(this probably would all be moot if conda-build allowed passing a user and token for uploading ;-))

Agreed! 👍

meawoppl · 2016-02-03T00:16:23Z

Our CI (circle) is currently pegged to 1.18.0.
NB: The --output flag is unreliable as well. The CI does some things to ENV vars.
This thread has some details near the end about this all: #713

stuarteberg · 2016-02-03T01:31:39Z

I've implemented #758 and #759 to deal with the issues described above.

ericdill · 2016-02-03T05:31:56Z

@stuarteberg One of the main offenders of the extra info that now comes out with the --output flag is https://github.com/conda/conda-build/blob/master/conda_build/source.py#L141. The print() call on line 135 and the Popen call on line 129 are not helping matters either.

Is it important that the output from the git commands be reported to the console?

pitrou · 2016-02-03T10:30:37Z

To me it's not important.

ericdill · 2016-02-03T14:27:28Z

One workaround for now is simply to call conda build /path --output twice.

$ conda clean --source-cache
git cache (/home/edill/miniconda/conda-bld/git_cache)
Size:                                        9.8 MB

hg cache (/home/edill/miniconda/conda-bld/hg_cache)
Size:                                           0 B

svn cache (/home/edill/miniconda/conda-bld/svn_cache)
Size:                                           0 B

source cache (/home/edill/miniconda/conda-bld/src_cache)
Size:                                           0 B

Total:                                       9.8 MB
Removing /home/edill/miniconda/conda-bld/git_cache
Removing /home/edill/miniconda/conda-bld/hg_cache
Removing /home/edill/miniconda/conda-bld/svn_cache
Removing /home/edill/miniconda/conda-bld/src_cache

$ conda build dev/conda/staged-recipes-dev/recipes/lmfit/ --output
fatal: Not a git repository: '/home/edill/miniconda/conda-bld/git_cache/__github.com_lmfit_lmfit-py.git'
fatal: Not a git repository: '/home/edill/miniconda/conda-bld/git_cache/__github.com_lmfit_lmfit-py.git'
Cloning into bare repository '/home/edill/miniconda/conda-bld/git_cache/__github.com_lmfit_lmfit-py.git'...
remote: Counting objects: 5905, done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 5905 (delta 10), reused 0 (delta 0), pack-reused 5878
Receiving objects: 100% (5905/5905), 9.57 MiB | 8.35 MiB/s, done.
Resolving deltas: 100% (3746/3746), done.
Checking connectivity... done.
Cloning into '/home/edill/miniconda/conda-bld/work'...
done.
Already on 'master'
Your branch is up-to-date with 'origin/master'.
/home/edill/miniconda/conda-bld/linux-64/lmfit-0.9.2.post27-27_ga8cbeb5_py35.tar.bz2

$ conda build dev/conda/staged-recipes-dev/recipes/lmfit/ --output
/home/edill/miniconda/conda-bld/linux-64/lmfit-0.9.2.post27-27_ga8cbeb5_py35.tar.bz2

jakirkham · 2016-02-03T14:40:57Z

Thanks for looking at this @stuarteberg.

It looks like both of his changes have been merged into master. Could you guys try rebuilding conda-build from master and see if this solves your problems?

stuarteberg · 2016-02-10T15:36:43Z

Now that --output works properly with GIT_ variables (albeit with noisy output in some cases), can you close this issue?

msarahan · 2016-05-06T23:41:32Z

Closed by #758, #759, and #923

meawoppl · 2016-05-10T00:11:43Z

Which release are these changes going to be a part of?

msarahan · 2016-05-10T00:22:41Z

@stuarteberg's fixes are in earlier releases - perhaps 1.20.0. However, those release had other issues, and many people avoided them. The 1.20.2beta release is at https://anaconda.org/conda-team/conda-build/files. Install with:

conda install -c conda-team/labels/dev conda-build

I'm hoping to wrap up a couple of other bugs and do a final release over the next 1-2 days, which will be 1.20.2 final.

github-actions · 2022-04-22T06:06:25Z

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!

jakirkham mentioned this issue Feb 2, 2016

WIP: Add version control environmental variables for --output flag #750

Closed

ericdill mentioned this issue Feb 3, 2016

MNT: Remove git spam from conda build --output #761

Closed

jankatins mentioned this issue Feb 13, 2016

conda-build master introduces a large volume of spam to conda-build-all conda-tools/conda-build-all#23

Closed

msarahan closed this as completed May 6, 2016

github-actions bot added the locked [bot] locked due to inactivity label Apr 22, 2022

github-actions bot locked as resolved and limited conversation to collaborators Apr 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

conda build --output doesn't expand GIT_xxx environment variables anymore #754

conda build --output doesn't expand GIT_xxx environment variables anymore #754

pitrou commented Feb 2, 2016

pitrou commented Feb 2, 2016

jakirkham commented Feb 2, 2016

jakirkham commented Feb 2, 2016

jakirkham commented Feb 2, 2016

ericdill commented Feb 2, 2016

pitrou commented Feb 2, 2016

jakirkham commented Feb 2, 2016

pitrou commented Feb 2, 2016

pitrou commented Feb 2, 2016

ericdill commented Feb 2, 2016

ericdill commented Feb 2, 2016

jakirkham commented Feb 2, 2016

jakirkham commented Feb 2, 2016

ericdill commented Feb 2, 2016

jakirkham commented Feb 2, 2016

jakirkham commented Feb 2, 2016

ericdill commented Feb 2, 2016

pitrou commented Feb 2, 2016

jakirkham commented Feb 2, 2016

stuarteberg commented Feb 2, 2016

pitrou commented Feb 2, 2016

pitrou commented Feb 2, 2016

ericdill commented Feb 2, 2016

meawoppl commented Feb 3, 2016

stuarteberg commented Feb 3, 2016

ericdill commented Feb 3, 2016

pitrou commented Feb 3, 2016

ericdill commented Feb 3, 2016

jakirkham commented Feb 3, 2016

stuarteberg commented Feb 10, 2016

msarahan commented May 6, 2016

meawoppl commented May 10, 2016

msarahan commented May 10, 2016

github-actions bot commented Apr 22, 2022

conda build --output doesn't expand GIT_xxx environment variables anymore #754

conda build --output doesn't expand GIT_xxx environment variables anymore #754

Comments

pitrou commented Feb 2, 2016

pitrou commented Feb 2, 2016

jakirkham commented Feb 2, 2016

jakirkham commented Feb 2, 2016

jakirkham commented Feb 2, 2016

ericdill commented Feb 2, 2016

pitrou commented Feb 2, 2016

jakirkham commented Feb 2, 2016

pitrou commented Feb 2, 2016

pitrou commented Feb 2, 2016

ericdill commented Feb 2, 2016

ericdill commented Feb 2, 2016

jakirkham commented Feb 2, 2016

jakirkham commented Feb 2, 2016

ericdill commented Feb 2, 2016

jakirkham commented Feb 2, 2016

jakirkham commented Feb 2, 2016

ericdill commented Feb 2, 2016

pitrou commented Feb 2, 2016

jakirkham commented Feb 2, 2016

stuarteberg commented Feb 2, 2016

pitrou commented Feb 2, 2016

pitrou commented Feb 2, 2016

ericdill commented Feb 2, 2016

meawoppl commented Feb 3, 2016

stuarteberg commented Feb 3, 2016

ericdill commented Feb 3, 2016

pitrou commented Feb 3, 2016

ericdill commented Feb 3, 2016

jakirkham commented Feb 3, 2016

stuarteberg commented Feb 10, 2016

msarahan commented May 6, 2016

meawoppl commented May 10, 2016

msarahan commented May 10, 2016

github-actions bot commented Apr 22, 2022