Add tifffile

[danielballan]

This adds a tifffile recipe. I once built this on appveyor (the packages are here) but I'm now fuzzy on the details about how it went. I need to build packages for Python 3.5 and, I think, numpy 1.10, so I thought we could try conda-forge.

Where do I go from here?

[scopatz]

You see if it works :) AppVeyor is breaking a lot this week.

[danielballan]

Appveyor likes 2.7 fine, but not 3.4.

[scopatz]

This is the same issue that I am seeing. It is probably because py2k uses VC10.

[scopatz]

For pure python stuff, though, there is no reason for the compiler to come into it. This is pretty annoying.

[danielballan]

Tifffile has optional (but useful) C extensions, so unfortunately I have to solve the hard problem.

[ocefpaf]

For pure python stuff, though, there is no reason for the compiler to come into it. This is pretty annoying.

Indeed, but that seems to be the new conda-build default. We need to adapt... Again...

[jjhelmus]

I believe the issue is that the new Miniconda3 installer creates a Python 3.5 root environment and there is not a Python 3.5 version of obvious-ci in the pelson/development channel, hence the conflict. Creating a Python 3.4 conda environment and installing obvious-ci there or uploading a Python 3.5 version of obvious-ci to the pelson/development channel would fix this. I think @pelson would need to do the latter.

[jjhelmus]

A third option is to use an older version of the Miniconda3 installer in the bootstrap script which still uses Python 3.4. I believe Miniconda3-3.16.0 installs a Python 3.4 root environment.

[scopatz]

I think that this is a growing pain. We should be able to support multiple versions of py3.x.

[scopatz]

Also, maybe the ObviousCI stuff should be moved into conda-forge so that we don't have to wait for @pelson for everything ;)

[ocefpaf]

Also, maybe the ObviousCI stuff should be moved into conda-forge so that we don't have to wait for @pelson for everything ;)

👍

[jjhelmus]

By mucking around with the appveyor.yml file I was able to get AppVeyor to build tifffile packages for all the selections in the matrix. I want to clean this up tomorrow and figure out the root cause of some of the issues. If anyone wants to take a look before then work in in the very messy add-tifffile-jh branch in jjhelmus/staged-recipes.

[danielballan]

Hmm, it does look like there a couple issues at once here. Thanks for digging into this.

[pelson]

I've triggered a re-build to see if recent changes (including a obvious-ci-feedstock for conda-forge) have solved the problem.

[jjhelmus]

I think with work @pelson did in PR #16 and the addition of Python 3.5 Obvious-CI conda packages in the conda-forge channel, the changed I made in my branch are mostly un-needed.

Would there be a benefit to using the pre-installed version of Miniconda, appveyor/ci#359, over installing it with the bootstrap script? It might speed up the build slightly.

Also, I'm happy to move this discussion to its own issue, as this is starting to get off topic from the tifffile recipe.

[pelson]

Would there be a benefit to using the pre-installed version of Miniconda, appveyor/ci#359, over installing it with the bootstrap script? It might speed up the build slightly.

Definitely 👍 on that. Agree we should do it in a separate PR.

[pelson]

Woops. @danielballan would you mind rebasing this PR?

(The error was

'obvci_conda_build_dir.py' is not recognized as an internal or external command,
operable program or batch file.
Command exited with code 1```
)

[jjhelmus]

I have to work on day-job items today, but tonight I'll put together a PR is no-one beats me to it.

[danielballan]

Rebased.

[danielballan]

Woo-hoo!

What's the next step? Create a feedstock? And, to echo @scopatz's question from the mailing list, where do the packages go?

[jjhelmus]

Until the PR is merged I do not think the packages go anywhere. Once merged they get uploaded to the conda-forge organization on anaconda.org. For example see the OS-X xonsh upload from Travis-CI and the resulting file on anaconda.org

[danielballan]

OK, that makes sense. And will this recipe will eventually removed from this repo -- sometime after it is merged?

[scopatz]

But that is only for this specific repo. Other feedstocks have to be manually hooked up.

[scopatz]

I think the idea is that these recipes will just live here forever.

[pelson]

Does tifffile have a numpy C ABI dependency? If so, we will need to follow the pattern layed down with shapely: https://github.com/conda-forge/staged-recipes/blob/master/recipes/shapely/meta.yaml#L21 and pin the numpy with numpy x.x in run and build requirements.

As for the other questions:

I think the idea is that these recipes will just live here forever.

Who knows! 😉 I certainly think that the original concept was for 1 recipe 1 repository (aka. a feedstock) but where we go from here depends ultimately on usability. @ocefpaf and I have now accrued a huge amount of experience managing these kinds of repositories and there is no doubt that it is a lot of work - if we can reduce that with individual feedstocks then I'm all for them.

Until the PR is merged I do not think the packages go anywhere

Exactly.

But that is only for this specific repo. Other feedstocks have to be manually hooked up.

For now. For instance, I had to do that this morning for the obvious-ci-feedstock, but I believe this is automatable (with very little effort).

will this recipe will eventually removed from this repo -- sometime after it is merged?

As commented above, that depends on where we take the project. One option would be for all feedstocks to be created automatically from recipes in this repo. The truth is, we still don't have the conda-smithy framework anywhere near polished enough to have clear answers. Despite that, it took less that an hour to create a new feedstock this morning, and that time will be hugely reduced with refinement.

[danielballan]

Thanks for quick answers, everyone. I like where this project is headed. I made a feedstock.

Does tifffile have a numpy C ABI dependency?

Yes. I was not aware of the numpy x.x specification. From what I can gather from some quick Googling, this means the latest minor release of every (modern-ish) major version. Let me know if I've misunderstood that. I pushed a new commit adding it to the meta.yaml -- we'll see how it goes....

[danielballan]

It passed here, but my feedstock is not faring as well:

python ci_support\upload_or_check_non_existence.py .\recipe conda-forge --channel=main
Traceback (most recent call last):
  File "ci_support\upload_or_check_non_existence.py", line 101, in <module>
    upload(cli, meta, owner, channel)
  File "ci_support\upload_or_check_non_existence.py", line 51, in upload
    env=os.environ)
  File "C:\conda\lib\subprocess.py", line 579, in check_call
    retcode = call(*popenargs, **kwargs)
  File "C:\conda\lib\subprocess.py", line 560, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\conda\lib\subprocess.py", line 950, in __init__
    restore_signals, start_new_session)
  File "C:\conda\lib\subprocess.py", line 1220, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

The full log is here. Has anyone seen this one before?

[danielballan]

And CircleCI has a separate problem, due to numpy x.x inducing it to try Python 3.5 with numpy 1.8. I guess that would be an issue with conda itself.

+ export CONDA_NPY=18
+ CONDA_NPY=18
+ export CONDA_PY=35
+ CONDA_PY=35
+ set +x
Removing old build environment
Removing old work directory
BUILD START: tifffile-0.3.1-np18py35_0
Fetching package metadata: ........
Solving package specifications: .......
Error: Unsatisfiable package specifications.
Generating hint: 
[      COMPLETE      ]|##################################################| 100%


Hint: the following packages conflict with each other:
  - python 3.5*
  - numpy 1.8*

full log

[danielballan]

It seems like anaconda-client is not being installed, both for the CircleCI and appveyor.

[pelson]

@danielballan it seems like anaconda-client is not being installed

See conda-forge/conda-smithy#31. My apologies if you went around the houses with that.

[pelson]

I suspect we should be including setuptools as a build dependency, but I wont hold the PR up on that basis as it is coming from numpy anyway (presently).

Once this is merged, keep an eye on https://anaconda.org/conda-forge/tifffile and watch the distributions come in (CircleCI is by far and away the quickest service, and the Linux distributions will be first, followed normally by the OSX and then a slow trickle of win32 and win64 ones).

👍

[pelson]

Seems I was too harsh on AppVeyor - one of the Windows builds came in at the same time as the first Linux one. 😉

[danielballan]

@pelson Hahah, I will definitely borrow the expression "went around the houses." It should have been able to spot that issue myself.

I'm onto the next error now...I'll open a new issue if I can't figure it out.