FileNotFoundErrors for current builds

[chapmanb]

@bioconda/core -- I'm running into a persistent build issue this morning and not sure how to fix. I keep getting conda errors on Linux machines with:

AttributeError: 'FileNotFoundError' object has no attribute 'read'

Here is the current build I'm stuck on:

https://travis-ci.org/bioconda/bioconda-recipes/builds/180035088

I found a conda issue that reports a similar problem (conda/conda#3947) but I've tested locally with multiple version of requests and it seems to work fine.

I'm confused as the OSX build worked okay (and that's where Travis was reporting issues this morning) so am not sure how to proceed. Does anyone have tips/tricks/ideas? Thanks so much.

[acaprez]

I'm having the same issue with my build yesterday: https://travis-ci.org/bioconda/bioconda-recipes/jobs/179859694. Some other builds I glanced at from yesterday had the same problem as well. See also #3075.

[pirovc]

#3068
#3032
Same issue here.

[colinbrislawn]

Yep. Same. Also, all the linux builds try to build

INFO bioconda_utils.build:build_recipes(266): Recipes to build: 
nudup
dsrc
<your actual recipe>

🙏 Help!

[johanneskoester]

The bug only occurs in the condaforge/linux-anvil docker container. One solution would be to pin the used version of condaforge/linux-anvil. However, they only seem to have the "latest" tag. As far as I know it is not possible to pull a specific build id.
@jakirkham do you see the same issue with your container already? Would it be possible to add a "stable" tag or something like that?

[jakirkham]

Maybe an issue with the new conda-build. IIRC was using 2.0.10 and now has 2.0.11.

[johanneskoester]

@daler, is it right that we just use the conda-build that is shipped with the container? This is problematic in any case I think, because we have a different conda-build version that is used for osx. Would you agree that up/downgrading everything to our definitions in bioconda/bioconda-utils/conda-requirements.txt is a proper fix and puts us back in control? This could be done in bioconda-utils, before the actual build command is executed in the container.

[daler]

We're using 2.0.7 (as configured here) and the first thing the build script does is install that version in the container. It can be configured from the cli but we're not doing that yet. The travis linux and osx environments use 2.0.7 as well (configured here, called here).

A versioned tag for the container would be very helpful!

[johanneskoester]

@daler, instead of only pinning conda-build in the script, what about enforcing the versions of all dependencies to be the same (i.e. using conda-requirements.txt)?

[daler]

Definitely. I was thinking the same thing after writing that last comment.

[johanneskoester]

Do you have time to do that quickly? I would do it, but I'm travelling and have to prepare a talk...

[daler]

On it now.

[johanneskoester]

Thanks!!!

[chapmanb]

Is anyone able to replicate this locally? I've been trying without success using different versions of bioconda-builder and requests with the latest condaforge/linux-anvil container. Everything works for me locally and then no go on Travis, so trying to figure out what is different to help with this.

[johanneskoester]

That's weird, especially since it is the same container. Maybe the bug is also present locally, but only triggered by a network issue on travis?

[daler]

@chapmanb I can't reproduce locally either. Just updated bioconda-utils and bioconda-recipes to try to be more explicit with pinning reqs, will have to wait for tests, travis has been bogged down recently.

@jakirkham, https://hub.docker.com/r/condaforge/linux-anvil/builds/ and https://quay.io/repository/condaforge/linux-anvil say a new build was triggered 3 days ago, but https://github.com/conda-forge/docker-images hasn't been updated in a while. I'm guessing I have the wrong github repo. Can you point to where the Dockerfile for condaforge/linux-anvil is stored, or more specifically, what changed 3 days ago?

[chapmanb]

Ryan -- thanks so much for digging into this. Fingers crossed that pinning the requirements will work. Let me know if I can help with anything from my side.

[jakirkham]

Yeah, so we had issues with Docker Hub's automated build back during the summer. So we switched to Quay. That is where the actual build is done. Docker Hub just mirrors the build from Quay. We do trigger Docker Hub to pull in the image from Quay at the end. Though lately I have noticed some lag in this process. So Docker Hub sometimes does not always get the latest image from Quay.

[jakirkham]

So we do trigger rebuilds manually even when the Dockerfile doesn't change. Mainly this is to make sure we have the latest versions of things in docker container. The reason that works is we run conda update --all. Thus there could be changes inbetween these builds. Also looking to see if there is anything else obvious.

[daler]

OK, thanks. In that case, maybe pinning will help on our side.

[jakirkham]

How are you guys using conda in the container?

[daler]

On each travis build, we build a new local container with

FROM condaforge/linux-anvil
COPY requirements.txt /tmp/requirements.tx
RUN /opt/conda/bin/conda install --file /tmp/requirements.txt

Then in the container,

# this is redundant with the above installation step
# and will change, but it's what we're doing now
conda install conda-build=2.0.7

conda build $RECIPE

Should we be calling /opt/conda/bin/conda directly?

[jakirkham]

So it looks like conda gets upgraded in the OP linked build, but conda-env is not? That seems like a potential source of problems.

[jakirkham]

Also retriggered the Docker Hub build as it was out-of-date w.r.t. the one on Quay and it has just recently completed. As such, it has the newest conda and conda-env. Would be curious to know if that still has issues.

[daler]

Ah, we're not pinning conda-env, I guess I assumed it came in with conda. I'll fix that.

I'll also change the FROM container to the quay.io container directly to avoid any build lag time.

Like I mentioned we're pinning versions, but I'll also try temporarily disabling that to try the newest conda and conda-env from the newly-built container.

[jakirkham]

So given that you guys are making more use of our image. I think it would be good to have someone from bioconda added as a maintainer of our image. I don't know who would be the best point person or two (redundancy is good) maybe let us know. Probably will need to discuss with core about it before it is finalized, but I think that should help.

[jakirkham]

Ah, we're not pinning conda-env, I guess I assumed it came in with conda. I'll fix that.

Yeah I did to, but it wasn't showing up as being upgraded in that log. Weird.

I'll also change the FROM container to the quay.io container directly to avoid any build lag time.

Sounds good. Maybe we should deprecate the Docker Hub one to avoid these sorts of problems.

[jakirkham]

So versioning came up a few times. That is definitely a smart idea.

Though there is the question of how to version things. In Jupyter's docker-stacks, we used git sha, which could be an option. If there are other ideas, please suggest.

The conda update --all command is definitely trouble when it comes to versioning. We need to switch from defaults packages to conda-forge packages. We would have to toss it though and pin versions of everything we install. However, that still might not cut it as it doesn't include new dependencies of conda and friends. So they would still come from defaults and/or have unpinned versions. Still pinning a set of them to major minor versions (floating the patch version) should help.

Something else that we have been discussing is actually having a conda-forge installer akin to Miniconda, but totally open sourced. This would then have fixed versions included in it and be versioned somehow. I think last this came up date based versioning was popular. Though git sha or something else could be used too. Also it would help standardize across OSes. We could then just install a particular version of the installer in the container and that could help. It would also make conda update --all unnecessary.

Maybe we should get an issue going on our docker repo so as to better discuss and track this.

[jakirkham]

Any luck @daler ?

[daler]

Still waiting on travis. Preliminary tests suggest that pinning did the trick, but code is split across two repos that are tested on osx, and that is slooooow recently on travis

[jakirkham]

Yeah, they have a bit of a backlog that they are still working through. Also fun fact both Linux and OS X builds on Travis CI share the same queue. Remember that being pretty surprising when I learned that. May help in prioritizing builds if you should desire to do so.

[daler]

Huh, I didn't realize that. Maybe we should disable OSX altogether until this is ironed out.

Then again, I'm trying to juggle this with work, so the spaced-out time intervals are kind of helpful.

[chapmanb]

Ryan -- you are awesome. With the merge I got a build to complete. Sweet. Going to rebase in my new updates for my branch and get it committed. Thanks so much for unsticking this.

[daler]

Great! I'll go around reminding folks with recent failing PRs to rebase and retry.

@jakirkham thanks for your help!

[johanneskoester]

@jakirkham I would volunteer for becoming a maintainer of the container. Regarding a conda-forge installer, I don't really see the benefit of that at the moment. Just sounds like additional effort and a source of further inconsistencies and confusion for the users. It is already a hassle to explain what is the difference between conda, anaconda and miniconda to non-CS people. I'd vote for just pinning the necessary parts of the build system, and getting rid of the update --all step. But I'd be happy to discuss this further if you agree to add me as a maintainer.