New CentOS5/manylinux based Docker image

[jjhelmus]

New Docker image based upon CentOS 5.11 and the manylinux image for building conda packages for linux-64.

[jjhelmus]

The Dockerfile and other files needed to create this image can be found in the jjhelmus/conda-forge-docker-image repository. I'm happy to move this repo over to the conda-forge organization.

[jjhelmus]

I've also tested this Docker image by building the arm_pyart recipes. I'm guessing if the image looks good then next step would be to submit a PR against conda-smithy and then re-render the feedstocks.

[ocefpaf]

👍

[msarahan]

If it's OK, I'd like to work towards using a docker image that Continuum is working on. We are going to be standardizing internally on this docker image very soon, and keeping conda-forge on the same platform will be very important for assuring compatibility going forward.

This work is at anaconda/docker-images#20

I will be reworking it (slightly) to follow manylinux more completely, with their compiler flags where appropriate and with their auditwheel tool. The main differences between this image and manylinux are:

• This image updates to GCC 5.2 and a newer binutils. Internally, we need this for some projects (dynd, numba), and we are committed to the Julia route of keeping up to date with libstdc++, so that we never shadow the system libstdc++ with an older version.
• This image compiles GCC with multilib so that only one image is required for 32 & 64 bit targets. This will require careful examination of CFLAGS in recipes, so that users do not clobber the "-m32" flag that emits 32-bit output.

I'll do my best to keep the important parts of manylinux - notably the nicer script approach to docker setup, compared to my concatenated string approach. @jjhelmus if you're amenable to this approach, I'd welcome your feedback/guidance.

[msarahan]

For purposes of discussion, I humbly submit jjhelmus/conda-forge-docker-image#1 - a tweak of this PR that uses the aforementioned Continuum build image.

[pelson]

@jjhelmus & @msarahan - this all sounds excellent. If we can arrive at a solution which is consistent with what both Continuum & manylinux are doing I'm all in favour 👍

[jjhelmus]

@msarahan and I will work out the details in the jjhelmus/conda-forge-docker-image repo and will get back once we have a solution.

We could change the image to the manylinux based on for the time being, although it will need replacing sometime in the future. Depend if we want to fix the matplotlib issue fast or wait for a more complete solution later.

[pelson]

@jjhelmus - at one point, I'd separated the linux build to do testing in a separate docker image. That meant I could have all the build tools I wanted in the build image, but still only the limited subset in the test image. Is that something we should be re-visiting?

[jjhelmus]

Having a base docker image which can be used for testing upon which the "build" docker image is built should be possible. Can one of the conda-forge members (@ocefpaf, @pelson?) with sufficient permissions make a docker-images repository under the conda-forge organization where these details can be hashed out and the Dockerfile and supporting files hosted?

[ocefpaf]

Can one of the conda-forge members (@ocefpaf, @pelson?) with sufficient permissions make a docker-images repository under the conda-forge organization where these details can be hashed out and the Dockerfile and supporting files hosted?

I am not it. But I also vote to give you adim rights too. What do you think @pelson?

[ocefpaf]

@jjhelmus how about https://github.com/conda-forge/docker-images?

[jjhelmus]

Works for me, thanks @ocefpaf

[jjhelmus]

I'm of the opinion that this PR should be merged so that the CentOS 5 docker image with the various shared libraries gets into any new pull requests. I'll also submit a PR to add the image to conda-smithy so it gets changed in the new feedstocks.

Eventually this docker image should be replaced with the one @msarahan and I are working on with all the details worked out but until then I think this is a good holdover. Packages build with the current image (pelson/obvious-ci:latest_x64) are linked to the new CentOS 6 glibc and will not work with CentOS 5 and will need to be re-built, see discussion #119. Making the switch now will at least reduce the number of packages which need to be rebuilt.

[msarahan]

+1

[jjhelmus]

conda-forge/conda-smithy#73 will change the feedstocks

[pelson]

I'm 👍 on the change, but think that for full reproducibility the Dockerfile should be merged into the repo (fine to do that directly) and the build should come from hub.docker's build system (as per https://hub.docker.com/r/pelson/obvious-ci/~/dockerfile/).

Thanks for doing this @jjhelmus - I think this is an excellent step forwards! 👍

[jjhelmus]

I had not yet played around with docker hub's build system, thanks for pointing it out. I will have to try it out.

[jakirkham]

Yeah, I have a bit of experience with Docker Hub's build system so can help out.

I like the idea of having an automated build and having it in a repo somewhere. Though I think it would be nice to keep the Docker images separate. The repo @ocefpaf created seems like a good place. ( https://github.com/conda-forge/docker-images )

[scopatz]

Is this needed to get C++11 support?

[jakirkham]

This is one of many proposals. Two other proposals and the bulk of the discussion is happening on this issue ( conda-forge/conda-forge.github.io#29 ).

[msarahan]

I think the minimum spec for C++11 is GCC 4.8, but @jakirkham is right, this is unsettled.

[jakirkham]

Weird. Sometimes ZenHub does whatever it wants with these pipelines lately.

[jakirkham]

Sorry for that.

Yeah, there are two main parameters being discussed. OS version (as a proxy to GLIBC version) and compiler version. There are other related things, but the rest gets pretty technical. Depends how deep you want to wade into it.

[scopatz]

I see, but conda-forge/conda-forge.github.io#29 hasn't been touched in 7 days. Is there are consensus forming on what has to happen?

[jakirkham]

Not really. Though I'm going to bring it up at our next meeting.

Just to note, we still have some way to do this now. It is just not as pretty as it could be.

[scopatz]

So what is the way to do it for now? I'd like to be able to get #390 working so that I can move forward with conda-forge as part of the release process for cyclus. As the maintainer of these C++11 packages, I'd be happy to update them later when there is a better way to do things.

[jakirkham]

Right, I added some notes over there about how to get that to work. Basically, using clang on Mac works great. On Linux, we use conda's gcc and libgcc packages, which work well enough. On Windows, one has to use vc14, which is done by forcing Python 3.5 (or later when that comes).

[scopatz]

Cool, where did you add the notes to @jakirkham? I don't see them on the hackpad or in the readme.

[jakirkham]

That would be a good idea. Definitely on my checklist. Right now just in a comment.

[scopatz]

Oh, I see, thanks! Sorry for the noise.

[jakirkham]

No worries.

[jjhelmus]

closing in favor of #514

[jakirkham]

Thanks @jjhelmus. I think we just need to pick a name ( conda-forge/docker-images#8 ) and we should be good.