Please support Intel compilers.

[ehiggs]

Hi there,
A lot of open source HPC code is built using travis with gcc and clang since it's a great resource. But then the projects are built for the actual HPC deployment, they use icc. This means the continuous integration is not building with the actual compiler that ends up being used. Not ideal!

I understand there may be licensing issues, but appveoyr provides MSVC as a compiler so maybe something can be done.

[rob-p]

I just want to chime in to strongly support this request! If something can be done to provide ICC as a target compiler in Travis, that would be great!

[rkh]

I'm not sure we can do this for license reasons (icc is only free for non-commercial projects).

[vdm113]

+1 for ICC

There could be expressed statement from user that the project is non-commercial, so it should be possible. If the user would lie, they could be prosecuted for fraud and violation of policy, terminating account, lawsuit, etc...

[vdm113]

please, see this thread: https://software.intel.com/en-us/forums/topic/564473

[vdm113]

PS: I am sorry for cross-posting, but I believe this one is in its right place so we can make a progress...

[boegel]

Intel has a "Open Source Contributor" license, free of cost, maybe contact Intel to see if you can obtain that to support Intel compilers?

https://software.intel.com/en-us/articles/open-source-contributor-faq

[joshk]

As we use the same VMs for .org and .com (public and private) repos, pre-installing this would mean we would potentially be breaking the Open Source Contributor license.

[ikirker]

I recognise that the issues around this are complex, but would like to add myself to the people hoping for it to happen regardless.

Our installation at least uses the Flex license manager: what if the binaries were available to add to the image but users had to make arrangements to let the instance contact a relevant license server? Would that suit people's compliance and use needs?

(I don't know whether you can use an open-source contributor license with the server-mode license manager.)

[nemequ]

I've been working on a script to download and install ICC on Travis. I've actually managed to get pretty close (I think I just need to figure out which components contain the missing libraries and install them), but I'm not sure when I'll have time to finish.

[jeffhammond]

I work for Intel in a research capacity and will engage my colleagues who make software for a living on this issue in January once everyone is back in the office.

For the Intel compilers, my hope is that there is a way to have the binaries installed but still require license keys, which would be encrypted.

On a related note, it would be nice if Travis could support MKL, IPP and DAAL since all of these (and TBB) are freely available to all via Community Licensing. I imagine that MKL would be wildly popular with the scientific types. I will create a new issue about this now.

[boegel]

Thank you @jeffhammond!

[nemequ]

For the Intel compilers, my hope is that there is a way to have the binaries installed but still require license keys, which would be encrypted.

FWIW that's what the script I mentioned above does. It should be a good starting point; it's a bit messy, but someone familiar with how the compiler is structured (which environment variables need to be set, which modules are necessary, etc.) should be able to clean it up quite nicely.

For MKL/IPP/DAAL/TBB, that script is designed to support them (just pass, for example, --components mkl,ipp). It just needs a bit of help to know which components to ask Intel's installer to install; for example, if you request tbb it will currently install "intel-tbb-libs__noarch;intel-mkl-ps-tbb__x86_64;intel-mkl-ps-tbb-devel__x86_64;intel-mkl-ps-tbb-mic__x86_64;intel-mkl-ps-tbb-mic-devel__x86_64;intel-tbb-source__noarch;intel-tbb-devel__noarch;intel-tbb-common__noarch;intel-tbb-ps-common__noarch;intel-tbb-psxe__noarch", but I'm not sure that's right, or if there are additional dependencies.

That said, if MKL/IPP/DAAL/TBB are redistributable, a better way might be to create an apt repository. I'm not sure Launchpad allows non-free software so a PPA may not be possible, but I think bintray would work. Of course it would be great if it were in RPM Fusion, too, but that's not really related to Travis.

[jeffhammond]

@nemequ Indeed, I saw your project and have already forked it with the intent of getting it to work. If I cannot get things working easily enough, I will request the necessary changes in a future release.

Ideally, Travis would just add Intel tools to /opt/intel in the standard VM/container so the user would only have to activate them, rather than everyone downloading and installing from scratch every time.

Having MKL/IPP/DAAL/TBB in a repo is ideal, and that was already on my list of things to suggest to my colleagues who manage the free software program.

[nemequ]

@nemequ Indeed, I saw your project and have already forked it with the intent of getting it to work. If I cannot get things working easily enough, I will request the necessary changes in a future release.

Okay, let me know if you have any questions. Email is fine, or you can /msg me on freenode or gimpnet. Or, of course, you can file an issue.

Ideally, Travis would just add Intel tools to /opt/intel in the standard VM/container so the user would only have to activate them, rather than everyone downloading and installing from scratch every time.

I don't think this is really too much of an issue; Travis seems to cache this stuff pretty heavily. When I was playing with the repo the builds were actually pretty quick, unless I didn't fire one off for a few hours in which case downloading took a long time. You can see the differences in times in the build history… If you (or anyone) can get it working properly and projects actually start using it I'm sure it would pretty much always be cached on Travis' infrastructure.

Having MKL/IPP/DAAL/TBB in a repo is ideal, and that was already on my list of things to suggest to my colleagues who manage the free software program.

That would be great. I know I'd be much more willing to use them in a project if there were packages available for Debian and Fedora. I actually had a Squash plugin for some of the compression stuff in IPP a while back, but I removed it because it was a pain to maintain.

[nemequ]

The script at https://github.com/nemequ/icc-travis now works.

[rprechelt]

I just wanted to check in here and see if there was any update on Intel allowing/providing or Travis providing access to MKL/VML (and the like) on Travis CI? I work on a number of open-source HPC projects on Github where access to MKL on Travis would be a game-changer.

@jeffhammond

[jeffhammond]

@rprechelt Sorry, I haven't had time to address this. I know six months is a long time, but my queue is deep. I have not forgotten about this and will try to set something up with the right people in a few weeks.

FWIW, I think @nemequ did a wonderful thing showing how to install Intel compilers from within Travis. I haven't figured out how to reproduce it but it should be easy enough to extend that infrastructure to support MKL. I got it working and am extending to support MPI, MKL, Fortran, etc. (which is trivial enough - I just had to write the tests).

[rscohn2]

MKL & other intel library shared libraries are now installable with apt-get/yum
https://software.intel.com/en-us/articles/intel-parallel-studio-xe-runtime-2017
https://software.intel.com/en-us/articles/installing-intel-parallel-studio-xe-from-apt-repository

No header files yet.

[jeffhammond]

@rscohn2 Fantastic! I really hoped that we would do this.

What's the ETA for headers? C/C++ projects that call MKL functions outside of the standard BLAS/LAPACK will need mkl.h? If not public info, please send me info at my Intel email. Thanks.

[ax3l]

Uhh... 😃 then we should open a PR to add those repos to the apt package source whitelists in travis so people can use it in non-sudo containers, too: https://github.com/travis-ci/apt-source-whitelist
Hopefully they need no set(uid|euid|gid) during the install...

[jeffhammond]

Many HPC folks are still waiting to see travis-ci/apt-package-safelist#406 resolved after 1.5 years of intense interest from users. I have no confidence that Travis whitelisting will occur in finite time 😞

[ax3l]

Honestly, just partition your travis build environments from source via spack for HPC applications and cache it.

[jeffhammond]

@ax3l I just build everything from source manually, which is a trivial extension of the scripting infrastructure I have for non-Travis purposes. It just frustrates me that Travis won't make MPICH a first-class citizen in their ecosystem even though they whitelist packages that depend on MPICH and whitelist Open-MPI, which does all of the same things that MPICH does.

[ax3l]

@jeffhammond I totally feel your pain.
My standard environment is a recent CUDA on top or a recent compiler with a recent Boost with a recent MPI with a recent HDF5 or ADIOS with a recent... we are getting off-topic ;)

But to be fair, the packages for MPICH seem to use set(uid|euid|gid) so we should better ask the package maintainer if that is necessary and if not ... their (reasonable) auto-tests will pass.

Update: I just checked the build and your comments... yeah, aww...

[jeffhammond]

@ax3l Please see travis-ci/apt-package-safelist#406 (comment) for analysis of set(uid|euid|gid). It's a canard but Travis folks don't appear willing to read on or act on my analysis.

[ax3l]

Yep, and it's well argued. I think they just automate their workflows heavily since they get tons of requests and offer a free service to the public.

[davidbeckingsale]

As a developer of a number of open source projects at Lawrence Livermore National Lab, I wanted to add my +1 to this request. Lots of our projects prefer to use vendor compilers, so being able to test with these but also use the Travis infrastructure would be a big plus.

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[jagerman]

Can someone (perhaps @ehiggs) please reopen this? 90 days without activity does not a solution make, despite what simple-minded bots may think.

[ehiggs]

@jagerman I don't seem to have permission to reopen this. Ping @rkh (first travis-ci member I see who has commented).

[akimd]

@rkh Could you please revive this?

[rscohn2]

We have published samples for all the public CI systems:
https://github.com/oneapi-src/oneapi-ci