Easyblocks for Anaconda and Conda Env

[jerowe]

Here are two easyblocks, referenced here: easybuilders/easybuild-framework#1791

The first, anaconda.py, installs the python distro anaconda as a module.

The second, condacreate.py, uses the conda create utility to create a conda environment, and optionally installs requirements into that environment. http://conda.pydata.org/docs/using/envs.html

Acknowledgements This work is being sponsored by NYU Abu Dhabi Center for Genomics and Systems Biology

[boegel]

@jerowe do you have example easyconfigs for these too? if so, please open a PR for those too (and refer to this PR as a requirement in the PR description)

[jerowe]

@boegel, the PR for the configs is here. PR 3173

[boegel]

@cfenoy Would you have some time to take a look at this?

[jerowe]

Configs are now here:

easybuilders/easybuild-easyconfigs#3337

Dropbox ruined my morning.

[jerowe]

Hi there.

I've been on vacation and dropped the ball on this. I'm going to go over the comments. I know I needed to add more files for tests, so I will get on that.

In regards to changing 'EB_anaconda' to just 'anaconda', when I try that it doesn't find the module. I would imagine that is because I'm running it in dev mode with

eb --include-easyblocks=/path/to/easybuild/easyblocks/a/anaconda.py anaconda-2.4.0.0.eb

?

[boegel]

@jerowe why would you want to change the class name to just anaconda? If you want EasyBuild to automagically pick up the easyblock for installing anaconda, you need to stick to the class naming scheme which involves prefixing the class name with EB_...

[jerowe]

I may have misread that. In any case I didn't change it. ;)

On Aug 11, 2016 5:19 PM, "Kenneth Hoste" notifications@github.com wrote:

@jerowe https://github.com/jerowe why would you want to change the
class name to just anaconda? If you want EasyBuild to automagically pick
up the easyblock for installing anaconda, you need to stick to the class
naming scheme which involves prefixing the class name with EB_...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#950 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFTFRiEqxW3ObUxpY3A0D33ysqL2zuCtks5qeyFHgaJpZM4It3EV
.

[jerowe]

I think it was that my configs should have

easyblock = 'anaconda'

Instead of

easyblock = 'EB_anaconda'

But when I try this I get an error saying the easyblock is not found.

[boegel]

@jerowe easyblocks that do not start with EB_ are only looked for in the generic subnamespace, so anaconda.py should be located in easybuild/easyblocks/generic/anaconda.py, or be included with --include-easyblocks /path/to/generic/anaconda.py

However, it's better to rely on the automagic derivation of which easyblock should be used based on the software name in this case imho (name = 'anaconda' => EB_anaconda)

[jerowe]

Thanks for the clarification. I think I get it now!

On Thu, Aug 11, 2016 at 10:55 PM Kenneth Hoste notifications@github.com
wrote:

@jerowe https://github.com/jerowe easyblocks that do not start with EB_
are only looked for in the generic subnamespace, so anaconda.py should be
located in easybuild/easyblocks/generic/anaconda.py, or be included with --include-easyblocks
/path/to/generic/anaconda.py

However, it's better to rely on the automagic derivation of which
easyblock should be used based on the software name in this case imho (name
= 'anaconda' => EB_anaconda)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#950 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFTFRvC5HIMxekU16_QH8TAyXC1Tj6DAks5qe2__gaJpZM4It3EV
.

[jerowe]

I had one other question I forgot to ask.

In each of these modules, Anaconda, CondaCreate, and CondaEnv, I have a few functions that are the same, such as post_install_cmd. Where would be a good place to put these? Since they are all specific to the anaconda module should they just go there? I don't like having duplicate functionality hanging out in there, but I wasn't sure where would make the most sense to put it.

[boegel]

@jerowe I think it makes sense to add them in the Anaconda easyblock, and then either import those functions in the other easyblocks, or use inheritance to derive CondaCreate from Anaconda and override the methods that are different.

Let me know if this makes sense.

[jerowe]

@boegel , I don't want to be pushy, but is there anything I can do to move this along? We have been using this method in production for our modules since August, and its been marvelous. There is internal documentation on the process, but I've been holding off on public documentation until these blocks are a part of easybuild core.

If you go here you will see that we have a build process that goes from conda env config -> easyblock config. As you can see, easybuild has become a huge part of our infrastructure. ;-)

For my part I will take a closer look at the changes you made to the anaconda block, and see if I can figure out some ways to clean up the CondaCreate and CondaEnv blocks.

[boegel]

@jerowe Don't worry about coming over as pushy, I appreciate that people bump PRs that are open for a while and are (still) very valuable to get merged.
I've been wanting to get back to it, but haven't found timefor it yet. I really want to get this in for the next EasyBuild release. I don't think it needsa lot of work, but there's just too much copy-pasting going on thatisgoing to come back to haunt us later.
Let me know if you've had any luck on reworking the Conda* easyblocks. I may be able to find some time myself this weekend...

[boegel]

@jerowe I gave this another look, there's quite a bit of opportunity for further cleanup imho.

See https://github.com/jerowe/easybuild-easyblocks/pull/2/files as another step in the right (according to me) direction. Let me know what you think.

I also think the (EB_)CondaCreate and EB_CondaEnv easyblocks should be merged into a single generic easyblock named Conda instead, with a switch to pick between conda create + conda install (i.e. what the CondaCreate easyblock now implements), and conda env create.

Maybe it's worthwhile to set up a short conf call to discuss this further, e.g. sometime next week?

[jerowe]

@boegel , thanks for taking the time to look things over! I'm really not a python developer, so this is very helpful for me.

I could do a conference call. I live in the UAE. Gulf Time Zone. UTC+04:00. Or just google 'time in dubai' ;-) What time zone are you in? So far any day but Thursday is good for me.

[boegel]

@jerowe I sent you a mail to figure out a good time for a conf call; I may spend some time this weekend on merging together CondaCreate and CondaEnv into a single generic Conda easyblock, I think that makes most sense.

[boegel]

Tested thoroughly with easybuilders/easybuild-easyconfigs#3337, good to go now.

Thanks a lot of all your work on this @jerowe, and for your patience!