configure complex_list for master and exec hosts #43

Open
delagoya opened this Issue Aug 4, 2011 · 7 comments

Comments

Projects
None yet
3 participants
@delagoya

delagoya commented Aug 4, 2011

Specifically I wanted to add the exclusive directive to the complex_list.

@delagoya

This comment has been minimized.

Show comment Hide comment
@delagoya

delagoya Aug 5, 2011

best left for a plugin to do. Shall I close the issue?

delagoya commented Aug 5, 2011

best left for a plugin to do. Shall I close the issue?

@jtriley

This comment has been minimized.

Show comment Hide comment
@jtriley

jtriley Aug 5, 2011

Owner

I've been meaning to pull the SGE code out into a separate plugin that's included with StarCluster.

This would allow users to configure it separately (via it's own [plugin] section) where could add an option for complex_list, for example, as well as queues to configure, etc. It would also allow users to run plugins before SGE is configured or just completely disable SGE altogether.

In this case I would likely want to add a setting like enable_queue to the cluster config to remove the need to define a new plugin section just to get the default SGE install/config back. Does this sound reasonable?

Owner

jtriley commented Aug 5, 2011

I've been meaning to pull the SGE code out into a separate plugin that's included with StarCluster.

This would allow users to configure it separately (via it's own [plugin] section) where could add an option for complex_list, for example, as well as queues to configure, etc. It would also allow users to run plugins before SGE is configured or just completely disable SGE altogether.

In this case I would likely want to add a setting like enable_queue to the cluster config to remove the need to define a new plugin section just to get the default SGE install/config back. Does this sound reasonable?

@delagoya

This comment has been minimized.

Show comment Hide comment
@delagoya

delagoya Aug 5, 2011

SGE plugin would make sense if you wanted to configure more than one scheduler (Condor, TORQUE, etc). In this case I would put in an setting in cluster config called "queue_engine" which defaults to SGE.

But this seems like a change better left for a major revision number. For this version I think the SGE support is the reason folks come to starcluster in the first place. I know it was for me.

I will see if I can get time to code a decent plugin to customize SGE installs with a few common options, starting with whether to remove master from exec host and add exclusive rights to the scheduler. Any other common options you would like to see?

delagoya commented Aug 5, 2011

SGE plugin would make sense if you wanted to configure more than one scheduler (Condor, TORQUE, etc). In this case I would put in an setting in cluster config called "queue_engine" which defaults to SGE.

But this seems like a change better left for a major revision number. For this version I think the SGE support is the reason folks come to starcluster in the first place. I know it was for me.

I will see if I can get time to code a decent plugin to customize SGE installs with a few common options, starting with whether to remove master from exec host and add exclusive rights to the scheduler. Any other common options you would like to see?

@ank

This comment has been minimized.

Show comment Hide comment
@ank

ank Aug 5, 2011

@delagoya, I just pushed a few plugins I find useful to save you little work. If you look at the 'extras' plugin it does some SGE config like setting the scheduler interval and adding the cluster user as the admin host. To remove the master as exec I think you could just add a 'qconf -de master' to my naive implementation.

https://github.com/bioteam/StarClusterPlugins

ank commented Aug 5, 2011

@delagoya, I just pushed a few plugins I find useful to save you little work. If you look at the 'extras' plugin it does some SGE config like setting the scheduler interval and adding the cluster user as the admin host. To remove the master as exec I think you could just add a 'qconf -de master' to my naive implementation.

https://github.com/bioteam/StarClusterPlugins

@delagoya

This comment has been minimized.

Show comment Hide comment
@delagoya

delagoya Aug 5, 2011

Thanks Adam, will take a look at those.

delagoya commented Aug 5, 2011

Thanks Adam, will take a look at those.

@jtriley

This comment has been minimized.

Show comment Hide comment
@jtriley

jtriley Aug 5, 2011

Owner

@ank excellent, thanks for forking StarClusterPlugins, this is precisely where I'm hoping to collect plugins from users. Eventually we should do a 0.1 release of StarClusterPlugins on pypi so that folks can install them easily. Feel free to submit a pull request whenever you're ready...

Owner

jtriley commented Aug 5, 2011

@ank excellent, thanks for forking StarClusterPlugins, this is precisely where I'm hoping to collect plugins from users. Eventually we should do a 0.1 release of StarClusterPlugins on pypi so that folks can install them easily. Feel free to submit a pull request whenever you're ready...

@jtriley

This comment has been minimized.

Show comment Hide comment
@jtriley

jtriley Feb 25, 2012

Owner

@delagoya @ank the SGE setup code has now been moved to a new SGE plugin in the develop branch (see starcluster/plugins/sge.py) in case you're interested in adding new settings (via init kwargs) to customize the SGE install/configuration. Currently users will need to specify disable_queue=True in their config given that SGE is a default for now and then define the SGE plugin and specify it in your PLUGINS list in your config in order to customize the extra options. I'm happy to help if you have questions.

Owner

jtriley commented Feb 25, 2012

@delagoya @ank the SGE setup code has now been moved to a new SGE plugin in the develop branch (see starcluster/plugins/sge.py) in case you're interested in adding new settings (via init kwargs) to customize the SGE install/configuration. Currently users will need to specify disable_queue=True in their config given that SGE is a default for now and then define the SGE plugin and specify it in your PLUGINS list in your config in order to customize the extra options. I'm happy to help if you have questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment